W&B config synced only at the end of the run about wandb-offline-sync-hook HOT 4 CLOSED

Thanks for the issue @CedricLeon and for the link to wandb/wandb#6952 (I hadn't seen that). Unfortunately I don't have any solution for this at the moment.

It is really frustrating to be honest, especially if runs fail/get killed by a scheduler (in that case, no config is ever uploaded).

One thing I've tried was to set the config via the wandb API during the run (with this script that takes the hyperparameters from the lightning config and then sends them to wandb), but it seems like they get overwritten with the next sync.

from wandb-offline-sync-hook.

Obviously, this is more of a bug in wandb, though wandb-osh could include a fix/workaround if only we had one (but I don't know of any). Let me know if you discover something!

from wandb-offline-sync-hook.

Yes, I tried a similar trick than you, but it gets overwritten at the next synchronization it seems.
Ok thanks for your insights, we will see if they ever try to close wandb/wandb#6952...

So far, my "fix" is to (from time-to-time) run a script I wrote that scans my wandb project for runs which do not have a specific key (coming from my HYDRA config) available on wandb. For each of this run, I find the corresponding log folder in my workspace, do some manual interpolation on the config (because OS variables have changed), and then overwrite the wandb config and update it.
If you are interested, I can try to clean and publish this script somewhere

from wandb-offline-sync-hook.

I described the full problem in a wandb issue: #7227

from wandb-offline-sync-hook.