Using sigterm / catching sigterm to enable checkpointing about syne-tune HOT 10 CLOSED

Thanks and thank you for all your replies! I think having the simple option of storing every epoch and the slightly more advanced, but probably also very simple to implement option of catching a sigterm for storing on demand would be great options to choose from for a user.
Basically, I got caught up in the docs that explain the helper functions and was a bit confused, also making clear that the recommended solution currently is to store every epoch also seems totally fine.

from syne-tune.

Btw, I forgot to mention we have this example that shows how to activate checkpointing for an xgboost script:

• example: https://github.com/awslabs/syne-tune/blob/main/examples/launch_checkpoint_example.py
• associated training script: https://github.com/awslabs/syne-tune/blob/main/examples/training_scripts/xgboost/xgboost_checkpoint.py

It may be useful as perhaps you are considering scikit-learn pipelines.

I am planning to have a look whether using sigterm works with checkpointing, I will let you know.

from syne-tune.

Thanks, I made checkpointing work, I'm actually using custom pytorch transformers 🤯
Crashed when the hard drive was full of checkpoints, now looking at your docs for fixing that :) The journey continues!

from syne-tune.

Hello Andreas,
we want to keep things as simple as possible on the side of the training script. In fact, in all our examples, checkpointing is done at the end of each epoch. Your suggestion would save time by checkpointing less often, but it would be more difficult to implement.

from syne-tune.

Ah ok, the documentation suggested using the helper functions, which seems more complicated than doing the exception as it requires passing the config objects around. But I saw that actually in the end it's always stored. So yes, it's more complicated than always storing but seems easier than the method that the code suggests was planned.

from syne-tune.

Hi Andreas,

This makes sense, you also made a good point about using sigterm instead of sigkill, I will take a look and let you know if we can support it.

from syne-tune.

https://syne-tune.readthedocs.io/en/latest/faq.html#checkpoints-are-filling-up-my-disk-what-can-i-do

The number of checkpoints stored only scales with the number of trials, not with how often a trial stores a checkpoint. Storing after every epoch simply just overwrites what was there before.

There is a speculative checkpoint removal feature, which is documented in the link above. Maybe this helps.

from syne-tune.

Hi @amueller, was that link helpful or do you need further support?

from syne-tune.

I'm good, thank you!

from syne-tune.

OK perfect, closing.

from syne-tune.