[Bug]: Run fails on TPU after 16 minutes with `INTERRUPTED_BY_NO_CAPACITY` about dstack HOT 4 CLOSED

@yixiaoer, are you running a spot or on-demand TPU? INTERRUPTED_BY_NO_CAPACITY means the instance became unavailable and this usually happens with spot instances when they get interrupted by the cloud provider.

If the task can handle interruptions, you can specify a retry policy to resubmit the run when it gets interrupted.

If the task cannot handle interruptions, consider using on-demand instances: pass --on-demand to dstack run.

from dstack.

I was running on a spot TPU. After specifying the retry option with interruption, it retried, but later still lost connection.
And also tried using the --on-demand option with dstack run, but the problem persists.

Can this be related to the TPU memory capacity? The dataset downloaded is quite large (approximately 22GB, specified to download in /dev/shm). However, no errors were reported for the code running; I also specified in .dstack.yml:

resources:
  memory: 100GB
  shm_size: 50GB

Is this the correct way to specify resources for TPUs? Given the situation, is there anything else I can do to resolve this issue?

from dstack.

@yixiaoer, it's quite strange that the problem persists with --on-demand. Could you please double-check it? Also, show dstack ps once you try it to see whether it used spot or not.

Also, to ensure on-demand is used, you can set in the YAML spot_policy to on-demand then to ensure it doesn't use spot instances.

Please let me know if you can check it.

Yes, the resources looks OK to me!

Also, in case it doesn't work again, could you please share the repo with train.dstack.yml and scripts so we can try to reproduce it?

from dstack.

thanks! I double-checked and tried running with --on-demand again. This time, it ran successfully.

from dstack.