Runtime error testing lightning pipeline about microdl HOT 5 CLOSED

I have reserved an OnDemand session on gpu-c-1 with one gpu, with 32 nodes, 20GB memory per node. The training config I am using is here:
/local/scratch/groups/cmanalysis.grp/microDL_SP/config/lit_test/HEK_dNucl_2023_04_21_15.yml
It seems to have trained for 70 epochs over a day.

70 epochs/day is not too bad IMO. But you can probably get better performance by using more CPU cores (e.g. 64) in data loading.

from microdl.

How many GPUs do you have? Currently it doesn't work with multiple GPUs.

For a Slurm session check with:

echo $CUDA_VISIBLE_DEVICES

If there's more than one you will need to manually modify this line to use one specific GPU:

from microdl.

Thanks @ziw-liu! It works with one gpu.
Is it normal for the error to appear a while after the training starts? I am trying to understand if my training is slow. I am working on the scratch space on gpu-c-1.

from microdl.

Is it normal for the error to appear a while after the training starts?

Depending on what do you mean by 'start'. By default it will first do several dummy validation iterations as a sanity check, which might omit the multi-GPU error.

Slow training should be unrelated to this though. In my experiments it seemed to be I/O bound. Can you provide more information about the hardware resources and training config?

from microdl.

I have reserved an OnDemand session on gpu-c-1 with one gpu, with 32 nodes, 20GB memory per node. The training config I am using is here:
/local/scratch/groups/cmanalysis.grp/microDL_SP/config/lit_test/HEK_dNucl_2023_04_21_15.yml
It seems to have trained for 70 epochs over a day.

I am also not seeing any logging from the training. The default_root_dir is set to this folder on the config:
/local/scratch/groups/cmanalysis.grp/microDL_SP/logs/lit_test/. Is there any other parameter that needs to be changed on my config to start logging? Or is it not able to log on scratch space?

from microdl.