PyTorch Lightning example not working about optuna-examples HOT 5 CLOSED

When using TorchMetrics (written by the same developers are PyTorch Lightning), the metrics are automatically distributed between devices during steps, and collected at the end of each step and epoch. These are made available in the hooks for those events, see TorchMetrics in PyTorch Lightning. I don't know why they don't populate the same trainer.callback_metrics dictionary in either case, might be an error on their part, but perhaps it could be solved by writing monitored metrics to a property of the model?

from optuna-examples.

Closed because I noticed it's a duplicate of optuna/optuna#1417

from optuna-examples.

Let me summarise the points -- thank you @jacanchaplais for pointing out them.

1. The multi-GPUs training does not work due to this error -- I suppose rank 0 has only the attribute val_acc and the others do not. If a machine has multi-GPUs, pytorch_lightining_simple.py uses all GPUs according to this line, so this error happens always on a machine with multi-GPUs.
2. Are there any issues with PyTorch-lightning with DDP training? If so, it would be great to mention it as comments in the example code or the callback page.

from optuna-examples.

I suppose rank 0 has only the attribute val_acc and the others do not.

Also, I don't think the issue is due to which device the metrics are stored to, as the line which tries to access trainer.callback_metrics['val_acc'] is after the trainer.fit() method has completed execution, so I would have thought that the data would not be partitioned to specific devices (although I am new to GPU parallelism, so let me know if I'm wrong).

from optuna-examples.

Submitted failure to populate callback_metrics as an issue to PyTorch Lightning, as I implemented with Ray[Tune] and ran into the same problem.

from optuna-examples.