DDP strategy doesn't work for on_validation_epoch_end, always hang about pytorch-lightning HOT 4 OPEN

@jzhanghzau self.log() issues collective calls, so you can't just call it on "rank zero" only. If you want to do that, pass self.log(rank_zero_only=True).

Here are the relevant docs for this:
https://lightning.ai/docs/pytorch/stable/visualize/logging_advanced.html#rank-zero-only

from pytorch-lightning.

@jzhanghzau self.log() issues collective calls, so you can't just call it on "rank zero" only. If you want to do that, pass self.log(rank_zero_only=True).

Here are the relevant docs for this: https://lightning.ai/docs/pytorch/stable/visualize/logging_advanced.html#rank-zero-only

Thanks for your quick reply! if i setself.log(rank_zero_only=True), it seems like that i am not allowed to use callback, what if i still want to use callback(earylstopping), how can i organize my code? remove if self.trainer.is_global_zero: and let every process go through the logic what i present above ?
Thanks again.

from pytorch-lightning.

@jzhanghzau self.log() issues collective calls, so you can't just call it on "rank zero" only. If you want to do that, pass self.log(rank_zero_only=True).
Here are the relevant docs for this: https://lightning.ai/docs/pytorch/stable/visualize/logging_advanced.html#rank-zero-only

Thanks for your quick reply! if i setself.log(rank_zero_only=True), it seems like that i am not allowed to use callback, what if i still want to use callback(earylstopping), how can i organize my code? remove if self.trainer.is_global_zero: and let every process go through the logic what i present above ? Thanks again.

@awaelchli

from pytorch-lightning.

Yes, that would be another option.

from pytorch-lightning.