DataLoader2 Memory Behavior is very strange on Epoch Resets about data HOT 9 OPEN

@ejguan: Do you have any suggestions for properly resetting Dataloader 2 after each epoch? With e.g. worker_reset_fn.

from data.

Hello, I have also encountered a situation where the DL2 memory usage has skyrocketed. I have temporarily decided to switch back to DL1. May I ask how to set up datapipe+DL1 for multi process and multi card training? Do I need to set up distributed sampling in DL1?

from data.

@Adenialzz: To get what I showed above, it's more or less the same setup as for the a torch dataset. Replace the dataset with a datapipe.

sampler = DistributedSampler(datapipe) if distributed else None
return DataLoader(datapipe, sampler=sampler, num_workers=num_workers, pin_memory=pin_memory, batch_size=batch_size)

from data.

This DistributedSampler requires my dataset(datapipe) must have len method, but the length of my datapipe cannot be calculated cause it is a iterable datapipe. Have you ever met problem like this?

from data.

I'll give it a try today.

from data.

Thanks, please let me know when you make progress.

from data.

Sorry for the delay @Adenialzz. You are correct that it doesn't work with DDP and without a length on an iterable data pipe. I reverted to DL2 despite its notably slower performance as it only really occurs at the start of the epoch.

from data.

set torch.utils.data.graph_settings.apply_sharding(datapipe, world_size, rank) seems to solve the problem in my case.

from data.

@Adenialzz hi, could i ask you for a clarification? how was it used to fixed which problem exactly? i'd appreciate it very much.

from data.