Distributed data parallel training about torch-ngp HOT 8 CLOSED

@xmk2222 there is a pure-pytorch implementation here.

from torch-ngp.

NVlabs/tiny-cuda-nn#63

Sadly, the --tcnn does not support multi-GPU training for sure.

from torch-ngp.

Yes, multi-GPU training for both CMD and GUI mode has not been supported now (the local_rank in trainer is not tested in this repo since it was borrowed from old projects).
In most cases the training speed should be fast enough on one GPU, may I know your using case for multi-GPU?

from torch-ngp.

(coming from the linked issue above)

I've been evaluating torch-ngp as a way to scale NeRF models as my dataset (750 4K images) no longer fits in any available GPU's VRAM -- only about 300 frames fit on a 40 GB A100. I saw there appeared to be some changes to only load portions of the training data at the time, could that support such a use-case?

from torch-ngp.

@slash-under that's strange, dataset size should not affect the runtime GPU memory, since it is only load one image at a time by default. Are you using the latest version, and not using the flat --preload for NeRFDataset?

from torch-ngp.

Thanks! Just looking for ways to speed up training, the training time of a single GPU is bearable.

Well, I'm still wondering if it is possible to implement everything in python including the hash-encoding? I think that would enable the training process to be distributed on multiple GPUs. Or perhaps the speedup gained by parallel computing would be cancelled out by the slowdown caused by the python implementation...

from torch-ngp.

hey @slash-under - apologies for tooting my own horn a bit, but I've run into similar issues where I've wanted to train NeRFs against thousands of high-resolution images. The best I could come up with is writing randomized batches of rgbs and rays across all of my images to disk as a preprocessing step, and then loading those from disk in chunks. You can take a look at https://github.com/cmusatyalab/mega-nerf/blob/main/mega_nerf/datasets/filesystem_dataset.py for inspiration and would be happy to hear of better suggestions.

from torch-ngp.

closed for now.

from torch-ngp.