Cache pool context from multiple python processes about tensorstore HOT 2 CLOSED

Regardless of whether you use fork or spawn, it is not possible for caches to be shared across multiple processes. Instead, each process will have an independent cache with the same configuration (i.e. total_bytes_limit).

It is unfortunate that the PyTorch data loader does not offer a way to use multiple threads rather than multiple processes --- ultimately that would be the best solution when using libraries like tensorstore that internally release the Python Global Interpreter Lock (GIL), and it seems to have been asked a number of times on the PyTorch discussion forum:

https://discuss.pytorch.org/t/dataloader-using-python-thread-as-opposed-to-multiprocess/86565

It might be worth looking into the PyTorch source code to see what would be required to allow DataLoader to use multiple threads and/or allow data loading to be done asynchronously. I expect that may ultimately be the easiest solution.

A hacky alternative would be something along the lines of the following:

• You start a single additional process (possibly using pytorch's multiprocessing facilities) that will use tensorstore to load data.
• Someone have each process started by PyTorch data loader forward its requests to this other process

from tensorstore.

@jbms Thanks for the answer! I will look into rolling my own cache implementation then.

from tensorstore.