Fix crash with gcsfs about predict_pv_yield HOT 5 CLOSED

Works with num_workers=0

d0f673f

from predict_pv_yield.

5dbfa6c

works with num_workers=2 and with NWPDataLoader being passed into SatelliteDataset

from predict_pv_yield.

It's working with gcsfs 2021.4.0! See fsspec/gcsfs#379 (comment)

from predict_pv_yield.

Still TODO: Try reverting:

• create a gcs object separately every time open_zarr_on_gcp is run. CONCLUSION: Creating a separate gcs object is fine!
• don't skip_instance_cache. CONCLUSION: Seems fine not to skip_instance_cache, and is maybe a little faster.
• create Zarr stores once in main process and copy it into worker processes . CONCLUSION: fsspec throws an error saying "this is not thread safe". So need to load separately in each process. That's fine.

from predict_pv_yield.

Fixed! Some main conclusions:

1. fsspec.asyn

gcsfs 2021.4.0 (the latest version at the time of writing) works fine with PyTorch multi-process loading iff the code includes this (see this discussion for a little more context). The two lines below need to be run in every worker process (it's not sufficient to run them once in the main process):

fsspec.asyn.iothread[0] = None
fsspec.asyn.loop[0] = None

2. Still need to read Zarr files in each process

Using gcsfs 2021.4.1, it's not possible to pass the zarr stores into each process. If you attempt to pass the zarr store into each process then fsspec throws an error saying fsspec is not thread safe.

from predict_pv_yield.