Comments (5)
Works with num_workers=0
from predict_pv_yield.
works with num_workers=2 and with NWPDataLoader being passed into SatelliteDataset
from predict_pv_yield.
It's working with gcsfs 2021.4.0! See fsspec/gcsfs#379 (comment)
from predict_pv_yield.
Still TODO: Try reverting:
- create a
gcs
object separately every timeopen_zarr_on_gcp
is run. CONCLUSION: Creating a separate gcs object is fine! - don't
skip_instance_cache
. CONCLUSION: Seems fine not toskip_instance_cache
, and is maybe a little faster. - create Zarr stores once in main process and copy it into worker processes . CONCLUSION: fsspec throws an error saying "this is not thread safe". So need to load separately in each process. That's fine.
from predict_pv_yield.
Fixed! Some main conclusions:
1. fsspec.asyn
gcsfs 2021.4.0 (the latest version at the time of writing) works fine with PyTorch multi-process loading iff the code includes this (see this discussion for a little more context). The two lines below need to be run in every worker process (it's not sufficient to run them once in the main process):
fsspec.asyn.iothread[0] = None
fsspec.asyn.loop[0] = None
2. Still need to read Zarr files in each process
Using gcsfs 2021.4.1, it's not possible to pass the zarr
stores into each process. If you attempt to pass the zarr
store into each process then fsspec throws an error saying fsspec is not thread safe.
from predict_pv_yield.
Related Issues (20)
- When running unittests: RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
- New paper: "Improving Vision Transformer Efficiency and Accuracy by Learning to Tokenize"
- Try nvidia Tensor RT: Claims up to 6x speedup of pytorch with single line of code
- [Meta] Train simple nwp model HOT 4
- Use Weights and Biases
- Trouble training HOT 8
- Move PVnet to separate repo
- huggingface model name for OCF OptiFlow model HOT 2
- GCS 404
- The files downloaded from the openclimatefix database are throwing a "blosc2" error. HOT 1
- Extend National Electricity Demand forecast
- Format docstrings so they pass `pydocstyle`'s criteria HOT 8
- Use Nowcasting Uilts
- Update to use FakeDataset
- NWP in Perceiver HOT 1
- Stop giving the ML model satellite images of the _future_! HOT 2
- Try without historical GSP PV HOT 3
- Update to new dataset and dataloder
- Experiment with ensemble of models
- No future satellite for conv3d
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from predict_pv_yield.