Comments (1)
Tried it. Doesn't seem to speed things up much (if at all). But I'll leave it in the code, just in case it speeds things up a bit!
Using consolidated metadata seems to significantly (double?) the read speed though.
Code:
# Pretty sure this sets partial_decompress to True:
import zarr
array = zarr.open_array(
FILENAME, path='stacked_eumetsat_data', partial_decompress=False, mode='r'
storage_options=dict(consolidated=True))
%%time
data = array[:12, :128, :128, 0] # takes about 35 ms with or without partial_decompress.
This code might set partial_decompress=True
, but I'm not certain!
ds = xr.open_dataset(
FILENAME,
engine='zarr',
consolidated=True,
storage_options=dict(partial_decompress=True)
)
from nowcasting_dataset.
Related Issues (20)
- Pad Satellite images if requested size goes out of range
- Satellite timestamps must be rounded HOT 1
- Bug: `drop_pv_systems_which_produce_overnight` expects normalised data, but it doesn't receive normalised data. HOT 3
- PV power warning
- Read Live GSP from database
- GSP locations might have changed since v15 was created HOT 2
- AYSNC load bug HOT 1
- Make sun data on the go
- PV interpoalte excection
- Load 15 mins satellite data if no satellite data
- PV internal ID (metadata) HOT 3
- Which Capacity should we use? HOT 8
- `hrv_satellite_fake` should return the requested `image_size_pixels`, not `3 x image_size_pixels` HOT 2
- Fake `hrvsatellite` and `satellite` batches need 2D osgb coords HOT 2
- Fake hrvsatellite and satellite batches need geostationary coords HOT 1
- metadata_model.py:121: FutureWarning: The frame.append method is deprecated and will be removed from pandas in a future version. Use pandas.concat instead.
- Update gsp boundaries and metadata
- Errors on generating GSP batches because of the national forecast is GSP ID 0
- Out of date link HOT 1
- extend Readme.md for making DEM.tif HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from nowcasting_dataset.