Loading slows down with large dataset about nowcasting_dataset HOT 2 CLOSED

Code for looking at speeds to read data:

import gcsfs
from pathlib import Path

PATH = Path('solar-pv-nowcasting-data/satellite/EUMETSAT/SEVIRI_RSS/OSGB36/all_zarr_int16_single_timestep.zarr/')

fs = gcsfs.GCSFileSystem(access='read_only')

%%time
with fs.open(PATH / 'stacked_eumetsat_data/0.0.0.0') as f:
    data = f.read()

len(data)

%%time
fs.glob(str(PATH / 'stacked_eumetsat_data/0.0.0.*'))

from nowcasting_dataset.

Actually, this is fixable without re-creating the Zarr! The trick is to open the Zarr file without dask by doing xr.open_dataset(filename, engine='zarr', chunks=None). Then use dask.delayed to construct our own graph. Gets 30 it/s again with full Zarr dataset (chunks: time=1). Compared to about 5it/s with the original zarr (with 36 timesteps per chunk)

from nowcasting_dataset.