I think this will be a tougher issue to solve than typos, math bugs, and formatting; so it will warrant some conversation with @thewtex and @tbirdso.
I have ngff-zarr running well on small in-memory sized datasets (e.g. 512x512x512 voxels). When formatted with the nested directory style dimension separator these files open just fine with the napari-ome-zarr plugin (just my first check for compatibility with other tools - I'll use other visualization software later as well). So my next task was to run on real data.
My real image is {'z': 1596, 'y': 8983, 'x': 5451} voxels and I am attempting the following:
ngff_image = to_ngff_image(
my_big_zarr_array,
dims=('z', 'y', 'x'),
scale={a:b for a, b in zip('zyx', my_voxel_spacing)},
axes_units={a:'micrometer' for a in 'zyx'},
)
multiscales = to_multiscales(
ngff_image,
scale_factors=[{'z':4, 'y':8, 'x':8}],
chunks=(64, 128, 128),
)
to_ngff_zarr(
'./my_data_as_ome_ngff_zarr.zarr',
'multiscales,
)Importantly, this does actually work. If I wait long enough I get the file I want. However there is a performance issue that IMO will be lethal for the package if not resolved. For this (4, 8, 8) downsample level, the task graph has about 2.5M tasks. The dask distributed scheduler really can't handle graphs of this size well, which it acknowledges with a warning that the task graph itself is about 220MB when compressed, which it knows is huge, and says it will cause long slow downs.
It takes about 45 minutes for the scheduler to finish parsing all those tasks (just about right with the 2.5M tasks and the documented 0.001 seconds per task), and during this time the workers are allocated but idle. And this has to happen for every scale level. In a real scenario I might have 10 or more scale levels I want to create. So that's a huge amount of time to be paying for cpus that are just waiting for the scheduler to finish parsing the task graph. Making matters worse, this is actually only a small to medium size image. We could be running this on things that are much much bigger.
I feel that there are three different options for approaching this problem:
(1) small hacks, maybe something will help
(2) optimize the task graph by reworking the series of dask array operations to simplify simplify simplify
(3) abandon dask in as many places as possible
I tried three different things for (1):
(a) ran dask.array.optimize on the big graph (only eliminated about 10K tasks and at the expense of running this function which was long)
(b) forced even in-memory size scale levels to run through the serialization stuff (resulted in 6 task graphs each about 500K tasks, so the total time when breaking the problem up was even longer)
(c) forced every scale to run through this "Minimize task graph depth" block (just resulted in the same graph as before)
So now I want to decide between options (2) and (3). I'd love for there to be a solution with (2). It would be great to stay consistent with the sort of intended dask workflow and also to not have to refactor your package too much. But while building my package bigstream, and running into the same kind of huge task graph problems, I ultimately settled on (3) and everything got really performant really fast. After all, we're just doing a bunch of smoothing and downsampling here. It's not rocket science. Having all of that be consistent with dask.array and coordinated (or micromanaged) by a scheduler is nice, but not if it turns the program into a bureaucracy.
So, sorry for the long message, but I'd love to discuss this with you guys to determine where to go next.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent