ValueError when using an array to index about tensorstore HOT 8 CLOSED

I have a fix for this bug, hopefully can push it out later today.

from tensorstore.

Thanks very much --- I can reproduce from your example. Hopefully I can get to the bottom of it quickly now.

from tensorstore.

Thanks for reporting this. I will have to look into this --- certainly if you are able to provide me a way to reproduce, that would help.

I can explain the meaning of the error, at least: the error is not referring to co[28], it is just indicating that the index array contains the value 28, which is outside the valid range [28928, 29056). The way it works is that when you slice a TensorStore object like vol[co[:, 0], co[:, 1], co[:, 2]], you end up creating an IndexTransform that holds the index arrays. Tensorstore does not validate index arrays immediately, instead it validates them only when necessary (because the correct bounds are not necessarily known when the TensorStore is created). But the IndexTransform keeps track of the relevant bounds that apply to the index values, initially the full bounds of the volume. The range [28928, 29056) that you see in the error message is a result of the way the grid partitioning works: for a chunked format like zarr, tensorstore performs the read by partitioning the full index transform for the read request by the grid, such that it ends up with a separate index transform for each grid cell. Each of those separate index transforms is used to copy data from the decoded chunk into the output array, and that copy process validates the index values against the bounds of the individual chunk. However, the partitioning should have already ensured that an index of 28 would not end up in a chunk for the range [28928, 29056). The fact that it occurred suggests some sort of memory corruption bug in tensorstore. You can check if your co index array even contained 28 in the first place --- if it does not, that would be even more indication of memory corruption.

Normally when debugging an issue like this I would attempt to reproduce with the python extension compiled with AddressSanitizer. I haven't attempted to do that with the open source tensorstore build yet, though. It seems like in principle it should be possible, though: https://stackoverflow.com/questions/55692357/address-sanitizer-on-a-python-extension

from tensorstore.

If you can say more about the conditions within your application that you are able to reproduce it, that may be helpful as well. E.g. if you break in pdb right when the error occurs, and you repeat with the same co array, does the error occur?

from tensorstore.

It's a web service, and by the time the code gets to this line the input values have been spatially binned and several workers have been spawned. It's reproducible in that I have a payload that will trigger it 100% of the time.

Thank you for the explanation for the IndexTransform. It validates the approach of clustering in space before doing the lookups.

I'll continue work on a minimal example tomorrow, e.g. pickling the inputs to the function in question.

E.g. if you break in pdb right when the error occurs, and you repeat with the same co array, does the error occur?

The immediate second attempt has the exact same error.

from tensorstore.

Thanks, I'm working on reproducing as well. Can you also confirm which version you are using?

from tensorstore.

I have a minimal example that breaks, but it needs local access to the data (does not work on a new, empty, array). I'll work on creating a minimal zarr now -- unless there is an http driver I am unaware of?

tensorstore==0.1.8

from tensorstore.

I have an example with zarr here.

This is reproducible on my Mac as well:
ValueError: In index array map for output dimension 0: Index 0 is outside valid range [28928, 29056)

from tensorstore.