Comments (4)
jbms
commented on May 30, 2024
3
zarr vs n5 performance will be very similar. In general I would recommend zarr over n5 since zarr is becoming more standard and offers a more functionality.
Compared to zarr-python, tensorstore is better able to take advantage of multiple CPU cores, and multiple concurrent IO operations, without relying on additional layers like dask for parallelism. In general with zarr-python it is usually necessary to use very large chunk sizes to get reasonable performance; that is not generally the case with tensorstore.
Of course actual benchmarks would be better than these anecdotes, but we haven't run anything systematic yet.
from tensorstore.
harpone
commented on May 30, 2024
+1 for this. Also would be nice to have something about pros/cons of e.g. zarr vs n5 drivers...
from tensorstore.
Axel-Jacobsen
commented on May 30, 2024
+1 - I am curious what performance impact one could expect
from tensorstore.
JackKelly
commented on May 30, 2024
This is helpful, thank you!
from tensorstore.
Related Issues (20)
- Loosen ml-dtypes dependency HOT 4
- How cant install tensorstore en ppc64 HOT 1
- AWS S3 authentication issues HOT 6
- Please create a pre-built PyPI wheel for linux arm64 HOT 4
- Support of large files for grpc_kvstore HOT 1
- S3 kvstore driver not being recognized HOT 3
- downsample driver produces repeating patterns HOT 8
- Debug logs for S3 driver, log requests and responses HOT 9
- Cache pool context from multiple python processes HOT 2
- Error while installing "tensorstore" on jetson linux (jetpack) HOT 1
- Assertion `IsUnmodified()' failed HOT 6
- How to use a custom bazel installation? HOT 2
- Building on linux with ppc64le arch
- Specify S3 credentials directly HOT 3
- Transactional/ACID semantics HOT 1
- Failing to build tensorestore as a cmake project HOT 4
- Using s3 kvstore with minio HOT 1
- Error reading shard index, Requested byte range... was not satisfied by response with byte range ... HOT 4
- Unable to include tensorstore as a cmake dependency
- Question: does tensorstore support array with multiple dynamic dimensions? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tensorstore.