Comments (4)
Parallelization didn't seem to work as well as sequential inserts with a large batch size. This feels counter intuitive to me so curious if this is to be expected
Writing in batches is much more efficient in LanceDB. You could write batches of 10-100k in parallel, and that might work well. But writing <1k rows in parallel will perform poorly and produce a bad table layout, which would need to be fixed by calling compact_files.
from lancedb.
Hi @aravindsrinivasan, thanks for trying out LanceDB.
The methods you are using to insert all seem reasonable. I don't think you are using LanceDB in a way that would make it slow.
I think the most productive thing you could do here is quantify how fast you are able to write on Azure with LanceDB vs some other library. For example, you can time how long it takes to insert 100,000 vectors. Then if you vectors are say 1024 dimensions, then you can estimate the data you wrote is about 100,000 * 1024 dim * 4 bytes ~= 400MB
. (If there are other columns, you will have to account for the size of those as well.) Divide that by the number of seconds it took and you get an estimate of how much throughput you are getting from writing with LanceDB. For comparison, it would be useful to know how slow that is compared to using the Azure CLI to upload a file directly (like this).
from lancedb.
@wjones127 thank you for the response. Turns out my internet was the culprit -- plugging in the ethernet cable made it 10x faster.
Generally speaking, what has your team found to be the fastest way to upload into an index? Parallelization didn't seem to work as well as sequential inserts with a large batch size. This feels counter intuitive to me so curious if this is to be expected
from lancedb.
Thanks @wjones127.
from lancedb.
Related Issues (20)
- bug(node/vectordb): schema mismatch when using custom embedding function HOT 4
- bug(python): Allow setting `allow_remote_code` in HF embedding function
- Clarity and Understanding around Prefilter performance and Schema design HOT 2
- Feature: explain_plan HOT 2
- bug(node): unable to 'add' to a table created from python
- bug(python): offset overflow when issing table update HOT 4
- bug(node, vectordb): error when using `Float64` embedding function HOT 1
- bug(node, lancedb): unable to 'add' to a table created with a `Float64` vector HOT 1
- Document cohere embedding function
- Fix semvar deprication warning HOT 1
- Langchain Docs HOT 1
- LlamaIndex Docs
- bug(python): table.merge_insert does not accept a list of columns HOT 1
- bug(python): LanceDB on Cloud Run (GCP) using GCS bucket mount - Generic LocalFileSystem error HOT 4
- Feature(python): LanceDB python layer for AWS Lambda HOT 4
- bug(python): Creating index for float16 vectors takes significantly longer time than float32 vector HOT 2
- create_fts_index doc missing
- Enable stemming and choosing tokenizer, when doing full text search in tantivy HOT 1
- bug(python): Reranker pyarrow compatibility update
- bug(python): null values do not preserve after write and read
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lancedb.