Comments (14)
Got idea about the batching:
One could accumulate the statistics of the batched documents in the doc_frequencies, in this case idf calculation would have to wait.
# 1)
index_corpus(batch_0) <- inits and calculates doc_frequencies, etc.. but not idfs
index_corpus(batch_1) <- updates
index_corpus.flush() <- calculates idfs etc.
# 2)
index_corpus(batch_0 + batch_1)
index_corpus.flush()
1 and 2 would be the same.
Again depends if it makes sense for the use case.
from bm25s.
I think it would make sense to make it into a util class, e.g.:
idfs = {}
tfcs = {}
for batch in batches:
idfs = bm25s.utils.corpus.update_idfs(idfs, batch)
tfcs = = bm25s.utils.corpus.update_tfcs(tfcs, batch)
Then, we could have a method of bm25
that allows manually specifying idfs and tfcs:
retriever.index_precomputed(idfs=idfs, tfcs=tfcs)
If that method works well (passes new unittests, as fast as original, does not increase memory consumption), then we can rewrite BM25.index
to use that new index_precomputed
method by default.
from bm25s.
Unfortunately it is not possible! The reason for that is that BM25 calculates the individual score based on the term frequency (tf), number of documents (|D|) and average document length (L_avg); thus, any new document added will change the information previously calculated.
As for a multi-stage approach (so you can remove the data from memory), I've thought about how to deal with memory with incremental passes and chunked indexing, but unfortunately I did not have the chance to come up with better ideas that would be straightforward to implement and use. I think it's good to keep this issue open if someone has a suggestion & potential PR.
from bm25s.
one major hurdle is that we need multiple pass on the corpus tokens (once to get term freq, and average length, a second for actual scoring). Although it might be possible to use memmap
to store tokens to file and read from disk, that'd make the indexing slower, and the overall code more complex. if you are willing to try, feel free to create a PR and I'm glad to review to make sure it's consistent with tthe rest of the API (proper new tests would be needed to avoid regressions).
from bm25s.
That said, if your goal is to reduce memory usage, you can also consider passing an iterator to the tokenizer instead of a list in memory, some pseudo code:
def load_corpus(fname):
with open(fname) as f:
for line in f:
yield json.loads(line)['text']
corpus_tokens = bm25s.tokenize(load_corpus(fname), stemmer=stemmer)
from bm25s.
@xhluca Great job! You obviously put a lot of time in this!
About the update method - look at my implementation..
I will be glad to spend some time and see if I can bring this to your implementation.
# ........
def index_corpus(corpus_batch):
corpus_tokens = bm25s.tokenize([d['text'] for d in corpus_batch], stopwords="en", stemmer=stemmer)
retriever.update(corpus_tokens) #udpate given the current state -> change to new state as if you would do .index(full_corpus)
#.......
# 1)
index_corpus(batch_0) <- inits and calculates idf etc..
index_corpus(batch_1) <- updated and calculates new idf etc...
# 2)
index_corpus(batch_0 + batch_1)
(1) and (2) give the same index.
from bm25s.
Hey that'd be an interesting addition! I'd be happy to review a PR with the incremental feature, docstrings and tests for the incremental update part.
That said, one reason I have not touched it is due to how BM25 is calculated, which is not directly compatible with precomputed BM25 scores. For example, the term frequency, average length and document count changes after an incremental update. Do you have an idea how to update existing pre-computed scores in this case?
from bm25s.
look at my code in the link.
But in some sense you compute again the idf scores with the updated information. Also you have updated document-word frequencies and you can score any query or document.
I will get to it at some point wont be in the next 2-3 weeks though, sorry.
from bm25s.
No worries, take your time!
But in some sense you compute again the idf scores with the updated information.
But wouldn't that be not very efficient if you process by batch? For example, if you have 100M docs, and you want to update by adding 10; it'd be very fast to update a term frequency dictionary and update the average doc length. However, updating the IDF calculation of all the 100M docs would take a lot more time since you can't increment the values, instead you need to recalculate the idf and tf component. Then, repeating that for each batch would simply make it very computationally inefficient (and would also need to keep everything in memory... which defeats the purpose of batched updates).
from bm25s.
Well yes, but this is the price to pay if you want to batch... I dont have solution otherwise at this moment. Its either you can do it by batch save the state and get rid of those documents or you need to retrain every time you have new batch documents on the whole corpus anyway.
from bm25s.
Thanks. I've been thinking about this but I was hoping I missed some solution...
In this case i guess index updates (i.e. for batching) would not be possible, but moving forward, I think it will be nice to eventually support low-memory tokenizing (i.e. a streaming tokenizer that yield
s the tokens as they are being iterated rather than returning them all at once) and sharded indexing (by calculate term frequency, avg doc len and num docs separately, and then shard the indexes which are saved as partial_index-{i}-of-{N}
)
from bm25s.
Sure, batching updates are not possible by incrementing the idf values. Still in my use case I don't mind recalculating everything but with only new documents as the old ones are already archived, its a trade off. What happens if you need to add 100 new documents to the index? You need to have them all at one place..
I can contribute for the other features a bit later. I think you can close this one in this case.
from bm25s.
Makes sense! I'll close now. Thank you!
from bm25s.
Yeah makes sense :)
from bm25s.
Related Issues (15)
- Minor bug: `show_progress` not propagated in `BM25.index` HOT 1
- Not Working for langchain Documents
- [Feature Request] Support attaching metadata to the corpus HOT 3
- how to dynamic add/delete documents
- Can you query without a tokenization step?
- 可以增量更新索引吗?
- 🚨Before submitting an issue, read this 🚨
- On-the-fly stemming HOT 1
- Other language than english for the stopwords list HOT 2
- How to apply bm25s to languages such as Chinese? HOT 2
- Order-based matching of corpus metadata to to tokens HOT 2
- Using with postgres? HOT 3
- Capability Inquiry: Retrieving Specific JSON Records Based on Text HOT 4
- Pre-computed TF-IDF
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from bm25s.