Comments (6)
trengrj
commented on May 23, 2024
Hi @zoltan-fedor we have found this is caused by #4643.
We will be releasing a new patch release soon to fix. Workaround would be to downgrade to 1.24.8.
from weaviate.
trengrj
commented on May 23, 2024
Thanks for your report, we have fixed the issue in #4721. Please upgrade to 1.24.10.
from weaviate.
zoltan-fedor
commented on May 23, 2024
Thanks for quickly fixing it!
from weaviate.
zoltan-fedor
commented on May 23, 2024
@trengrj
While initially this fixed looked good, this same slowdown issue with lots of lsm_compaction log messages came back once the Weaviate nodes reached their peak allocated memory.
Once downgraded to v1.24.8, it works fine even when approaching peak allocated memory.
from weaviate.
trengrj
commented on May 23, 2024
Hi @zoltan-fedor, can you confirm the lsm_compaction log messages said skipping compaction due to memory pressure?
from weaviate.
zoltan-fedor
commented on May 23, 2024
@trengrj ,
No, I wasn't getting the skipping compaction due to memory pressure message, but instead the following messages and all read operations against the Weaviate nodes slowed down tremendously.
This was v1.24.10 once the memory utilization of the Weaviate nodes have reached 90-95+%
With v1.24.9 the same thing was happening even without the Weaviate nodes reaching 90-95% memory utilization.
2024-04-23 16:52:37.140
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":5145,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:37Z"}
2024-04-23 16:52:37.071
{"action":"lsm_compaction","class":"C20240423185840","compaction":7352,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"SUPSd819ycbw","time":"2024-04-23T20:52:37Z"}
2024-04-23 16:52:37.071
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":2482,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"SUPSd819ycbw","time":"2024-04-23T20:52:37Z"}
2024-04-23 16:52:36.961
{"action":"lsm_compaction","class":"C20240423185840","compaction":2482,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"SUPSd819ycbw","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.940
{"action":"lsm_compaction","class":"C20240423185840","compaction":5145,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.916
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":2778,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.672
{"action":"lsm_compaction","class":"C20240423185840","compaction":2778,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.672
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":4543,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.509
{"action":"lsm_compaction","class":"C20240423185840","compaction":4543,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.509
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":6354,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.326
{"action":"lsm_compaction","class":"C20240423185840","compaction":6354,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.326
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":1332,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.215
{"action":"lsm_compaction","class":"C20240423185840","compaction":6565,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"quTOWa12kqnd","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.214
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":8867,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"quTOWa12kqnd","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.046
{"action":"lsm_compaction","class":"C20240423185840","compaction":1332,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.046
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":9748,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"Gml41SUV7mGh","time":"2024-04-23T20:52:36Z"}
2024-04-23 16:52:36.010
{"action":"lsm_compaction_finished","class":"C20240423185840","compaction":7592,"index":"C20240423185840","level":"info","msg":"start lsm compaction","shard":"fhDBmYIx4jbj","time":"2024-04-23T20:52:36Z"}
from weaviate.
Related Issues (20)
- [v1.24] Tx can get stuck on some nodes when tx commit fails
- Better tenant observability
- `with_near_vector` function does not work HOT 2
- `X-Azure-Api-Key` doesn't work in headers. Asking openai api key instead. while calling withNearText search. HOT 3
- Windows is not supported with Embedded
- Multivector: knn search: distance between entrypoint and query node: got a nil or zero-length vector at docID 1115 HOT 5
- Duplicate and inconsistent results of BM25 search
- Improvements to geo-coordinate based filtering HOT 5
- explainScore containing only one score when performing hybrid search HOT 2
- Weaviate backup using S3 closed pipe http 499 after 1m 3s
- Update by Query
- Duplicate CONTEXTIONARY_URL in dev setup HOT 1
- Data encryption in weaviate for securing sensitive information
- text2vec-ollama is not working HOT 2
- Observability metrics for async indexing queue
- Existing replication factor increase implementation does not work with raft
- High variance of latency when doing a benchmark of query (GET) by id
- [proposal] Vectorizer select and exclude properties – in one place
- Hybrid should skip properties with indexSearchable=False HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from weaviate.