[Bug]: Querynode memory usage is very unbalanced, some report memory quota exceeded. about milvus HOT 14 OPEN

According to these issues:
#32714
#32894

It seems that this scene is caused by the high writing throughput, thus some datanodes became unhealthy, so there will be too much growing segments in some querynode. (But why the already sealed segment loaded num is decreased? I am not very clearly about what happened inside).

I tried to re-deploy the querynodes and re-load the collection, the memory usage becomes back to normal:

Hope there will be some helpful info.

• Should I scale up the datanodes?
• And how to control the writing rate or concurrency for large scale data scene?

from milvus.

@tianshihan818 thank you for the issue and updates. It seems that something stuck in that querynode, but we cannot tell without milvus logs. Could you please refer this doc to export the whole Milvus logs for investigation?
BTW, which index type are you running?

/assign @tianshihan818

from milvus.

@yanliang567 Thanks! Here are the logs:
milvus-log.tar.gz

And I use IVF_SQ8 index type:

from milvus.

/assign @congqixia
please help to take a look.

from milvus.

Additional info:
The initial logs are lost, so I reproduce this test(concurrent insert), and get the above logs, i.e.
milvus-log.tar.gz

This time the insert API first report error at June 17th 22:50: pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=message send timeout: TimeoutError)> （The time on the machine nodes is set to be 8 hours later, so in logs should be around 14:50.)
And I checked the metrics and pods list during this test period.

• Pod list

• Memory

• Segement Loaded Num

• Queryable Entity Num

• Goroutines Total

• Goroutines

• CPU

• OS Threads

• GC Max duration seconds

• Other metrics

from milvus.

And I continue inserting, the memory usage of querynode miltest-milvus-querynode-5559dc65d-knhn7 is increasing, when it reached the quota water level, will report the same error as before: pymilvus.exceptions.MilvusException: <MilvusException: (code=9, message=quota exceeded[reason=memory quota exceeded, please allocate more resources])>

And the logs of this time are here:
milvus-log-new.tar.gz

from milvus.

you can try to increase shard number.

In our design, best practice is to have 100 - 200 million data for each shard.

On your case, you have roughly 1B data, we would recoomend to use 8 shards.

The reason you see one query nodes cpu very high is because it is the shard delegator. If index build can not catch up, data may accumulate. also since there is only delegator forward request, you may see it becomes the bottleneck on heavy search.

If you can use pprof to check why the memory is high it could be also helpful

from milvus.

@tianshihan818
The segment is still been loaded into the delegator querynode, which is not by design
could you please provided a birdwatcher backup and output for show segment-loaded-grpc command?
https://github.com/milvus-io/birdwatcher/releases/tag/v1.0.4

from milvus.

Thank for your reply! @xiaofan-luan @congqixia

I use birdwatcher to get pprof heap files:
bw_pprof_heap.240619-045745.tar.gz

And the segment-loaded-grpc list results are in this file:
show_segment_loaded_grpc.log

from milvus.

@tianshihan818 thanks for the quick reply, could you please provided the backup file as well?

from milvus.

@congqixia The etcd backup is here:
bw_etcd_ALL.240619-060804.bak.gz

from milvus.

DataNode painiked on konwn bugs, should be fixed by #33829

from milvus.

@xiaofan-luan Hello! Thanks for sharing the tech detail. I verified in the logs, the "delegator" is indeed on the abnormal querynode. Is this "delegator" responsiable for forwarding the loading or searching request and gathering the search results?
So under the concurrent insert case, once called collection.load(), the querynode will load data later automatically, as my collection's consistency_level is set to be "Strong", if the index build process (use IVF_SQ8 index type in this test, ideally the memory usage of the index files will be around 30% of the raw data size) don't catch up the insert speed, the querynode will load the raw data instead of the index files into the memory, thus causing the high memory usage, right?
I am still not very clearly on these things:

• Why only the "delegator" querynode itself bear the load pressure in this case?
• Will it better for "load all data after insert all data"?

from milvus.

delegator hold all the growing segment data. when segment is sealed and index is built, all other node can take the work .

@XuanYang-cn upgrade to 2.4.5 could solve the problem? do we need special fix for this segment not found error?

from milvus.