[Bug]: Load a very large collection, the memory suddenly drops! about milvus HOT 4 OPEN

@rh01 As the error msg says, your total memory is ~131072MB, and the thresholdFactor is 0.9, so the maximum memory milvus can use for loading collection is 117964.8, which is larger than your collection index size.
please try to add more memory for the querynodes

/assign @rh01
/unassign

from milvus.

Memory thresholds of the single query node?

from milvus.

@rh01
How many data do you have?

I think under your case, it's reasonable to change memory threshold to 0.95 to host more data. 0.9 is more like for smaller querynodes.

but you do need to calculate on how many data you have and what's the potential memory requirement.

I'd like to offer help if necessary ([email protected])

Also we'd like to know about the loading time on such a larger dataset and see how things could be improved

from milvus.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen.

from milvus.