[Bug]: When constructing duplicate data using query expressions, the returned data may be incomplete. about milvus HOT 9 OPEN

OK, I'll do it. @lentitude2tk

from milvus.

/assign @congqixia
/unassign

from milvus.

@lentitude2tk
could you try latest milvus and check if this is reproducible?

from milvus.

/assign @MrPresent-Han

from milvus.

@lentitude2tk could you try latest milvus and check if this is reproducible?

OK, I will upgrade to the latest version 2.4 and run the test.

from milvus.

@xiaofan-luan I have upgraded to version 2.4.2-20240514-f48a7ff-72fc31b and conducted tests, which met expectations. From the results, it appears that Milvus 2.4 has optimized its query strategy compared to 2.3.

For the same dataset with offset=0, limit=900, and expr="", querying the data (0-999) repeated 5 times:

1. Version 2.3 could only return over 300 records, with some data missing.
2. Version 2.4 returned a complete set of 900 deduplicated records, covering the range 0-999.

from milvus.

/assign

from milvus.

already fixed by #32567 , in this pr, the reduce of query will try the best to get the limit results (availableCount < limit).

from milvus.

already fixed by #32567 , in this pr, the reduce of query will try the best to get the limit results (availableCount < limit).

milvus/internal/querynodev2/segments/result.go

Line 551 in f76d167

for j := 0; j < loopEnd && (limit == -1 || availableCount < limit); j++ {

Can this handling strategy code be synchronized to version 2.3?

from milvus.