Comments (5)
yanliang567
commented on June 16, 2024
/assign @weiliu1031
/unassign
from milvus.
weiliu1031
commented on June 16, 2024
should be fixed by #31798, please verify this with latest image
from milvus.
weiliu1031
commented on June 16, 2024
/assign @NicoYuan1986
from milvus.
NicoYuan1986
commented on June 16, 2024
The error has not occurred since the pr merged. So close the issue.
I will reopen it if the issue reproduces.
from milvus.
NicoYuan1986
commented on June 16, 2024
reproduced on 2.3 branch.
- link: https://jenkins.milvus.io:18080/blue/organizations/jenkins/Milvus%20Nightly%20CI/detail/2.3/171/pipeline/237
- log: artifacts-milvus-distributed-pulsar-nightly-171-pymilvus-e2e-logs.tar.gz
- collection name: search_collection_VKAMMKtC
- failed time: [2024-04-17T23:04:45.481Z] [gw0] [ 66%] FAILED testcases/test_search.py::TestCollectionSearch::test_search_with_consistency_bounded[200-128-True-True-True]
- error message:
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:39 - DEBUG - ci_test]: (api_request) : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - pymilvus.decorators]: RPC error: [load_collection], <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>, <Time:{'RPC start': '2024-04-17 23:04:39.053386', 'RPC error': '2024-04-17 23:04:44.068447'}> (decorators.py:146)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: Traceback (most recent call last):
[2024-04-17T23:44:54.981Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-04-17T23:44:54.981Z] res = func(*args, **_kwargs)
[2024-04-17T23:44:54.981Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-04-17T23:44:54.981Z] return func(*arg, **kwargs)
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/orm/collection.py", line 419, in load
[2024-04-17T23:44:54.981Z] conn.load_collection(
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 147, in handler
[2024-04-17T23:44:54.981Z] raise e from e
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 143, in handler
[2024-04-17T23:44:54.981Z] return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 182, in handler
[2024-04-17T23:44:54.981Z] return func(self, *args, **kwargs)
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 122, in handler
[2024-04-17T23:44:54.981Z] raise e from e
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/decorators.py", line 87, in handler
[2024-04-17T23:44:54.981Z] return func(*args, **kwargs)
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/client/grpc_handler.py", line 1046, in load_collection
[2024-04-17T23:44:54.981Z] check_status(response)
[2024-04-17T23:44:54.981Z] File "/usr/local/lib/python3.8/site-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-04-17T23:44:54.981Z] raise MilvusException(status.code, status.reason, status.error_code)
[2024-04-17T23:44:54.981Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).Call
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:567 github.com/milvus-io/milvus/internal/util/grpcclient.(*ClientBase[...]).ReCall
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:107 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.wrapGrpcCall[...]
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/distributed/rootcoord/client/client.go:275 github.com/milvus-io/milvus/internal/distributed/rootcoord/client.(*Client).ShowPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/meta/coordinator_broker.go:97 github.com/milvus-io/milvus/internal/querycoordv2/meta.(*CoordinatorBroker).GetPartitions
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/job_load.go:118 github.com/milvus-io/milvus/internal/querycoordv2/job.(*LoadCollectionJob).Execute
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:159 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).process
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/querycoordv2/job/scheduler.go:135 github.com/milvus-io/milvus/internal/querycoordv2/job.(*Scheduler).processQueue
[2024-04-17T23:44:54.981Z] /usr/local/go/src/runtime/asm_amd64.s:1598 runtime.goexit: rpc error: code = DeadlineExceeded desc = context deadline exceeded)>
[2024-04-17T23:44:54.981Z] (api_request.py:45)
[2024-04-17T23:44:54.981Z] [2024-04-17 23:04:44 - ERROR - ci_test]: (api_response) : <MilvusException: (code=65535, message=stack trace: /go/src/github.com/milvus-io/milvus/pkg/tracer/stack_trace.go:51 github.com/milvus-io/milvus/pkg/tracer.StackTrace
[2024-04-17T23:44:54.981Z] /go/src/github.com/milvus-io/milvus/internal/util/grpcclient/client.go:553 github.com/milvus-io/milvus/internal/util/grpcclient.(*Cli...... (api_request.py:46)
from milvus.
Related Issues (20)
- [Bug]: The milvus instance does not have any requests but the query count(*) results become larger HOT 1
- [Bug]: test_search_pagination_group_by failed HOT 2
- [Bug]: Chunkcache files may potentially leak HOT 1
- [Enhancement]: row-based import, only allow one JSON file each time HOT 2
- [Bug]: [standalone][LRU] Milvus standalone goroutine leak when enable lru cache HOT 3
- [Enhancement]: Make querycoordv2 collection observer task driven
- [Bug]: GPU_BRUTE_FORCE Index Milvus Search error HOT 1
- [Bug]: The number of L0 segmtn loaded by the querynode keeps growing HOT 1
- [Feature]: Support brew install and brew services HOT 3
- [Enhancement]: Do not list prefix on S3 when load partition HOT 3
- [Bug]: TestAvgReassignPolicy/test_normal_reassigning_for_one_available_nodes ut failed
- [Bug]: The query result is empty after Milvus recovers from a querynode pod kill or pod failure chaos HOT 3
- [Enhancement]: DataNode Compactor tracing and timing is confusing
- [Bug]: [benchmark][standalone][LRU] search and query raiase error in concurrent DQL and insert scene HOT 2
- [Bug]: falled to query: node offline HOT 4
- [Bug]: Fail to search on QueryNode: When metricType is L2 and range_filter < radius, precision loss error occurs. The range_filter must be less than the radius. HOT 8
- [Bug]: [benchmark][cluster] After queryNode pod failure is restored, hybrid_search RT increases in replica=2 scene
- [Bug]: Docker arm64 image milvusdb/milvus:v2.4.0: symbol lookup error HOT 17
- [Bug]: ARM64 compilation problems HOT 2
- [Bug]: prefix for CN works well, but not postfix HOT 19
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.