Comments (3)
zhuwenxing
commented on June 26, 2024
search also failed with this error
failed job: https://qa-jenkins.milvus.io/blue/organizations/jenkins/chaos-test-for-release-cron/detail/chaos-test-for-release-cron/11369/pipeline
log:
artifacts-querynode-pod-failure-11369-server-logs.tar.gz
[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request) : [Connections.connect] args: ['default', '', '', ''], kwargs: {'host': '10.255.151.122', 'port': 19530} (api_request.py:62)
[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request) : [Connections.has_connection] args: ['default'], kwargs: {} (api_request.py:62)
[2024-03-24T23:19:38.262Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : True (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request) : [Collection] args: ['Checker__ZYZ0pfxV', {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}, {'name': 'varchar', 'description': '', 'type': <DataType......, kwargs: {'consistency_level': 'Strong'} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_response) : <Collection>:
[2024-03-24T23:19:38.263Z] -------------
[2024-03-24T23:19:38.263Z] <name>: Checker__ZYZ0pfxV
[2024-03-24T23:19:38.263Z] <description>:
[2024-03-24T23:19:38.263Z] <schema>: {'auto_id': False, 'description': '', 'fields': [{'name': 'int64', 'description': '', 'type': <DataType.INT64: 5>, 'is_primary': True, 'auto_id': False}, {'name': 'float', 'description': '', 'type': <DataType.FLOAT: 10>}...... (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:48 - DEBUG - ci_test]: (api_request) : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_request) : [Collection.compact] args: [180], kwargs: {} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:13:54 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:54 - DEBUG - ci_test]: (api_request) : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - INFO - ci_test]: assert create collection: 0.03525233268737793, init_entities: 745955 (test_all_collections_after_chaos.py:49)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_request) : [Collection.insert] args: [[[-3000, -2999, -2998, -2997, -2996, -2995, -2994, -2993, -2992, -2991, -2990, -2989, -2988, -2987, -2986, -2985, -2984, -2983, -2982, -2981, -2980, -2979, -2978, -2977, -2976, -2975, -2974, -2973, -2972, -2971, -2970, -2969, -2968, -2967, -2966, -2965, -2964, -2963, -2962, -2961, -2960, -2959, -29......, kwargs: {'timeout': 180} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_response) : (insert count: 2000, delete count: 0, upsert count: 0, timestamp: 448612819296583683, success count: 2000, err count: 0) (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - INFO - ci_test]: assert insert: 0.5283844470977783 (test_all_collections_after_chaos.py:61)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:55 - DEBUG - ci_test]: (api_request) : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request) : [Collection.flush] args: [], kwargs: {'timeout': 180} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - INFO - ci_test]: assert flush: 3.569885492324829, entities: 747955 (test_all_collections_after_chaos.py:71)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - INFO - ci_test]: index info: [{'collection': 'Checker__ZYZ0pfxV', 'field': 'float_vector', 'index_name': 'vec_index', 'index_param': {'index_type': 'HNSW', 'metric_type': 'L2', 'params': {'M': 48, 'efConstruction': 500}}}] (test_all_collections_after_chaos.py:87)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request) : [Collection.load] args: [None, 1, 180], kwargs: {} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_response) : None (api_request.py:37)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:14:59 - DEBUG - ci_test]: (api_request) : [Collection.search] args: [[[0.1394604147661045, 0.10148974298142811, 0.09137981234031903, 0.03233836942294032, 0.09185298370886213, 0.08251813531741621, 0.058100080704232296, 0.007344297647441485, 0.08989386617268842, 0.09960945674128009, 0.058880340721370904, 0.03205196773015912, 0.10427486666482212, 0.10932439188056996, 0......, kwargs: {} (api_request.py:62)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:15:00 - ERROR - pymilvus.decorators]: RPC error: [search], <MilvusException: (code=503, message=fail to search on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_11_448612279430753456v0])>, <Time:{'RPC start': '2024-03-24 23:14:59.857512', 'RPC error': '2024-03-24 23:15:00.755138'}> (decorators.py:139)
[2024-03-24T23:19:38.263Z] [2024-03-24 23:15:00 - ERROR - ci_test]: Traceback (most recent call last):
[2024-03-24T23:19:38.263Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 32, in inner_wrapper
[2024-03-24T23:19:38.263Z] res = func(*args, **_kwargs)
[2024-03-24T23:19:38.263Z] File "/home/jenkins/agent/workspace/tests/python_client/utils/api_request.py", line 63, in api_request
[2024-03-24T23:19:38.263Z] return func(*arg, **kwargs)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/orm/collection.py", line 792, in search
[2024-03-24T23:19:38.263Z] resp = conn.search(
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 140, in handler
[2024-03-24T23:19:38.263Z] raise e from e
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 136, in handler
[2024-03-24T23:19:38.263Z] return func(*args, **kwargs)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 175, in handler
[2024-03-24T23:19:38.263Z] return func(self, *args, **kwargs)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 115, in handler
[2024-03-24T23:19:38.263Z] raise e from e
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/decorators.py", line 86, in handler
[2024-03-24T23:19:38.263Z] return func(*args, **kwargs)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 775, in search
[2024-03-24T23:19:38.263Z] return self._execute_search(request, timeout, round_decimal=round_decimal, **kwargs)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 735, in _execute_search
[2024-03-24T23:19:38.263Z] raise e from e
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/grpc_handler.py", line 728, in _execute_search
[2024-03-24T23:19:38.263Z] check_status(response.status)
[2024-03-24T23:19:38.263Z] File "/usr/local/lib/python3.8/dist-packages/pymilvus/client/utils.py", line 54, in check_status
[2024-03-24T23:19:38.263Z] raise MilvusException(status.code, status.reason, status.error_code)
[2024-03-24T23:19:38.263Z] pymilvus.exceptions.MilvusException: <MilvusException: (code=503, message=fail to search on QueryNode 14: distribution is not servcieable: channel not available[channel=by-dev-rootcoord-dml_11_448612279430753456v0])>
from milvus.
xiaofan-luan
commented on June 26, 2024
/assign @congqixia @tedxu
We don't really need the "distribution is not servcieable" everything should be just find the correct shard delegator and let the delegator decide whether to host or wait until recovery done.
Currently design is too error prone and there is no easy way to sync complex state between node
from milvus.
congqixia
commented on June 26, 2024
Segment lack error could be related to #31584 #31602
it looks like this error does not exist in recent chaos test
/assign @zhuwenxing
from milvus.
Related Issues (20)
- [Bug]: The milvus instance does not have any requests but the query count(*) results become larger HOT 1
- [Bug]: test_search_pagination_group_by failed HOT 2
- [Bug]: Chunkcache files may potentially leak HOT 1
- [Enhancement]: row-based import, only allow one JSON file each time HOT 2
- [Bug]: [standalone][LRU] Milvus standalone goroutine leak when enable lru cache HOT 3
- [Enhancement]: Make querycoordv2 collection observer task driven
- [Bug]: GPU_BRUTE_FORCE Index Milvus Search error HOT 1
- [Bug]: The number of L0 segmtn loaded by the querynode keeps growing HOT 1
- [Feature]: Support brew install and brew services HOT 3
- [Enhancement]: Do not list prefix on S3 when load partition HOT 3
- [Bug]: TestAvgReassignPolicy/test_normal_reassigning_for_one_available_nodes ut failed
- [Bug]: The query result is empty after Milvus recovers from a querynode pod kill or pod failure chaos HOT 3
- [Enhancement]: DataNode Compactor tracing and timing is confusing
- [Bug]: [benchmark][standalone][LRU] search and query raiase error in concurrent DQL and insert scene HOT 2
- [Bug]: falled to query: node offline HOT 4
- [Bug]: Fail to search on QueryNode: When metricType is L2 and range_filter < radius, precision loss error occurs. The range_filter must be less than the radius. HOT 8
- [Bug]: [benchmark][cluster] After queryNode pod failure is restored, hybrid_search RT increases in replica=2 scene HOT 3
- [Bug]: Docker arm64 image milvusdb/milvus:v2.4.0: symbol lookup error HOT 17
- [Bug]: ARM64 compilation problems HOT 2
- [Bug]: prefix for CN works well, but not postfix HOT 19
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.