Comments (3)
chyezh
commented on June 6, 2024
1
is_system_field_ready failure is fixed, please verify it.
@wangting0128
from milvus.
chyezh
commented on June 6, 2024
LoadIndex failure with an unexpected error which report creating directory failure, cannot be fixed in this turn.
[2024/05/06 12:34:55.352 +00:00] [WARN] [segments/load_index_info.go:231] ["CStatus returns err"] [traceID=c1245c727fbb577430a785d2c2f4d933] [error="filesystem error: cannot create directories: Not a directory [/var/lib/milvus/data/mmap/449570705034082594/101]"] [extra="AppendIndex failed"]
And segment_loader's error handling is broken, then assertion failure is_system_field_ready reported.
I will fix it, and wait the first bug reproduced, please don't release the cluster in next test.
from milvus.
wangting0128
commented on June 6, 2024
verification passed
image: master-20240514-f48a7ff8
argo task: lru-fouramf-wx7vx
test result:
{'server': {'deploy_tool': 'helm',
'deploy_mode': 'standalone',
'config_name': 'standalone_8c16m',
'config': {'standalone': {'resources': {'limits': {'cpu': '2',
'memory': '8Gi',
'ephemeral-storage': '70Gi'},
'requests': {'cpu': '2',
'memory': '8Gi'}},
'messageQueue': 'pulsar',
'extraEnv': [{'name': 'LOCAL_STORAGE_SIZE',
'value': '70'}],
'disk': {'size': {'enabled': True}}},
'cluster': {'enabled': False},
'etcd': {'replicaCount': 1,
'metrics': {'enabled': True,
'podMonitor': {'enabled': True}}},
'minio': {'mode': 'standalone',
'metrics': {'podMonitor': {'enabled': True}},
'persistence': {'size': '320Gi'}},
'pulsar': {'enabled': True},
'metrics': {'serviceMonitor': {'enabled': True}},
'log': {'level': 'debug'},
'extraConfigFiles': {'user.yaml': 'queryNode:\n'
' '
'diskCacheCapacityLimit: '
'51539607552\n'
' mmap:\n'
' mmapEnabled: '
'true\n'
' lazyload:\n'
' enabled: true\n'
' waitTimeout: '
'300000\n'
' '
'useStreamComputing: '
'true\n'
' cache:\n'
' warmup: off\n'},
'image': {'all': {'repository': 'harbor.milvus.io/milvus/milvus',
'tag': 'master-20240514-f48a7ff8'}}},
'host': 'lru-32987-dml-dql-milvus.qa-milvus.svc.cluster.local',
'port': '19530',
'uri': ''},
'client': {'test_case_type': 'ConcurrentClientBase',
'test_case_name': 'test_concurrent_locust_custom_parameters',
'test_case_params': {'dataset_params': {'metric_type': 'L2',
'column_name': 'float32_vector',
'dim': 768,
'scalars_params': {'int64_1': {'params': {'is_partition_key': True}}},
'dataset_name': 'laion1b_nolang',
'dataset_size': '10w',
'ni_per': 10000},
'collection_params': {'other_fields': ['int64_1'],
'num_partitions': 64},
'index_params': {'index_type': 'HNSW',
'index_param': {'M': 30,
'efConstruction': 360}},
'concurrent_params': {'concurrent_number': 10,
'during_time': '12h',
'interval': 20,
'spawn_rate': None},
'concurrent_tasks': [{'type': 'insert',
'weight': 1,
'params': {'nb': 64,
'timeout': 3000,
'random_vector': True}},
{'type': 'delete',
'weight': 1,
'params': {'delete_length': 64,
'timeout': 3000}},
{'type': 'flush',
'weight': 0,
'params': {'timeout': 3000}},
{'type': 'load',
'weight': 1,
'params': {'timeout': 3000}},
{'type': 'search',
'weight': 1,
'params': {'top_k': 1,
'nq': 10,
'search_param': {'ef': 64},
'expr': 'int64_1 '
'>= '
'0 '
'&& '
'int64_1 '
'<= '
'4',
'timeout': 3000,
'random_data': True}},
{'type': 'query',
'weight': 1,
'params': {'expr': 'int64_1 '
'!= '
'2',
'timeout': 3000,
'offset': 0,
'limit': 10}}]},
'run_id': 2024051412144173,
'datetime': '2024-05-14 12:53:34.093352',
'client_version': '2.2'},
'result': {'test_result': {'index': {'RT': 63.8284},
'insert': {'total_time': 27.8717,
'VPS': 3587.8687,
'batch_time': 2.7872,
'batch': 10000},
'flush': {'RT': 4.9282},
'load': {'RT': 3.5367},
'Locust': {'Aggregated': {'Requests': 624242,
'Fails': 0,
'RPS': 14.45,
'fail_s': 0.0,
'RT_max': 68522.33,
'RT_avg': 689.36,
'TP50': 98,
'TP99': 15000.0},
'delete': {'Requests': 124461,
'Fails': 0,
'RPS': 2.88,
'fail_s': 0.0,
'RT_max': 20965.18,
'RT_avg': 48.13,
'TP50': 34,
'TP99': 280.0},
'insert': {'Requests': 124810,
'Fails': 0,
'RPS': 2.89,
'fail_s': 0.0,
'RT_max': 20390.22,
'RT_avg': 348.05,
'TP50': 260.0,
'TP99': 1800.0},
'load': {'Requests': 125020,
'Fails': 0,
'RPS': 2.89,
'fail_s': 0.0,
'RT_max': 20068.64,
'RT_avg': 75.72,
'TP50': 53,
'TP99': 430.0},
'query': {'Requests': 125121,
'Fails': 0,
'RPS': 2.9,
'fail_s': 0.0,
'RT_max': 68510.11,
'RT_avg': 1502.58,
'TP50': 130.0,
'TP99': 24000.0},
'search': {'Requests': 124830,
'Fails': 0,
'RPS': 2.89,
'fail_s': 0.0,
'RT_max': 68522.33,
'RT_avg': 1469.38,
'TP50': 110.0,
'TP99': 23000.0}}}}}
from milvus.
Related Issues (20)
- [Bug]: [benchmark][cluster] queryNode panic `what(): Error:RemoveDir:Directory not empty` in the scenario of serial load-release collection which replica=2
- [Bug]: Query iterator with expr `int64 < 100 || int64 > 2000` gets unexpected result HOT 2
- [Bug]: The size of the import segments is uneven HOT 1
- [Enhancement]: optimize json to reduce memory cost
- [Bug]: The deleted collection reappears in the cluster and cannot be deleted. Therefore, the cluster collection cannot be used HOT 6
- [Enhancement]: Use BatchPkExist to reduce bloom filter func call cost HOT 1
- [Feature]: Reranking step on binary search
- [Bug]: [GOSDK] Search with OutputFields but return empty fields data
- [Enhancement]: speed up array-equal operator via inverted index
- [Bug]: Lose valid data when setting radius and range filter in vector search HOT 2
- [Bug]: No error is returned when searching an empty collection with a vector that has a different schema dim HOT 2
- [Bug]: No error is returned when searching an empty collection with a vector that has a different schema DataType HOT 4
- [Bug]: System continues to create empty segments after upgrade to 2.4 HOT 2
- `id` [Feature]: test
- `env`[Feature]: test
- [Bug]: Flush timeout and search fails for nearly 17 minutes after chaos recovery from pod-kill standalone HOT 1
- [Feature]: Support Query GroupBy for Milvus HOT 2
- [Bug]: failed to create index, C Runtime Exception ，segcoreCode=2004 HOT 1
- [Feature]: add a pinnable cache
- [Bug]: Datacoord panics if querying an import job that does not exist
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.