Comments (11)
/assign @LoveEachDay
please help to take a look
/unassign
from milvus.
@LoveEachDay Hello! Could you help to look?
from milvus.
we should fix that on next version of helm chart.
@zwd1208 and should ensure there is no more hard coded configs
from milvus.
@tianshihan818
The root cause of the issue is the use of the emptyDir volume type here, which uses the kubelet working directory as storage. For chunk cache, we recommend using local NVMe disks to achieve better performance. Therefore, the usual approach we suggest is: 1) Schedule the querynode to a worker node with local NVMe disks. 2) Modify the working directory of kubelet in the worker node with local NVMe disks to point to the NVMe disk's directory.
from milvus.
@tianshihan818 The root cause of the issue is the use of the emptyDir volume type here, which uses the kubelet working directory as storage. For chunk cache, we recommend using local NVMe disks to achieve better performance. Therefore, the usual approach we suggest is: 1) Schedule the querynode to a worker node with local NVMe disks. 2) Modify the working directory of kubelet in the worker node with local NVMe disks to point to the NVMe disk's directory.
why can not user specify the disk they want to use?
The problem is helm chard hard coded /var/lib
from milvus.
@xiaofan-luan ok, we will support all storage path related configuration in the next version of helm chart and operator.
from milvus.
We'll add a extra hostPath
volume type which allow user to specify extra disk path to store disk index data and chunk cache, other than default /var/lib/kubelet
.
from milvus.
Thanks all! This will be much helpful!
from milvus.
@LoveEachDay Hello! It passed 3 weeks. I checked the latest version update info of milvus-helm, nothing changed about this issue. Would you support the custom path recently?
from milvus.
after offline discussion, there is actaully much we need to do on hostPath, reason is:
- when multiple pod load on one physical node, we need to seperate the path for each querynode.
- when pod is reconciled, the path need to be cleaned.
@LoveEachDay is still working on it.
from milvus.
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
Rotten issues close after 30d of inactivity. Reopen the issue with /reopen
.
from milvus.
Related Issues (20)
- [Bug]: datacoord binlog size metrics include dropped segment HOT 1
- [Enhancement]: Proposal to Isolate the Milvus Conan Recipe from Upstream Changes HOT 1
- milvus:standond v2.4.6 [Bug]: at /workspace/source/internal/core/src/storage/Util.cpp:84\n => read binlog failed: out range of binlog data" HOT 6
- [Bug]: compaction failed because of imported segment's state is not flushed HOT 7
- [Enhancement]: Enable limiter for restful server
- [Bug]: make: *** [Makefile:253: generated-proto] Error 1 HOT 17
- [Bug]: search failed with error `segment lacks[segment=451679606836035900]: channel not available` after standalone pod kill chaos test HOT 2
- [Bug]: is too hard to make the project,i try my best. HOT 12
- [Bug]: Memory size and delta count in SegmentInfo may be inaccurate
- [Bug]: Unable to Delete Data from Collection After Field Modification in Milvus HOT 3
- [Bug]: "print: command not found" when compile HOT 3
- [Bug]: The `authorizationEnabled` flag reverts to its original state in Milvus standalone mode when using Docker. HOT 2
- [Bug]: checkResultTicker leak HOT 3
- [Bug]: `BloomFilterSet.BatchPkExist` may return false negative when `K` values differ among candidates HOT 2
- [Enhancement]: Check for channel cp lag can be removed
- [Bug]: Strong consistency , delete action, doesnt actually deletes immediately, i could still see the search results HOT 4
- [Feature]: Support to replicate the rbac operation HOT 1
- [Bug]: watch channel stuck forever HOT 2
- [Bug]: milvus crash after dropping a collection if compaction disabled HOT 1
- [Bug]: The decribe index api returns an incoherent response structure when using orm create index and milvusclient create index respectively HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from milvus.