milvus-io / milvus Goto Github PK

Describe the solution you'd like
Add Jenkins CI for Milvus

Describe the bug
test_scheduler core_dump when run ./codecoverage

Steps/Code to reproduce behavior
NA

Expected behavior
NA

Environment details
NA

Screenshots
NA

Additional context
NA

Describe the bug
-rw-r--r-- 1 root root Oct 13 05:36 libfaiss.a
-rw-r--r-- 1 root root Oct 19 11:49 libknowhere.a
remove these files in compiled libs

Steps/Code to reproduce behavior
N/A

Expected behavior
No more files not related to milvus-libs after build

Environment details
N/A

Screenshots
N/A

Additional context
N/A

Describe the bug

[ PASSED ] 39 tests. [ FAILED ] 1 test, listed below: [ FAILED ] DBTest2.DELETE_BY_RANGE_TEST 1 FAILED TEST

Expected behavior
All unit test pass

Environment details
CI environment

Screenshots

Additional context

**Is your feature request related to a problem?
No

Describe the solution you'd like
Just involve all tests in projects

Describe alternatives you've considered
May create another REPO to store test cases, but in same REPO will be much earier to synchronize server/test version.

Additional context
N/A

I have add milvus into codebeat. Now I need add a codebeat icon into README.md

Describe the bug
Get github milvus, build and run, create SQ8H index, milvus hangs.

Steps/Code to reproduce behavior
Get github milvus, build and run, create SQ8H index, milvus hangs.

Expected behavior
Milvus should give a message 'Unsupported index type' for this case

Describe the bug
Server no response

Steps/Code to reproduce behavior
Create index: ivf_sq8h with SDK-python

Expected behavior
Error code, with message "not support"

Environment details
branch: 0.5.0/master

Describe the bug
Some troubleshoot messages in the Milvus software do not provide enough information. Users still need to refer to the documentation for details. This might have a negative effect on user experience. We need to improve these messages in the software.

Expected behavior
Please refer to the suggested text below:

Refer to Milvus Documentation for all troubleshooting messages documented in the current release.

Note that the original descriptions are based on the current Milvus documentation. If there are any software changes that are not captured by the documentation or the suggested text is not technically accurate, please let me know.

Describe the bug
make clang-format failed after run build.sh -l

Steps/Code to reproduce behavior
cd milvus/core
build.sh -l
cd cmake_build
make clang-format

result:failed to format the code

Expected behavior
make clang-format successfully after build.sh -l

Describe the bug

• docker rmi registry.zilliz.com/milvus/engine:0.5.0-ubuntu18.04-release
  Error response from daemon: conflict: unable to remove repository reference "registry.zilliz.com/milvus/engine:0.5.0-ubuntu18.04-release" (must force) - container 39c6e133c5b4 is using its referenced image 7e73856a0d3f

Expected behavior
Make sure that a container does not exist that is using the image.

As title.

1. release notes 链接无效。

2. Step 1 Install dependencies

   $ cd [Milvus sourcecode path]/core
   ./ubuntu_build_deps.sh
   

   第二行是不是缺了一个$？

Describe the bug
Memory usage increased slowly during searching vectors

Steps/Code to reproduce behavior
Keep searching about 1000 times

Expected behavior
Memory usage keep stable

Environment details
N/A

Screenshots
N/A

Additional context
version 0.5.1 and version before 0.5.1

Describe the bug
Now, each CI test triggered by PR never pass totally.

Steps/Code to reproduce behavior
Just issue a PR without 'skip ci' message

Expected behavior
Normally, all cases can pass now.

Environment details
CI environment

Screenshots
N/A

Additional context
N/A

Is your feature request related to a problem? Please describe.
Default gpu_cache_capacity is 0，it makes SQ8H index work slow.

Describe the solution you'd like
Set default gpu_cache_capacity in server_config.template.

This link is broken

Report incorrect documentation

Location of incorrect documentation
README

Describe the problems or issues found in the documentation
Code format and code coverage should be in CONTRIBUTING.md

Suggested fix for documentation
Move the content from README to CONTRIBUTING

Is your feature request related to a problem? Please describe.
To test original FAISS benchmark.

Describe the solution you'd like
Add this benchmark under core/index/unittest

Describe alternatives you've considered
N/A

Additional context
test SQ8/SQ8H with L2/IP

Add dumpable interface, make class inherit from it.

Describe the solution you'd like
Add Ubuntu18.04 Build and Runtime environment

Is your feature request related to a problem? Please describe.
Build time is too long.

Describe the solution you'd like
Remove some redundant third party packages in Milvus
Change Arrow's configure arguments and use AUTO approach to build Boost

Describe the solution you'd like
Add ELK for Jenkins CI

Describe the bug
A clear and concise description of what the bug is.

Steps/Code to reproduce behavior
Follow this guide to craft a minimal bug report. This helps us reproduce the issue you're having and resolve the issue more quickly.

Expected behavior
A clear and concise description of what you expected to happen.

Environment details

• Hardware/Softward conditions (OS, CPU, GPU, Memory)
• Method of installation (Docker, or from source)
• Milvus version (v0.3.1, or v0.4.0)
• Milvus configuration (Settings you made in server_config.yaml)

Screenshots
If applicable, add screenshots to help explain your problem.

Additional context
Add any other context about the problem here.

1. Java-SDK tests need to update
2. Add README.md in tests

Is your feature request related to a problem? Please describe.
Some developers don't have Nvidia gpu card on their machine, they cannot compile/run Milvus.

Describe the solution you'd like
Use pre-compiled macro to disable gpu function code

Describe the bug
The server start error messages lack vital information for troubleshooting.

Expected behavior

Describe the bug
Start Milvus server, then run C++ sdk example(sdk_simple) from command line. The example program get grpc error.

Steps/Code to reproduce behavior
1 .Start Milvus server
2. Run C++ sdk example(sdk_simple) from command line

Expected behavior
The example should run successfully.

Environment details

• Milvus version (v0.5.0)
• Default setting

There is a directory named "external" used for external project which source code embedded in project.

Is your feature request related to a problem? Please describe.
I wish I could know Milvus version from log files.

Describe the solution you'd like
Print version information into global.log file in Server::Start()

"Thanks" section in this project looks very well. Maybe we can do it in our readme.
https://github.com/nlohmann/json

@JinHai-CN

Is your feature request related to a problem? Please describe.
I wish I could use Milvus to delete vectors via their generated date，the generated date means vector's actual production date that different from the vector's insert date.

For example : Some vectors come from pictures which generated between 2019-05-01 and 2019-05-25 , but we import the vectors to milvus database at 2019-06-01. If we want delete the vectors between 2019-05-01 and 2019-05-10 , existing apis can not support.

Describe the solution you'd like
Add a api which like "milvus.delete_vectors_by_range('test01', '2019-06-01', '2020-01-01')" but the date means vector's actual production date not vector's insert date.

Describe the solution you'd like
Add code coverage for Milvus Jenkins CI

Is your feature request related to a problem? Please describe.
I wish I could use Milvus to find and delete vector by it's corresponding ID. Currently the SDK supports only removing the vectors using the date-time, which is a little bit useless for the following cases; suppose I've inserted a huge dataset of animals and the last image vector of a person was mistakenly added, so in order to remove the last vector I need to remove the whole table.

Describe the solution you'd like
Delete vector by its corresponding ID.

Describe alternatives you've considered
Actually this feature is already implemented in FAISS search engine and the other ones. I think you know that it is possible to remove the ID from the index, using remove_ids() in FAISS. Is this possible to add the same feature for the Milvus?

Additional context
https://github.com/facebookresearch/faiss/wiki/Special-operations-on-indexes

Release 0.5.1 need new CHANGELOG.md

Is your feature request related to a problem? Please describe.
I wish I could use Milvus to create partition within a table. User can insert vectors into a partition by specifying a partition tag, and search vectors from a certain partition of table. Partition can be dropped.

Describe the solution you'd like
to be discussed..

Additional context
Related issue:
#28 Add new api about vector deletion via generated date not insert date

Is your feature request related to a problem? Please describe.
I wish I could use Milvus to check meta version before server start. For old version meta, if it is migrate-able, Milvus migrate it automatically, otherwise, Milvus should give an error message and exit.

Describe the solution you'd like

1. Add a new column into meta, column name: 'version'.
2. Add new interface to Meta class: Status Migrate();
3. Implement the new interface for SqliteMetaImpl/MySQLMetaImpl class;
4. Call Migrate() in DBImpl::Start();

Describe the bug
Currently, whole compiling time is too long. Downloading boost library source code is one of the reasons.

Steps/Code to reproduce behavior
Each clean compiling

Expected behavior
Shorten the compiling time.

Describe the solution you'd like
Create Mysql container on Jenkins CI build stage

Describe the bug
Topk result is incorrect for small dataset.

Steps/Code to reproduce behavior

1. create a table
2. insert 5 vectors
3. search, nq = 2, topk = 10
   result:
   topk result for no.1 vector:
   id = 0 distance = 0.0
   id = 2 distance = 88.97325134277344
   id = 3 distance = 90.40662384033203
   id = 4 distance = 91.63673400878906
   id = 1 distance = 91.72418212890625
   topk result for no.2 vector:
   id = -1 distance = 3.4028234663852886e+38
   id = -1 distance = 3.4028234663852886e+38
   id = -1 distance = 3.4028234663852886e+38
   id = -1 distance = 3.4028234663852886e+38
   id = -1 distance = 3.4028234663852886e+38

Expected behavior
the topk result for no.2 vector should be like no.1

Additional context
Change k from 10 to 5, the result is correct

Describe the bug
server_test failed in ConfigTest

Steps/Code to reproduce behavior
Run server_test in cmdline, it failed

Describe the solution you'd like
Add Codecov Yaml file

Is your feature request related to a problem? Please describe.
Currently Milvus keep origin data in disk so that user can change index type freely. But it cost too much disk space. I wish I could drop origin data after index is built successfully.

Describe the solution you'd like
Add a parameter in create_index sdk api. "drop_origin"(boolean), default is false.
If create_index(drop_origin=true), the table only contain index files, and user cannot specify any other index type. Call drop_index to this table, the table will become an empty table.
If create_index(drop_origin=false), keep old behavior.