Although I specify the argument of num_threads in Index.insert(), it seems that there is only one CPU working. If I "top" in the command line, there is only one process with about 100% CPU usage. And it also won't change when I change the value from 16 to 40 (40 CPUs on the machine).

In addition, I am using the Python API via jupyter notebook.

Look like index.insert return id what is an one more than expected:

import ngtpy

ngtpy.create(b"index", 20)
index = ngtpy.Index(b"index")
vector = numpy.zeros(20)
vid = index.insert(vector)  # it return id=1 on first inserted vector
logging.warning(f"\tInsert returned vector id={vid}")
index.build_index()
index.save()
try:
    stored_vector = index.get_object(vid)   # but id=1 raise error...
    logging.warning(f"\tVector of id={vid} was success get")
except Exception as e:
    logging.error(f"\tCan not get index by id={vid}", exc_info=e)
vid -= 1
stored_vector = index.get_object(vid)    # if we use id=0 then it success returning inserted data
logging.warning(f"\tVector of id={vid} was success get")

Out is:

WARNING:root:	Insert returned vector id=1
ERROR:root:	Can not get index by id=1
Traceback (most recent call last):
  File ".../test.py", line 85, in <module>
    stored_vector = index.get_object(vid)
RuntimeError: /work/NGT/lib/NGT/ObjectSpaceRepository.h:390: NGT::ObjectSpaceRepository: The specified ID is out of the range. The object ID should be greater than zero. 2:2.
WARNING:root:	Vector of id=0 was success get

System info:

% pip3 freeze -l
ngt==1.8.1
pybind11==2.4.3
 % uname -a
Linux ... 5.2.0-3-amd64 #1 SMP Debian 5.2.17-1 (2019-09-26) x86_64 GNU/Linux

Does NGT support query on ARM?

I am following 'ngpy' tutorial , here https://github.com/yahoojapan/NGT/tree/master/python
under simple samples , ngpy section.
I am executing this line 'ngtpy.create(b"tmp", dim)'
I am getting this error.
Can you explain, what 'tmp' is? Is it a directory or file or what?
Why have we put 'b' infront of it?

OS - Ubuntu 16
Python Version - Python 3.5

If cv2 python package is imported after importing ngtpy, it results to a free(): invalid pointer. Below is a example:

(base) ➜  search_test git:(master) ✗ ipython                 
Python 3.7.4 (default, Aug 13 2019, 20:35:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import ngtpy 
   ...: import cv2 
   ...:                                                                 
free(): invalid pointer
[1]    703 abort (core dumped)  ipython

But importing cv2 before would'nt result to this issue.

(base) ➜  search_test git:(master) ✗ ipython
Python 3.7.4 (default, Aug 13 2019, 20:35:49) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.8.0 -- An enhanced Interactive Python. Type '?' for help.

In [1]: import cv2 
   ...: import ngtpy 
   ...:                                                                 

In [2]:     

Hi, thanks again for the prompt responses to the previous question #23 .

We are trying to generalize your code to Jaccard distance and test it under the ann benchmarking. We assume that the input is the same as the input of hamming distance, but the distance function is changed to Jaccard. E.g., for two bit vectors "A=10111" and "B=10011", their hamming distance is 1 but jaccard distance is 1-popcount(A&B)|/popcount(A|B)=1-3/4=0.25.

What we did are that for every code including hamming, generating corresponding code for jaccard. For example in our repository,
https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/lib/NGT/PrimitiveComparator.h#L287-L303

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/lib/NGT/ObjectSpaceRepository.h#L97-L114

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/lib/NGT/Index.h#L113

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/lib/NGT/Command.cpp#L155-L157

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/lib/NGT/ObjectSpace.h#L162

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/python/src/ngtpy.cpp#L67-L68

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/python/ngt/base.py#L131

https://github.com/chunjiangzhu/ngt/blob/22a99c1eeb13590bae707afbe006247e80b32f5e/python/ngt/base.py#L254-L256

Our input data is an N*1024 numpy array of data type int (or bool). We have tested the code using parameters epsilon=[0.0,0.1], edge_size=[100, 200, 300, 500, 1000], outdegree=[10, 30, 50, 70, 100], indegree=[10, 30, 50, 70, 120], query epsilon=[0.1, 0.2, 0.4, 0.6, 0.8, 1.0, 1.2, 1.4, 1.6, 1.8, 2.0] and object_type=Byte. But the resulting recall are consistently lower than 20%. We believe that there are something wrong. We noticed the 16 boundary you mentioned in #21 . Since our data dimension is always a power of 16, e.g. 1024, the error should not be here.

Could you please give suggestions on the generalization? We hope that a successful generalization may be helpful to support more distance metrics, before you make a custom distance function. If you need more information, e.g. a dataset, please let me know. Thank you so much!

I use C api and call open and close index function.
But memory dosen't release after call ngt_close_index.
I expect memory release on call that.

After opening the database of about 900 MB, I examined the memory usage at the time of closing by the free command.

The environment and programs are as follows:

$ uname -a
Linux localhost.localdomain 3.10.0-693.2.1.el7.x86_64 #1 SMP Wed Sep 6 20:06:13 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ cat /etc/redhat-release
CentOS Linux release 7.4.1708 (Core)

#include "NGT/Capi.h"
#include <stdlib.h>
#include <stdio.h>
int main(int argc, char *argv[]){
   char str[256];

   NGTError err = ngt_create_error_object();
   NGTIndex index  = ngt_open_index("/path/to/my/database", err);

   printf("Opened\n");
   scanf("%s", str); // -> used 997M

   ngt_close_index(index);
   ngt_destroy_error_object(err);

   printf("Closed\n");
   scanf("%s", str); // -> used 966M. I expect 73M.

   return(0);
}

I execute gcc -L/usr/local/lib -lstdc++ -lngt hoge.c && LD_LIBRARY_PATH=/usr/local/lib ./a.out and free command.

I get output of free command:

# Opened
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.7G        997M        1.8G        708K        1.9G        3.5G
Swap:          1.5G         58M        1.4G

# Closed
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.7G        966M        1.8G        708K        1.9G        3.5G
Swap:          1.5G         58M        1.4G

# Exited program.
$ free -h
              total        used        free      shared  buff/cache   available
Mem:           4.7G         73M        2.7G        708K        1.9G        4.4G
Swap:          1.5G         58M        1.4G

Thank you for publishing a wonderful product.

When using index.search or index.get_object, it sometimes returns object ids that are larger than the total length of the inserted data. When I try to delete these object Ids, I recieve an error saying that there objects are not in the index.

If I comment the remove function and uncomment the get_object, the code runs without error.
NGT-1.9.0
Python 3.8.2

RuntimeError Traceback (most recent call last)
in
2 for i in tqdm(range(3606235, 3607995)):
----> 3 index.remove(i)
4 # index.get_object(i)
5 index.save()

RuntimeError: /home/jupyter/NGT-1.9.0/lib/NGT/Index.h:1622: remove:: cannot remove from tree. id=3606236 /home/jupyter/NGT-1.9.0/lib/NGT/Tree.h:191: VpTree::remove: Inner error. Cannot remove object. leafNode=203725:/home/jupyter/NGT-1.9.0/lib/NGT/Node.cpp:260: VpTree::Leaf::remove: Cannot find the specified object. ID=3606236,0 idx=80 If the same objects were inserted into the index, ignore this message.

Thanks for building NGT, this is excellent!
I'm wondering what would be the most efficient way to compute the approximate k nearest neighbours for all data points in the index in python. I see you have a batch_insert function but I do not see an equivalent batch_query function.
The equivalent code in e.g. scikit-learn would be

import sklearn.neighbors
distances, indices = sklearn.neighbors.NearestNeighbors(n_neighbors).fit(data).kneighbors(data)

What is exactly dimension parameter in create function?

I use NGTpy.

And encounter issue:

ngtpy::insert: Error! dimensions are inconsitency. 20:600

An extra null pointer check is not needed in functions like the following.

• CompactVector:clear
• Index::close
• Thread

Hi,

I was wondering if NGT in python supports insertion of new elements after the index is built. Basically for scenarios that we want to grow the index object as the data grows without inserting all elements again and rebuilding a new index object.

Thanks,
Mehrsa

I have seen that no loop is used around a call of the function “JobQueue::wait”.
Would you like to reuse anything from my message on the topic “spurious wakeup”?

I explored NGT with some real dataset (100K 129d normalized vectors, top 100, 10 queries) with default parameters used in Command.cpp as below. I turned on AVX, OpenMP and opt optimization using GCC, but I didn't use NGT_SHARED_MEMORY_ALLOCATOR setting to keep the index in memory.

  NGT::Property prop;
  prop.edgeSizeForCreation = 10;
  prop.edgeSizeForSearch = 40;
  prop.batchSizeForCreation = 200;
  prop.insertionRadiusCoefficient = 1.1;
  prop.truncationThreshold = 0;
  prop.dimension = transDimension;;
  prop.threadPoolSize = 24;
  prop.pathAdjustmentInterval = 0;
  prop.dynamicEdgeSizeBase = 30;
  prop.buildTimeLimit = 0.0;
  prop.distanceType = NGT::Index::Property::DistanceType::DistanceTypeNormalizedCosine;

  NGT::GraphAndTreeIndex index(prop);
  index.append(transData.data(), dataSize);
  index.createIndex(24);

  vector<int64_t> outputIndexes(queryDataSize * k);

  for (size_t i = 0; i < queryDataSize; i++) {
    NGT::Object *query = index.allocateObject(
        transQueryData.data() + i * transDimension, transDimension);

    NGT::SearchContainer sc(*query);
    NGT::ObjectDistances objects;
    sc.setResults(&objects);
    sc.setSize(k);
    sc.setRadius(FLT_MAX);
    sc.setEpsilon(0.1);

    int64_t startTime = getWallTimeMS();
    index.search(sc);
    int64_t endTime = getWallTimeMS();

    LOG(INFO) << " query " << i
      << ": ts = " << endTime - startTime
      << " ms, # of result = " << objects.size();

    for (size_t j = 0; j < objects.size() && j < k; j++) {
      outputIndexes[i * k + j] = objects[j].id - 1;
    }
    index.deleteObject(query);
  }

Here is the test result

query 0: ts = 93 ms, # of result = 100
query 1: ts = 94 ms, # of result = 100
query 2: ts = 0 ms, # of result = 0
query 3: ts = 93 ms, # of result = 100
query 4: ts = 0 ms, # of result = 0
query 5: ts = 86 ms, # of result = 100
query 6: ts = 88 ms, # ofl result = 100
query 7: ts = 90 ms, # of result = 100
query 8: ts = 89 ms, # of result = 100
query 9: ts = 89 ms, # of result = 100

There are several issues in this test:

• There is no result for two of ten queries. Did I miss anything in the parameter settings?
• We need to create ObjectDistance for every query. Is there a way that we can reuse the objects to avoid allocating memory for wide-range and high-frequency queries?
• The latency is high. As a comparison, it took 26ms in total to calculate ten results using CBLAS+OpenMP brute force matrix multiplication. I tried to tune the setEpsilon to be 0.03. It took effect to reduce the latency to half with a sacrifice on recall. However, it is still more than the brutal force on average. Are there other parameters or compilation options that I can try to improve the CPU performance?

Thanks!

Thanks for releasing NGT.
I have tried running the provide sample python script.

It seems that the scripts works as expected (quick and quite accurate) after I create the conda environment for running NGT package.
However, after I deactivate and activate the same environment again, I get the error message:

ModuleNotFoundError: No module named 'ngtpy'

It seems to be the problem with how the path is specified.
But, I don't know how to fix this problem. Could you recommend how to fix this path problem, or may be could you give me away to force the path to both ngtpy and ngt modules?

I performed the following command lines, which leads to the problem:

1. Firstly, I installed NGT on my linux (base-environment)

$ unzip NGT-x.x.x.zip
$ cd NGT-x.x.x
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install
$ ldconfig /usr/local/lib

2. Then, I install the python package in base-environment using:

$ pip3 install pybind11
$ pip3 install numpy
$ cd NGT_ROOT/python
$ python3 setup.py sdist
$ pip3 install dist/ngt-x.x.x.tar.gz

3. Then, I create a conda environment, activate it to run the sample.py, and then, deactivate the conda environment.
4. After deactivation, I just simply activate the conda environment that I have just created. Then, it doesn't work.

Hi,
I was wondering if you were planning to add support for custom distance_type's in the python API? I have a complicated function in python that I wish to use as a custom distance metric and am interested in the scalability of this package.

For example, I would like to set distance_type=func.
As with sklearn's NearestNeighbors, the function typically has arguments for each point input (can be input as 2D numpy vectors). Would this work for NGT?

def func(x1, x2): #L1
    return np.sum(np.abs(x1-x2), axis=1)

What are the advantages and disadvantages of using the normalised distance function?

I was curios if these algorithms would work with binary data? I didn't see anything about that in the the papers on the algorithms.

I executed the following command：
$ unzip NGT-x.x.x.zip
$ cd NGT-x.x.x
$ mkdir build
$ cd build
$ cmake ..
$ make
$ make install
$ ldconfig /usr/local/lib
when execute "make",an warning is reported:
/NGT-1.7.7/lib/NGT/MmapManager.cpp:367:15: warning: ignoring return value of ‘char* strerror_r(int, char*, size_t)’, declared with attribute warn_unused_result [-Wunused-result]
strerror_r(err_num, err_msg, 256);
~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~

when execute "make install",an error is reported:
CMake Error at lib/NGT/cmake_install.cmake:41 (file):
file INSTALL cannot copy file
"/home/che/NGT-1.7.7/build/lib/NGT/libngt.so.1.7.7" to
"/usr/local/lib/libngt.so.1.7.7".
Call Stack (most recent call first):
lib/cmake_install.cmake:42 (include)
cmake_install.cmake:42 (include)

Makefile:73: recipe for target 'install' failed
make: *** [install] Error 1

Is NGT only able to implement the onng algorithm via the ngt command? How does ngtpy implement onng algorithm? Ann-benchmarks seems to be the onng algorithm implemented by the ngt command.

I am trying to run the example code for ctype in this guide below: https://github.com/yahoojapan/NGT/blob/master/python/README.md
I followed this instructions to install NGT for ctype.
https://github.com/yahoojapan/NGT/blob/master/README.md#build
However, since I don't have permission to write into /usr/local/lib/ folder, I had to run the installation like follow:

 $ tar zxvf NGT-1.7.6.tar.gz
 $ cd NGT-1.7.6
 $ mkdir build
 $ cd build 
 $ cmake -DCMAKE_INSTALL_PREFIX=<MY_NGT_DIR> ..
 $ make 
 $ make install

Then I set LD_LIBRARY_PATH in ~/.bashrc pointing to <MY_NGT_DIR>/lib/ folder and run source ~/.bashrc.
Finally, I run "pip install ngt". I tried to run the ctype example code but it gave me error like below:

AttributeError: python: undefined symbol: ngt_open_index

However, if I put the example code file in the <MY_NGT_DIR>/lib/ folder, it runs without problems.

When building with cmake -DNGT_SHARED_MEMORY_ALLOCATOR=ON .., the reconstruct-graph command doesn't get built.

Running:
ngt reconstruct-graph
Returns:
ngt::reconstructGraph: Not implemented
Aborted (core dumped)

Can the reconstruct-graph command be run on an shared memory ANNG? Thanks! This is a cool system

I did not know this exactly pytorch issue or ngt issue. I also submit this issue in pytorch
pytorch/pytorch#26405 (comment)

🐛 Bug

• I can import pytorch properly
  

• After import other library (ngtpy, flair, or other), it raised segfault core dump
  

• If I swap the import (ngtpy first), it will raised free(): invalid pointer.
  

I did not know what happened. I use ngt (https://github.com/yahoojapan/NGT)

To Reproduce

Steps to reproduce the behavior:

1. Import torch, then other library -> Segfault
2. Import other library, then torch -> Free(): invalid pointer

Expected behavior

can import properly

Environment

PyTorch version: 1.2.0
Is debug build: No
CUDA used to build PyTorch: 10.0.130

OS: Ubuntu 18.04.3 LTS
GCC version: (Ubuntu 7.4.0-1ubuntu1~18.04.1) 7.4.0
CMake version: version 3.10.2

Python version: 3.6
Is CUDA available: Yes
CUDA runtime version: 10.0.130
GPU models and configuration:
GPU 0: Tesla V100-DGXS-32GB
GPU 1: Tesla V100-DGXS-32GB
GPU 2: Tesla V100-DGXS-32GB
GPU 3: Tesla V100-DGXS-32GB

Nvidia driver version: 410.104
cuDNN version: /usr/lib/x86_64-linux-gnu/libcudnn.so.7.5.0

Versions of relevant libraries:
[pip3] numpy==1.17.2
[pip3] pytorch-transformers==1.2.0
[pip3] torch==1.2.0
[conda] blas                      1.0                         mkl
[conda] mkl                       2019.3                      199
[conda] mkl-service               1.1.2            py37he904b0f_5
[conda] mkl_fft                   1.0.10           py37ha843d7b_0
[conda] mkl_random                1.0.2            py37hd81dba3_0

Additional context

I inspect it using gdb, and the backtrace output:

#0  0x00007fff1eaf3e3e in pybind11::detail::make_new_python_type (rec=...) at /opt/python/cp36-cp36m/include/python3.6m/pybind11/detail/class.h:565
#1  0x00007fff1eaf938e in pybind11::detail::generic_type::initialize (this=this@entry=0x7fffffffc900, rec=...) at /opt/python/cp36-cp36m/include/python3.6m/pybind11/pybind11.h:902
#2  0x00007fff1eac1a97 in pybind11::class_<Index>::class_<> (name=0x7fff1eb61378 "Index", scope=..., this=0x7fffffffc900)
    at /opt/python/cp36-cp36m/include/python3.6m/pybind11/pybind11.h:1092
#3  pybind11_init_ngtpy (m=...) at src/ngtpy.cpp:436
#4  0x00007fff1eac34d0 in PyInit_ngtpy () at src/ngtpy.cpp:421
#5  0x00000000005e4268 in _PyImport_LoadDynamicModuleWithSpec ()
#6  0x00000000005e4522 in ?? ()
#7  0x000000000056246e in PyCFunction_Call ()
#8  0x00000000004fed26 in _PyEval_EvalFrameDefault ()
#9  0x00000000004f6128 in ?? ()
#10 0x00000000004f7d60 in ?? ()
#11 0x00000000004f876d in ?? ()
#12 0x00000000004f98c7 in _PyEval_EvalFrameDefault ()
#13 0x00000000004f7a28 in ?? ()
#14 0x00000000004f876d in ?? ()
#15 0x00000000004f98c7 in _PyEval_EvalFrameDefault ()
#16 0x00000000004f7a28 in ?? ()
#17 0x00000000004f876d in ?? ()
#18 0x00000000004f98c7 in _PyEval_EvalFrameDefault ()
#19 0x00000000004f7a28 in ?? ()
#20 0x00000000004f876d in ?? ()
#21 0x00000000004f98c7 in _PyEval_EvalFrameDefault ()
#22 0x00000000004f7a28 in ?? ()
#23 0x00000000004f876d in ?? ()
#24 0x00000000004f98c7 in _PyEval_EvalFrameDefault ()
#25 0x00000000004f4065 in _PyFunction_FastCallDict ()
#26 0x000000000057c8f1 in _PyObject_FastCallDict ()
#27 0x000000000057cc5e in _PyObject_CallMethodIdObjArgs ()
#28 0x00000000004cf5dd in PyImport_ImportModuleLevelObject ()
#29 0x00000000004fb864 in _PyEval_EvalFrameDefault ()
#30 0x00000000004f6128 in ?? ()
#31 0x00000000004f9023 in PyEval_EvalCode ()
#32 0x00000000006415b2 in ?? ()
#33 0x000000000064166a in PyRun_FileExFlags ()
#34 0x0000000000643730 in PyRun_SimpleFileExFlags ()
#35 0x000000000062b26e in Py_Main ()
#36 0x00000000004b4cb0 in main ()

We've met some bad cases.

search top 120 minimun (1-cos(x,y)) in a dataset of size 100,000 which is 33 dimensions float costs 840ms. CPU xeon e5

when creating index, some warnings occur: LeafNode::splitObjects: Too many same distances. Reduce internal children size for the tree index or not use the tree index.

Does NGT support fast index build and query on GPU, or a mixed CPU and GPU?
Thanks a lot.

I tried NGT in some data, and I am very impressed by its recall and speed. I noticed that most of the examples are for topK, and I am wondering this algorithm is fit for range_search – search for all vectors within a specific radius of a given vector.

In the python wrapper, the radius setting is not exposed https://github.com/yahoojapan/NGT/blob/master/python/src/ngtpy.cpp#L126. However, according to the C++ example https://github.com/yahoojapan/NGT/blob/master/bin/search/search.cpp#L54, we can set the radius along with size.

Is this algorithm fit for range_search (by make size=data_size and control radius)? If so, do we have data compared with other approaches (classic inverted files, vp-tree, or other approaches)? Thanks!

I was trying to build memory-shared index from 102GB tsv file. However, it came out with the following error.

The machine I am using has a RAM of 250GB, and I was able to build non-shared version index from 200+GB tsv file before.

It seems that there is a limit of memory size in the memory-shared version?

alloc size over. size=1073741824.
Fatal Error: Allocating memory size is too big for this settings.
             Max allocation size should be enlarged.
Aborted (core dumped)

I install ngt using pip3 install ngt and on running the sample code - https://github.com/yahoojapan/NGT/blob/master/python/sample/sample.py -- I get the following error

  File "sample.py", line 38, in <module>
    print('# of distance computations=' + str(index.get_num_of_distance_computations()))
AttributeError: 'ngtpy.Index' object has no attribute 'get_num_of_distance_computations'

1. How is number of distance defined?
2. Is there any other way to calculate the number of distances?

I have installed both base and python packages successfully. However, when I try to run the sample code in python/README.md, it throws error as below:

Traceback (most recent call last):
  File "ngt_sample.py", line 11, in <module>
    index = ngt.Index.create("tmp", dim)
  File "/root/workspace/NGT-1.3.1/python/ngt/base.py", line 250, in create
    index = Index.__ngt.ngt_create_graph_and_tree(path, prop, err)
ctypes.ArgumentError: argument 1: <class 'TypeError'>: wrong type

Since this sample code does not work, I cannot find out how to use NGT to create an index...

If we build ngpty manually (after building NGT C++ manually), the Python version is not consistent with the C++ version.

Building sdist is fine:

root@laptop:/NGT-1.10.0/python# python setup.py sdist --static-library
use the NGT static library
/usr/local/lib/python3.7/site-packages/setuptools/dist.py:454: UserWarning: Normalizing '1.10.0
' to '1.10.0'
  warnings.warn(tmpl.format(**locals()))
running sdist
running egg_info
writing ngt.egg-info/PKG-INFO
writing dependency_links to ngt.egg-info/dependency_links.txt
writing requirements to ngt.egg-info/requires.txt
writing top-level names to ngt.egg-info/top_level.txt
reading manifest file 'ngt.egg-info/SOURCES.txt'
writing manifest file 'ngt.egg-info/SOURCES.txt'
running check
creating ngt-1.10.0
creating ngt-1.10.0/ngt
creating ngt-1.10.0/ngt.egg-info
creating ngt-1.10.0/src
copying files to ngt-1.10.0...
copying README.md -> ngt-1.10.0
copying setup.py -> ngt-1.10.0
copying ngt/__init__.py -> ngt-1.10.0/ngt
copying ngt/base.py -> ngt-1.10.0/ngt
copying ngt.egg-info/PKG-INFO -> ngt-1.10.0/ngt.egg-info
copying ngt.egg-info/SOURCES.txt -> ngt-1.10.0/ngt.egg-info
copying ngt.egg-info/dependency_links.txt -> ngt-1.10.0/ngt.egg-info
copying ngt.egg-info/requires.txt -> ngt-1.10.0/ngt.egg-info
copying ngt.egg-info/top_level.txt -> ngt-1.10.0/ngt.egg-info
copying src/ngtpy.cpp -> ngt-1.10.0/src
Writing ngt-1.10.0/setup.cfg
Creating tar archive
removing 'ngt-1.10.0' (and everything under it)

Installing deps is fine:

root@laptop:/NGT-1.10.0/python# pip install numpy pybind11
Collecting numpy
  Using cached numpy-1.18.3-cp37-cp37m-manylinux1_x86_64.whl (20.2 MB)
Collecting pybind11
  Using cached pybind11-2.5.0-py2.py3-none-any.whl (296 kB)
Installing collected packages: numpy, pybind11
Successfully installed numpy-1.18.3 pybind11-2.5.0

However, when installing dist/ngt-1.10.0.tar.gz it will always use version 1.8.0:

root@laptop:/NGT-1.10.0/python# pip install dist/ngt-1.10.0.tar.gz
Processing ./dist/ngt-1.10.0.tar.gz
Requirement already satisfied: numpy in /usr/local/lib/python3.7/site-packages (from ngt==1.8.0) (1.18.3)
Requirement already satisfied: pybind11 in /usr/local/lib/python3.7/site-packages (from ngt==1.8.0) (2.5.0)
Building wheels for collected packages: ngt
  Building wheel for ngt (setup.py) ... done
  Created wheel for ngt: filename=ngt-1.8.0-cp37-cp37m-linux_x86_64.whl size=2163885 sha256=d99c8d886f4cf683b1baa6d1746eccc8b71f0b97f974e3347296e3d4097abb44
  Stored in directory: /root/.cache/pip/wheels/f7/49/42/c4dfb80dde389e6625f9ce343ea83054debfe23687ec21e49b
Successfully built ngt
Installing collected packages: ngt
Successfully installed ngt-1.8.0

I think this is due to the hardcoded version = '1.8.0' in setup.py.

Moreover there is no variable ngt.__version__ to check what version we are using in Python, this is a bit confusing.

Unfortunately, this NGT package does not seem to support installation on the loongson CPU(mips) self-designed by China.

I am new to this project. Probably this question has been discussed before.

When I pull this repository and compile. It has shown copy elision warning as below:

In file included from /Users/lit/github/NGT/lib/NGT/Command.cpp:20:
/Users/lit/github/NGT/lib/NGT/Optimizer.h: In member function 'NGT::Optimizer::MeasuredValue NGT::Optimizer::measure(std::istream&, std::istream&, NGT::Command::SearchParameter&, std::pair<float, float>, double)':
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  620 |       return std::move(v);
      |              ~~~~~~~~~^~~
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: note: remove 'std::move' call
[ 14%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Graph.cpp.o
[ 17%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Index.cpp.o
[ 21%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/MmapManager.cpp.o
[ 25%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Node.cpp.o
[ 28%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/SharedMemoryAllocator.cpp.o
[ 32%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Thread.cpp.o
[ 35%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/Tree.cpp.o
[ 39%] Linking CXX static library libngt.a
[ 39%] Built target ngtstatic
Scanning dependencies of target ngt
[ 42%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/ArrayFile.cpp.o
[ 46%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Capi.cpp.o
[ 50%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Command.cpp.o
In file included from /Users/lit/github/NGT/lib/NGT/Command.cpp:20:
/Users/lit/github/NGT/lib/NGT/Optimizer.h: In member function 'NGT::Optimizer::MeasuredValue NGT::Optimizer::measure(std::istream&, std::istream&, NGT::Command::SearchParameter&, std::pair<float, float>, double)':
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  620 |       return std::move(v);
      |              ~~~~~~~~~^~~
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: note: remove 'std::move' call
[ 53%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Graph.cpp.o
[ 57%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Index.cpp.o
[ 60%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/MmapManager.cpp.o
[ 64%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Node.cpp.o
[ 67%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/SharedMemoryAllocator.cpp.o
[ 71%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Thread.cpp.o
[ 75%] Building CXX object lib/NGT/CMakeFiles/ngt.dir/Tree.cpp.o
[ 78%] Linking CXX shared library libngt.dylib
[ 78%] Built target ngt
Scanning dependencies of target ngt_exe
[ 82%] Building CXX object bin/ngt/CMakeFiles/ngt_exe.dir/ngt.cpp.o
In file included from /Users/lit/github/NGT/bin/ngt/ngt.cpp:18:
/Users/lit/github/NGT/lib/NGT/Optimizer.h: In member function 'NGT::Optimizer::MeasuredValue NGT::Optimizer::measure(std::istream&, std::istream&, NGT::Command::SearchParameter&, std::pair<float, float>, double)':
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: warning: moving a local object in a return statement prevents copy elision [-Wpessimizing-move]
  620 |       return std::move(v);
      |              ~~~~~~~~~^~~
/Users/lit/github/NGT/lib/NGT/Optimizer.h:620:23: note: remove 'std::move' call
[ 85%] Linking CXX executable ngt
[ 85%] Built target ngt_exe
Scanning dependencies of target search
[ 89%] Building CXX object bin/search/CMakeFiles/search.dir/search.cpp.o
[ 92%] Linking CXX executable search
[ 92%] Built target search
Scanning dependencies of target ngtq_exe
[ 96%] Building CXX object bin/ngtq/CMakeFiles/ngtq_exe.dir/ngtq.cpp.o
[100%] Linking CXX executable ngtq

It is not an important thing, but should we let the compiler to do RVO instead?

Hi I meet a problem when using python library. The below is the error message. Do I have to build from scratch before I install the python library by pip?
(Just in case I forgot to install any dependency.)

>>> from ngt import base
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "ngt/base.py", line 42, in <module>
    class Index(object):
  File "ngt/base.py", line 88, in Index
    __ngt.ngt_open_index.argtypes = [c_char_p, c_void_p]
  File "/usr/lib/python2.7/ctypes/__init__.py", line 375, in __getattr__
    func = self.__getitem__(name)
  File "/usr/lib/python2.7/ctypes/__init__.py", line 380, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: python: undefined symbol: ngt_open_index

Hi,
I'm using cosine similarity as the distance metrics. I set output size is 15 and I saw that the distance is increasing. This is confusing me because if the angle between 2 points is very small then their cosine will reach 1. How can I change the arrangement of the output by descending order when using cosine similarity metric ?
Thanks in advances !
Dong
{
Rank ID Name Distance
1 137 NVH1 0.478774
2 133 NVH2 0.494715
3 136 NVH2 0.498162
4 135 NVH2 0.520622
5 134 NVH2 0.522601
6 138 NVH2 0.536414
7 851 TVX 0.644147
8 2340 DTV 0.647083
9 3338 HND 0.65958
10 1012 NTH 0.662446
11 1604 HMH 0.66996
12 958 NMN 0.672668
13 3233 NHH 0.676299
14 2611 HMH 0.685966
15 1013 NMJ 0.688811

}

How can I build ngtpy with pybind11 from source? Following the instruction from the readme, I am only able to build ngt with ctypes.

Background: I'd like to include ngtpy in one of my projects. On my machine I can simply pip install ngt, and I get ngtpy. However, the same gives me segmentation faults and others errors on Travis CI. So I try to install from source, but I only manage to get ngt (ctypes).

Sometimes, its helpful to be able to incrementally add (and sometime delete) to a NN map. Is this supported?

Hi, thanks for this awesome library 🎉 Just wanted to let you know there are now Ruby bindings for NGT (thanks to the C API).

I was hoping to support ONNG and wondering if there's a way to call reconstruct-graph with the C API. It looks like Python has support with pybind11 and Optimizer class. If not, what are your thoughts on adding it to the C API?

I also submitted a pull request to Homebrew to make installation even faster/easier on Mac: Homebrew/homebrew-core#45704

Hello,

I would like to ask a question about removing nodes from an existing index.

When I remove nodes from my index I receive errors like this:
Graph::removeEdgeReliably: Lost conectivity! Isn't this ANNG? ID=2 anyway continue...

and also
/NGT-1.9.1/lib/NGT/Index.h:1622: remove:: cannot remove from tree. id=5 /NGT-1.9.1/lib/NGT/Tree.h:191: VpTree::remove: Inner error. Cannot remove object. leafNode=1693:/NGT-1.9.1/lib/NGT/Node.cpp:260: VpTree::Leaf::remove: Cannot find the specified object. ID=5,0 idx=21 If the same objects were inserted into the index, ignore this message

After removing a subset of nodes from an existing index, will running index.build_index() restore connectivity?

If I receive failures when attempting to remove a node, do I need to build a new index using a dataset that excludes the node(s)?

An example of my current usage could be:

# initial creation    
# 65 thousand items added to index, index built and saved
# at a later time, we update the index

# load the existing index within a python3 application.    
index = ngtpy.Index(index_path)

# process that updates index
index.remove(0)
# Graph::removeEdgeReliably: Lost conectivity! Isn't this ANNG? ID=2 anyway continue...
index.remove(1)
index.remove(2)
/NGT-1.9.1/lib/NGT/Index.h:1622: remove:: cannot remove from tree. id=5 /NGT-1.9.1/lib/NGT/Tree.h:191: VpTree::remove: Inner error. Cannot remove object. leafNode=1693:/NGT-1.9.1/lib/NGT/Node.cpp:260: VpTree::Leaf::remove: Cannot find the specified object. ID=5,0 idx=21 If the same objects were inserted into the index, ignore this message

# in my test I removed the first 10 nodes I previously added when building the index

# at end of update process  
index.build_index()
index.save()

# Is connectivity repaired for our index? Were all nodes removed successfully? 

My current test index is 65 thousand nodes. My use case is to create an index of 3-5 millions nodes, periodically remove a small subset of nodes and add new nodes. I am hoping to avoid inserting millions of nodes, rebuilding a fresh index every time.

Issues that cite adding and removing nodes support:

https://github.com/yahoojapan/NGT/issues/38
https://github.com/yahoojapan/NGT/issues/19

Thank you for your time and consideration. Please let me know if there is any additional information that would help address my question.

Regards,
Erik

I'm following the install instructions for macos, but when I get to make it returns:

[  2%] Built target versiondef
[  5%] Building CXX object lib/NGT/CMakeFiles/ngtstatic.dir/ArrayFile.cpp.o
clang: warning: -lrt: 'linker' input unused [-Wunused-command-line-argument]
clang: error: unsupported option '-fopenmp'
make[2]: *** [lib/NGT/CMakeFiles/ngtstatic.dir/ArrayFile.cpp.o] Error 1
make[1]: *** [lib/NGT/CMakeFiles/ngtstatic.dir/all] Error 2
make: *** [all] Error 2

I've tried recompiling GCC-9.2

Hi,
do you have any example of ngt usage on C++ with hamming distance?

I'm currently creating an index with this Properties for use with a 128 bits query:

NGT::Property property;
    property.setDefault();
    property.dimension = 8; // 4bytes per element x 8 elements 
    property.distanceType = NGT::Property::DistanceType::DistanceTypeHamming;
    property.indexType = NGT::Property::IndexType::GraphAndTree;
    property.objectType = NGT::Property::ObjectType::Uint8;

While it does compile and run, no result is found. I'm appending data as float which should be the same size as an uint but maybe that's incorrect.

const float *p = desc.ptr<float>(0);
        std::vector<float> obj(p, p + 8);

        // allocate query object.
        query = index->allocateObject(obj);

        NGT::SearchContainer sc(*query);     

Using L2 and floats works fine so i'm probably doing something wrong on my end, an example would really help me.

Thanks in advance.

This is the first product of this kind that i've got up and running over existing heavyweight VSR dump (multilayer embeddings from image recognition CNN-s). Spent not more than 30 minutes on interface replacement toward NGTd. Now no buggy vendor drivers, kdumps OOMS and other common problems Everything is VERY convinient. (I've used FAISS, Annoy and some in-home Yandex stuff previously)

Algorythmic part is exceptional, especially PQ post-verification trick, finally there is no need to investigate precision problems for cases of irrelevant output and focus on VSR construction logic.

I wish all luch to NGT

Thanks for publishing precompiled wheels for Linux and MacOS. Would it be possible to publish wheels for Windows as well?

Thanks!

I want an exact nearest neighbor search with 100% recall within considerable time. Speed is OK around 0.1s.
Any suggestions on parameter tuning? Thanks a lot.

I am trying to evaluate NGT library for Python under Mac OS X Catalina 10.15.2 with Visual Studio Code.

1. "pip3 install ngt" does not work for Python 3.8.x
2. for Python version 3.7.5 and below it works

Hi, is the object type "Byte" well developed and fully tested in the current code? By the way, I assume the algorithms work for custom distance function, am I right? Thank you!

inner product distance support?

can index be optimized by multithread？

ObjectID of each record seems to be specified using sequence number of each line.
Is there any way to specify ObjectID using user defined id?

The performance of the index is impressive. When I load a dumped index from disk, can I re-insert other objects(ndarray) in the index and re-build it?