Comments (7)
Since I found that v2.0.15 does not work well in some cases, I have released v2.0.16 to fix it.
QG and QBG can handle float16 as well.
from ngt.
Hello,
V2.0.13 has been released and includes the C APIs you requested.
internal_data_type
is the vector data type in the graph for QBG. You should always set float
to internal_data_type
and data_type
, because testing against other data types is not sufficient.
dimension
must be set as the dimension of the given data. When the rotation is enabled, the given vector can be extended inside QBG to improve quantization errors. extended_dimension
should be equal to or larger than dimension
, and should be a multiple of 16.
from ngt.
Hello,
Thank you for the updates and the explanations.
I was able to use qbg_append_object_as_uint8
and qbg_get_object_as_uint8
on my side.
However I am not sure to understand your answer regarding data_type
/internal_data_type
. If I look at the C API I can see that QBGConstructionParameters
is initialized as follows:
void qbg_initialize_construction_parameters(QBGConstructionParameters *parameters)
{
// ...
parameters->internal_data_type = NGTQ::DataTypeFloat;
parameters->data_type = NGTQ::DataTypeFloat;
// ...
}
So I thought that we could also use NGTQ::DataTypeUint8
to create a QBG containing uint8_t
objects. Is this correct ?
Or is the only possible value NGTQ::DataTypeFloat
?
Actually, if I try to set them both to NGTQ::DataTypeUint8
I get a SIGABRT with the following gdb backtrace:
Thread 9 "qbg::index::tes" received signal SIGABRT, Aborted.
[Switching to Thread 0x7fffdbffd6c0 (LWP 221840)]
0x00007ffff729f26c in ?? () from /usr/lib/libc.so.6
(gdb) bt
#0 0x00007ffff729f26c in ?? () from /usr/lib/libc.so.6
#1 0x00007ffff724fa08 in raise () from /usr/lib/libc.so.6
#2 0x00007ffff7238538 in abort () from /usr/lib/libc.so.6
#3 0x00005555557ae73a in NGTQ::QuantizerInstance<unsigned char>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) ()
#4 0x0000555555769b75 in NGTQ::QuantizerInstance<unsigned char>::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, NGT::Property&, bool)
()
#5 0x000055555578567f in NGTQ::Index::getQuantizer(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, NGT::Property&, bool) ()
#6 0x0000555555785b38 in NGTQ::Index::open(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) ()
#7 0x0000555555785c1e in QBG::Index::Index(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) ()
#8 0x0000555555785ff7 in qbg_open_index ()
#9 0x00005555555c8eef in ngt::qbg::index::QbgIndex<u8, ngt::qbg::index::ModeWrite>::create<u8, &std::path::Path> (path=..., create_params=...) at src/qbg/index.rs:50
from ngt.
Hello,
NGTQ::DataTypeUint8
is for the obsolete module NGTQ. I will consider reimplementing uint8
as a data type for QBG in the near future.
from ngt.
Hello,
I have released v2.0.15 which includes reimplementation of uint8
.
data_type
is the type of the stored objects. internal_data_type
is the type of the objects on the graph used by QBG. You can specify different data types for them.
from ngt.
Thanks a lot, it works fine now !
Looking at your commit it seems that QBG can also handle float16 ? Does QG handle float16 too ?
from ngt.
Thank you for the precision.
Then I guess some more functions would be needed in the C API for float16, I will list them in #141
from ngt.
Related Issues (20)
- Is there any benchmark result for NGT QG/QBG? HOT 4
- file descriptor leak on `index.build_index` HOT 2
- Missing functions and types in the C API HOT 14
- Python bindings for QG/QBG HOT 3
- bugs HOT 3
- Fixed seeds for deterministic results HOT 1
- Building with -DNGT_QBG_DISABLED=ON still trying to link with LAPACK and BLAS HOT 2
- How to update NGT from older version to new one? HOT 2
- pip install in python==3.11 fails : could not find a version that satisfies HOT 4
- Running sample codes (e.g., qg-l2-float) HOT 6
- Remove vector causes error, only for "Normalised" distance types: Index.h:remove:1544: Not found the specified id HOT 7
- Quantization in qg-l2-float.cpp HOT 1
- colab import ngt dose not work HOT 1
- Paper and references for QG HOT 1
- 2.2.0 failed to build on macos HOT 2
- Building bin/qbg is happening even with NGT_QBG_DISABLED HOT 2
- Missing bfloat16 property setter in C API HOT 1
- what tool to draw the image HOT 3
- One Build Error: _mm512_extract_epi64 is not supprted for some cpu versions. HOT 1
- One Bug
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ngt.