[ENHANCEMENT] filtered search on hnsw about marqo HOT 3 OPEN

Hi @patelprateek is this a request for more better documentation?

from marqo.

Sure , that would help a lot , right now i dont have visibility into what kind of attribute index are being created for different types (scala float, int or categorical) . I can have been using hnswlib opensource lib and using their documentation to get estimates for my ann index size , but the attribute index and how it is exactly used to do filtered search would be helpful for indexing and creating proper queries

'Are there any docs or code i can look at for understanding how hnsw index is created for a scalar attribute and how it exactly helps with filtered search , my initial assumption are we build some inverted index for attributes and then use that index to get filtered eligible results that is used during ann search , but from documentation it seems not to be the approach here

from marqo.

Hi @patelprateek , sorry for the slow response! Marqo uses OpenSearch as a backend, and float, int and text data are indexed as OpenSearch-defined indexes on those datatypes.

The algorithm used for filtering is Lucene's k-NN filtering algorithm: https://opensearch.org/docs/latest/search-plugins/knn/filter-search-knn/#lucene-k-nn-filter-implementation

from marqo.