Supported datatypes about pgvector HOT 4 CLOSED

I forgot that this was also discussed indirectly upstream.

I guess the main difference is that pgvector types store just the length, oid and array data

Even less 😉 vector is dimension and the vector itself (with some bits used for handling variable length types and reserved). halfvec is similar, whereas sparsevec is set up for handling the absence of data.

The PostgreSQL array types have a bit more overhead, including OID.

So the vector types are a specialization for a single dimension that allow no nulls. Is that a good summary?

Yes -- that's a vector in a nutshell (magnitude and direction) 😄

While I do think it'd be great to have native vector search support in core PostgreSQL in the fullness of time, the current setup with pgvector allows us to go a bit faster. That said, there are likely some changes we need in core PostgreSQL to make things better (which I'm planning to discuss @ PGConf.dev 2024 in a few weeks), such as how we search over data in TOAST tables that happen to be part of the hot path of a query, or dealing with indexing data that goes beyond 8KB.

from pgvector.

@beikov I've spoken at great length numerous times on this topic 😄

The short answer is that pgvector provides indexable mechanisms for searching over vector data. The slightly longer answer is that while we're still sussing out the optimal ways to store vectors (storage format, indexing methods), pgvector can move a bit faster than the PostgreSQL project, which is released once a year. At the PostgreSQL developer conference last year, it this was also a strategic decision to let pgvector move faster than the project and get new vector support and functionality into the hands of developers faster.

The good news is that you can fast PostgreSQL array types to vector, and vice versa. Additionally, pgvector 0.7.0 added index support for bit strings, which are a native PostgreSQL type.

from pgvector.

Thanks for sharing that. I was not aware that you spoke about this topic in your talks.

I just found this documentation about how standard arrays are stored. I guess the main difference is that pgvector types store just the length, oid and array data. So the vector types are a specialization for a single dimension that allow no nulls. Is that a good summary?

from pgvector.

Thanks for clarification

from pgvector.