Server crash when building an ivfflat index with a high number of clusters about pgvector HOT 2 CLOSED

Hey @ArthurMelin, thanks for the great reporting and debugging! Just pushed a fix.

A few notes to self:

• lowerBound indexing overflows at ~6500 lists since 6500 * (6500 * 50) is around 2^31 (centers * (centers * samples per center))
• halfcdist indexing does not overflow since 32768 * 32768 is less than 2^31 (IVFFLAT_MAX_LISTS * IVFFLAT_MAX_LISTS), but could if IVFFLAT_MAX_LISTS is increased

from pgvector.

Found the issue, it's integer overflows in ivfkmeans.c for array accesses:
https://github.com/pgvector/pgvector/blob/master/src/ivfkmeans.c#L63
and similar others in ElkanKmeans().

I fixed it by changing the type of j and k to int64. @ankane do you want me to make a PR?

from pgvector.