BUG when autocompleting unicode strings about redisearch HOT 7 CLOSED

Fixed as far as I can test (haven't run your test code though). Feel free to verify.

from redisearch.

Fix confirmed! Thanks!

from redisearch.

Going over this, it looks like working with 32 bit runes will not be too hard to do, and will allow full fuzzy support in unicode. but it will make memory consumption terrible. Is having unicode supported fuzzy matching critical for you?

from redisearch.

It's essential, yes. How much would the memory requirements increase?

from redisearch.

The idea is not to use variable length encoding like UTF-8 and UTF-16, and use a fixed length encoding, probably 32 bit - per letter.

So for pure latin text, which in utf-8 is represented by 1 byte, you would have to use 4 bytes. Some of the memory consumption is of course pointers and metadata, so we are talking about x2-x3 the amount of RAM.

For purely non-latin, which takes up 2-3 bytes per letter in utf-8, it would probably be x1.5-x2.

If I manage to get away with 16 bit per letter, which won't cover all languages but will cover the most popular ones IIRC, it won't be so bad.

from redisearch.

Yeah, 16 bits should cover all the languages we are aiming at so that's okay.

from redisearch.

yeah, range 0x0000-0xFFFF covers everything a sane person would need. https://en.wikipedia.org/wiki/Plane_(Unicode)#Basic_Multilingual_Plane

Although Unicode and sanity don't go hand in hand, in this case it might be a good compromise.

from redisearch.