research search libraries for implementing searchIndex spec about stremio-core HOT 4 OPEN

simsearch: turns out it's a good basis but it has a very poor scoring system; definitely something that can be easily improved

using Ivshti/simsearch-rs@492d1ec, I performed a quick test to find The Office and Game of Thrones in 10k documents

simsearch perf, WASM

10k dataset of cinemeta items
two searches were performed for the "search" timing

in conclusion, we are OK to do a separate search on each keystroke

Firefox

indexing: 486ms - timer ended
searching: 13ms - timer ended

indexing: 419ms - timer ended
searching: 11ms - timer ended

Chrome

indexing: 536.856689453125ms
searching: 20.677978515625ms

indexing: 471.500244140625ms
searching: 22.561767578125ms

from stremio-core.

A new API can be created, where the SimSearch is created with a vector of items, and an index and rank closures

It should own that vector of items, and use the item's position in the vector as the ID

this will also eliminate the need for a forward map, since that's only used when deleting. And so, it will make indexing faster (2x?)

Searching actors could be implemented by making a separate instance (searchIndex catalog) for actors, that, upon opening each actor, points to a particular catalog with an extra param (byActor=...); the UI should handle that by opening it in Discover

from stremio-core.

Starting from simsearch-rs, we can make this usable for our purpose:

• Make an immutable index API version
• Inverse frequency (to implement tf-idf) by storing each doc score for a token ; may use a tree so that we can have token -> doc -> score; this would also allow popularity boosting
• New scoring: TF-IDF and maybe n-gram
• Automated tester that enters popular things.. partially, fully and misspelled
• Proper stop words and/or stemmer
• Optional prefix search (autocomplete) for the last term
• Cinemeta will use a made up “mixed” type for its catalog named “movies and series”; so that those will be searched and ranked together

from stremio-core.

Some classic movies are outside the top 10k by moviedb popularity, so in conclusion: we must index a larger set, or by a different factor; or a combo $or query

• top 15-20k by stremio
• OR db.metadata.count({'popularities.stremio': {$gt: 0.15}}); that's 18135 items as of 17.08.2019
• OR an $or query where the other condition is some combo between imdbRating and trakt popularity

from stremio-core.