Feature request: don't index stop words about pagefind HOT 1 CLOSED

Any stop word handling is actually something I removed before release — I can talk through why.

Reason №1 that people implement stop words is to make the index smaller, which isn't too much of a concern for Pagefind due to the chunking strategy. Since chunks are roughly fixed sizes, we can afford to care about stop words.

Reason №2 is to improve search ranking, so that searching for the editor cares more about editor than the. Removing stop words altogether is a heavy handed way of solving this, and the planned implementation of word ranking with BM25 will de-rank the the automatically without needing to put it on a list, while still keeping it in the index.

The biggest reasons to continue indexing all words is that it improves our exact phrase matching — without stop words searching for "around the world" might return results with any other word in the middle. Also, it's hard to find stop word lists that are reliably safe to ignore — especially for documentation sites where a lot of common stop words are important to search for. The first list I used had date as a stop word, for example. Or maybe you want to search MDN for let.

In any case the word weighting work will provide everything we need around improving the search rankings, without harming search precision 🙂

from pagefind.