能够支持中文吗 about elasticlunr.js HOT 3 CLOSED

Hi, @lierisme , if you want to support Chinese, you just need to update the tokenizer, you could find a Chinese tokenizer, or, you could just tokenize Chinese by each word.
Chinese tokenizer is a complex package, so if you want to support a good chinese tokenizer, I think elasticlunr.js should not be run in browser

from elasticlunr.js.

I replace the default to use jieba, but I do not why "Pipeline" return a empty array.

from elasticlunr.js.

@caihaibin1991

For supporting Chinese ,

1. Change tokenizer or do tokenization before elasticlunr's default tokenizer. For example, "能够支持中文吗"->"能够 支持 中文 吗", after this preprocessing, the default tokenizer can work by split space. For tokenization, nodejieba, @node-rs/jieba or regex to split every character can be used.

2. [Your issue] Remove default elasticlunr.trimmer, by index.pipeline.remove(elasticlunr.trimmer). The default trimmer trims all non-english characters, which results your issue.

3. Further work. Make your own pipeline to process text, including tokenizer, trimmer, stemmer and stopword filter. Examples are in weixsong/lunr-languages.

from elasticlunr.js.