Tansaku

Building a search engine from scratch using JavaScript and some math.

Installation

Clone and install everything.

$ git clone https://github.com/gmittal/tansaku
$ cd tansaku
$ [sudo] npm install

Crawling for Data

$ node crawl

Searching

$ node app "query"

tansaku's People

Watchers

tansaku's Issues

Add lemmatization

Lemmatize the words in both the query vector and the document vectors to produce better results.

Add better tokenization

Implement better word tokenization and text preprocessing for cleaner vectors.

Remove linear growth search algorithm

The ultimate ranking tool uses vector cosine similarity, but in order to get those results, the program currently looks through every single document (bad bad bad). Use a function that has a better order of growth. It would be pretty simple to add a search function that has a logarithmic order of growth.

Add tf-idf weightings for better relevancy

This would improve results drastically. Would have to account for fillers/stop words ("a", "this", "the"), which appear a lot but aren't actually that useful without removing them completely (e.g. "to be or not to be" is a relevant query composed of only stop words).

Add spelling correction

Would be a nifty feature to have. Peter Norvig has a great article on how to implement it.

Fix memory leaks

Something is getting stored in RAM instead of on the disk during the crawl process.

This keeps happening:

(node:77659) Warning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 pipe listeners added. Use emitter.setMaxListeners() to increase limit
(node:77659) Warning: Possible EventEmitter memory leak detected. 11 end listeners added. Use emitter.setMaxListeners() to increase limit

Recommend Projects