Twitter tokenizer about anserini HOT 4 CLOSED

@LuchenTan Here's an example of a test case you can adapt for tokenization:
https://github.com/lintool/twitter-tools/blob/master/twitter-tools-core/src/test/java/cc/twittertools/index/TokenizationTest.java

from anserini.

My tokenizer would result different as this tokenizer. For example, no
stemmer performed, url removal. Anyway, I will update the tokenizer
tonight, and modify this test class.

On Mon, Nov 2, 2015 at 3:49 PM, Jimmy Lin [email protected] wrote:

@LuchenTan https://github.com/LuchenTan Here's an example of a test
case you can adapt for tokenization:

https://github.com/lintool/twitter-tools/blob/master/twitter-tools-core/src/test/java/cc/twittertools/index/TokenizationTest.java

—
Reply to this email directly or view it on GitHub
#38 (comment).

Luchen Tan

David R. Cheriton School of Computer Science
University of Waterloo
Waterloo, Ontario, Canada N2L 3G1

from anserini.

@LuchenTan Yes, the result of the tokenization will be different, but you can still use the general format of the test case.

from anserini.

Not much interest in working on tweet search (anymore)...

from anserini.