hideaki-t / whoosh-igo Goto Github PK
View Code? Open in Web Editor NEWtokenizers for Whoosh designed for Japanese language
License: Apache License 2.0
tokenizers for Whoosh designed for Japanese language
License: Apache License 2.0
================================ Japanese Tokenizers for Whoosh ================================ About ===== Tokenizers for Whoosh full text search library designed for Japanese language. This package contains two Tokenizers. * IgoTokenizer + requires igo-python(http://pypi.python.org/pypi/igo-python/) and its dictionary. * TinySegmenterTokenizer + requires TinySegmenter (https://pypi.python.org/pypi/tinysegmenter3) * MeCabTokenizer * requires one of MeCab python bindings https://pypi.python.org/pypi/mecab-python or https://pypi.python.org/pypi/mecab-python3 How To Use ========== IgoTokenizer:: import igo.Tagger import whooshjp from whooshjp.IgoTokenizer import IgoTokenizer tk = IgoTokenizer() scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk)) TinySegmenterTokenizer:: import tinysegmenter import whooshjp from whooshjp.TinySegmenterTokenizer import TinySegmenterTokenizer tk = TinySegmenterTokenizer() scm = Schema(title=TEXT(stored=True, analyzer=tk), path=ID(unique=True,stored=True), content=TEXT(analyzer=tk)) Note ==== IgoTokenizer ------------ Whoosh stores a schema including a tokenizer into an index. If the stored schema includes the dictionary used in IgoTokenizer, the size of index might be too large. Storing a schema into an index is a great idea. You just need an index to search. Even your dictionary has been updated, the index has the dictionary used at indexing time. Also an error will be happened when opening an index with mmap mode IgoTokenizer, since mmaped dictionary will not be stored into an index. If you already built an index with mmap mode IgoTokenizer, you can open it by passing a schema to whoosh.index.open_dir. e.g. open_dir(path_to_index, schema=an_schema) To avoid storing *full* tokenizer, IgoTokenizer has 2 instantiation modes. 1. passing an igo tagger object. i.e. IgoTokenizer(IgoTagger(path='dic')). an instance created by this will store everything including loaded dictionary. 2. passing arguments for instantiating an igo tokenizer. i.e. IgoTokenizer(path='dic'). Only the arguments will be stored, and instantiate an igo tokenizer with the arguments when an index which uses IgoTokenizer with this mode is opened. If no arguments are not given to IgoTokenizer initializer, i.e. IgoTokenizer(), it is also considered as a this mode, an IgoTagger will be instantiated without any arguments.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.