An Open Source Japanese NLP Library based on Universal Dependencies

GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models are distributed under The MIT License. You must agree and follow The MIT License to use GiNZA NLP Library and GiNZA Japanese Universal Dependencies Models.

spaCy is the key framework of GiNZA. spaCy LICENSE PAGE

SudachiPy provides high accuracies for tokenization and pos tagging. Sudachi LICENSE PAGE, SudachiPy LICENSE PAGE

This project is developed with Python 3.7 and pip for it.

The footprint of this project is about 250MB. Sudachi dictionary is 200MB. The word embeddings from entire Japanese Wikipedia is 50MB.

(Please see Development Environment section located on bottom too)

Run following line

pip install "https://github.com/megagonlabs/ginza/releases/download/v1.0.1/ja_ginza_nopn-1.0.1.tgz"

or download pip install archive from release page and specify it as below.

pip install ja_ginza_nopn-1.0.1.tgz

Run following line and input some Japanese text + Enter, then you can see the parsed results with conll format.

python -m spacy.lang.ja_ginza.cli

Following steps shows dependency parsing results with sentence boundary 'EOS'.

import spacy
nlp = spacy.load('ja_ginza_nopn')
doc = nlp('依存構造解析の実験を行っています。')
for sent in doc.sents:
    for token in sent:
        print(token.i, token.orth_, token.lemma_, token.pos_, token.dep_, token.head.i)
    print('EOS')

Please see spaCy API documents.

Add new Japanese era 'reiwa' to system_core.dic.

First release version

git clone --recursive 'https://github.com/megagonlabs/ginza.git'

For normal environment:

./setup.sh

For GPU environment(cuda92):

./setup_cuda92.sh

Prepare nopn_embedding/, nopn/, and kwdlc/ in your project directory, then run below. (We're preparing the descriptions of training environment. Coming soon.)

shell/build.sh nopn 1.0.1

You can speed up training and analyze process by adding -g option if GPUs available.

After a while, you will find pip installable archive.

target/ja_ginza_nopn-1.0.1.tgz