Code Monkey home page Code Monkey logo

underthesea's Introduction

Underthesea - Vietnamese NLP Toolkit

Documentation Status Updates

https://raw.githubusercontent.com/magizbox/underthesea/master/logo.jpg

underthesea is a suite of open source Python modules, data sets and tutorials supporting research and development in Vietnamese Natural Language Processing.

Installation

To install underthesea, simply:

$ pip install underthesea==1.1.6rc2
✨🍰✨

Satisfaction, guaranteed.

Usage

1. Word Segmentation

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import word_sent
>>> sentence = u"Chúng ta thường nói đến Rau sạch, Rau an toàn để phân biệt với các rau bình thường bán ngoài chợ."

>>> word_sent(sentence)
[u"Chúng ta", u"thường", u"nói", u"đến", u"Rau sạch", u",", u"Rau", u"an toàn", u"để", u"phân biệt", u"với",
u"các", u"rau", u"bình thường", u"bán", u"ngoài", u"chợ", u"."]

>>> word_sent(sentence, format="text")
u'Chúng_ta thường nói đến Rau_sạch , Rau an_toàn để phân_biệt với các rau bình_thường bán ngoài chợ .'

2. POS Tagging

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import pos_tag
>>> text = u"Chợ thịt chó nổi tiếng ở TP Hồ Chí Minh bị truy quét"
>>> pos_tag(text)
[(u'Chợ', 'N'),
 (u'thịt', 'N'),
 (u'chó', 'N'),
 (u'nổi tiếng', 'A'),
 (u'ở', 'E'),
 (u'TP HCM', 'Np'),
 (u'bị', 'V'),
 (u'truy quét', 'V')]

3. Chunking

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import chunk
>>> text = u"Bác sĩ bây giờ có thể thản nhiên báo tin bệnh nhân bị ung thư?"
>>> chunk(text)
[(u'Bác sĩ', 'N', 'B-NP'),
 (u'bây giờ', 'P', 'I-NP'),
 (u'có thể', 'R', 'B-VP'),
 (u'thản nhiên', 'V', 'I-VP'),
 (u'báo tin', 'N', 'B-NP'),
 (u'bệnh nhân', 'N', 'I-NP'),
 (u'bị', 'V', 'B-VP'),
 (u'ung thư', 'N', 'I-VP'),
 (u'?', 'CH', 'O')]

4. Named Entity Recognition

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import ner
>>> text = u"Chưa tiết lộ lịch trình tới Việt Nam của Tổng thống Mỹ Donald Trump"
>>> ner(text)
[('Chưa', 'R', 'O', 'O'),
 ('tiết lộ', 'V', 'B-VP', 'O'),
 ('lịch trình', 'V', 'B-VP', 'O'),
 ('tới', 'E', 'B-PP', 'O'),
 ('Việt Nam', 'Np', 'B-NP', 'B-LOC'),
 ('của', 'E', 'B-PP', 'O'),
 ('Tổng thống', 'N', 'B-NP', 'O'),
 ('Mỹ', 'Np', 'B-NP', 'B-LOC'),
 ('Donald', 'Np', 'B-NP', 'B-PER'),
 ('Trump', 'Np', 'B-NP', 'I-PER')]

5. Text Classification

Install dependencies and download default model

$ pip install Cython
$ pip install future scipy numpy scikit-learn
$ pip install -U fasttext --no-cache-dir --no-deps --force-reinstall
$ underthesea data

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import classify
>>> classify("HLV đầu tiên ở Premier League bị sa thải sau 4 vòng đấu")
['The thao']
>>> classify("Hội đồng tư vấn kinh doanh Asean vinh danh giải thưởng quốc tế")
['Kinh doanh']
>>> classify("Đánh giá “rạp hát tại gia” Samsung Soundbar Sound+ MS750")
['Vi tinh']

6. Sentiment Analysis

Install dependencies

$ pip install future scipy numpy scikit-learn==0.19.0 joblib

Usage

>>> # -*- coding: utf-8 -*-
>>> from underthesea import sentiment
>>> sentiment("Gọi mấy lần mà lúc nào cũng là các chuyên viên đang bận hết ạ")
('CUSTOMER SUPPORT#NEGATIVE',)
>>> sentiment("bidv cho vay hay ko phu thuoc y thich cua thang tham dinh, ko co quy dinh ro rang")
('LOAN#NEGATIVE',)

Up Coming Features

  • Text to Speech
  • Automatic Speech Recognition
  • Machine Translation
  • Dependency Parsing

Contributing

Do you want to contribute with underthesea development? Great! Please read more details at CONTRIBUTING.rst.

underthesea's People

Contributors

vietdung113 avatar jacknhat avatar khanh96le avatar

Watchers

James Cloos avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.