Code Monkey home page Code Monkey logo

nlp_thai_resources's Introduction

Thai Natural Language Processing (Thai NLP) Resource

Collection of Thai Natural Language Processing (NLP) software libraries, dictionaries, and corpus. Always welcome for pull requests.

Thai NLP Libraries

Thai Character Cluster

Library Description Programming Languages Features License Author & Link
JTCC Thai Character Cluster Java GPL-3.0 Wittawat
TCC Thai Character Cluster Python Apache 2.0 Wannaphong

Thai Soundex

Library Description Programming Languages Features License Author & Link
LK82 + Udom83 Thai Soundex Python Korakot

Word Segmentation

Library Description Programming Languages Features License Author & Link
Swath SWATH (Smart Word Analysis for THai) is a word segmentation for Thai C Longest Matching, Maximal Matching and Part-of-Speech Bigram. GPL CMU
Lexto Lexto: Thai Lexeme Tokenizer Java LGPL NECTEC
Python 2 LGPL Python2 Wrapper
Python 3 LGPL Python3 Wrapper
Wordcut Thai word breaker for Node.js JavaScript, Node.JS LGPL-3.0 veer66, github
wordcutpy A simple Thai word tokenizer written in 1 Python file Python 3 LGPL-3.0 veer66, github
CutKum Thai Word-Segmentation with Deep Learning in Tensorflow. RNN. Python 0.93 F-measure. MIT Pucktada, github
Thai Language Toolkit (tltk) Based on a paper by Wirote Aroonmanakun in 2002. Word segmentation is based on a maximum collocation approach. Syllable segmentation is based on 3grams statistics. (Dataset is included) Python 0.9786 F-measure. (It was tested on a different testset; it is not fair to compare it with other models.) GPLv3 awirote, the Python Package Index
DeepCut A Thai word tokenization library using Deep Neural Network. CNN. Python 0.988 F-measure. MIT rkcosmos, github
SynThai Thai Word Segmentation and Part-of-Speech Tagging with Deep Learning. RNN. LSTM. Python 0.992 F-measure. MIT KenjiroAI, github
CutThai Thai word segmentation written in coffee-script Edit Coffee-script MIT Pureexe/cutthai Github

Part of Speech Tagging (POS Tagging)

Library Description Programming Languages Features License Author & Link
Jitar+NAiST A simple Trigram HMM part-of-speech tagger Java Ver66, Jitar + NAiST, 1 + NAiST, 2
SynThai Thai Word Segmentation and Part-of-Speech Tagging with Deep Learning. RNN. LSTM. Python 0.9163 F-measure. RNN. LSTM MIT KenjiroAI, github

Name Entity Recognition

Library Description Programming Languages Features License Author & Link
Named Entity Tagging (Thai NEST) Thai Named Entity tagging Specification and Tools GPL KINDML, SIIT, AIAT

News Structure Tagging

Library Description Programming Languages Features License Author & Link
News Structure Tagging Program Thai News Structure Tagging Program Metadata tagging, Structure tagging, Automatic News Title Generation GPL AIAT

Syntactic Parsing & Tools

Library Description Programming Languages Features License Author & Link
Chart-parser Extract Syntactic Structure from POS Tagged Sentence. C All rights reserved Thanaruk T. ([email protected])
Grammar Processing Labelled Brackets -> Context Free Grammars (CFGs) Python Transform and compute probability Thodsaporn C.

Thai Word Embedding

Library Description Programming Languages Features License Author & Link
kobkrit-word-embedding Tensorflow implementation of Thai word embedding Python Source code, Example, Word distance graph LGPL Kobkrit V.

Dictionaries / Translation Pairs

Library Description Size Features License Link
Transliteration Corpus 31K pairs Thai-Eng Translation Pair CC BY-NC-SA 3.0 TH NECTEC
LEXiTRON Thai<->English Dictionary TH->EN, EN->TH LEXiTRON License NECTEC
Yaitron LEXiTRON in machine readable format (XML) TH->EN, EN->TH LEXiTRON License Veer66 Schema, Data & Conversion Code

Downloadable Text Corpus

Library Description Size Features License Link
ORCHID 30K sent. Word Seg., POS Tagged. CC BY-NC-SA 3.0 TH NECTEC
InterBEST 2009/2010 5M words Word Seg. CC BY-NC-SA 3.0 TH NECTEC
Thai Wikipedia Formal Articles 1.49GB (~213.1 MB compressed) XML GFDL WIKIPEDIA
TNC Top-5000 Words Word frequency 5,000 words Frequency of Thai words in various genres, EXCEL All rights reserved CHULA
Click Bait Sentences Thai Click Bait Sentence 330 sent. (90.7KB) MIT Wannaphongcom
Thai Sentimental Word List Thai Sentimental Words List 52KB Seperated Words as Adj, V MIT Wannaphongcom
Prime Minister 29 Prime Minister 29's Speech Sentences 338KB Word segged, Name Entity Tagged MIT Wannaphongcom
Thai named entity corpora named entity corpora by Wirote Aroonmanakun's students 266KB-1.5MB syllable seg., word seg., Named Entity tagged GPLv3(not sure, but tltk is using this license) นัชชา ถิระสาโรช Data
ศศิวิมล กาลันสีมา Data
ณัฐดาพร เลิศชีวะ Data
Thai WordNet THE CONSTRUCTION OF THAI WORDNET OF 1ST ORDER ENTITY COMMON BASE CONCEPTS USING A BI-DIRECTIONAL TRANSLATION METHOD AND WITH DICTIONARIES OF DIFFERENT COMPILATIONAL APPROACHES(ธนนท์ หลีน้อย)

THE CONSTRUCTION OF THAI WORDNET OF 2ND ORDER ENTITY COMMON BASE CONCEPTS USING A BI-DIRECTIONAL TRANSLATION METHOD : A STUDY OF THE DIVERSITY OF MEANINGS AFFECTING TRANSLATIONAL ACCURACY (ปริศนา อัครพุทธิพร)		 WordNet N/A ธนนท์ หลีน้อย 2008
ปริศนา อัครพุทธิพร Data 2008

Web Query Text Corpus

Library Description Size Features License Link
Thai National Corpus 2 32M words Query text by genre, domain All rights reserved CHULA
Thai Medical Document 3,594 docs Document and dynamic keyword map All rights reserved KINDML, SIIT
Southeast Asian Languages Library Thai News, Web Text, Pop Music, Literature, Toponyms 20M chars Phase around a search text SEALang
HSE Thai Corpus Modern texts written in Thai language (mostly news websites) 50M tokens Query by word form, lexeme, translation, grammatical attributes, lexical attributees HSE School of Linguistics

Pre-trained Word Vectors

Pre-trained Model Description Size Dimensions License Link
fastText Skip-Gram model trained on Wikipedia using fastText 300 CC BY-SA 3.0 Facebook + Bin & Text + Text Only
thai2vec AWS LSTM Language Model trained on Wikipedia. Perplexity of 46.61 with 51556 embeddings. 147.6MB 300 MIT thai2vec / pyThaiNLP

Text Classification Benchmarks

Model Description Dataset Accuracy License Link
thai2vec Finetuned AWS LSTM Language Model BEST 94.4% MIT thai2vec / pyThaiNLP

Not found? Try to look at another Thai NLP Awesome List/Resource (Like this one)

http://aiat.in.th/resources/

Acknowledgements

nlp_thai_resources's People

Contributors

kobkrit avatar c4n avatar tchayintr avatar bact avatar cstorm125 avatar bi89 avatar veer66 avatar

Watchers

James Cloos avatar Tanya Sattaya-aphitan avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.