masakhane-io / masakhanepreprocessor Goto Github PK

credits: https://github.com/keleog/PidginUNMT

pidgin_corpus.txt

Language: Igbo
Lang code: ig

Don't worry about language codes so much if you do not know. We will gladly take care of that.

allowed_symbols: ['.-]
diacritics: True
Example sentene to test: Dịka● ndọrọndọrọọchịchị maka ntuliaka ọkwa Gọvanọ Anambra steeti si na-aga nke afọ 2021, ndị nọ.

I have been compiled the package and now you can install with pip install masakhane-preprocessor .

But when I try to access the Preprocessor with the following code it still does not work.

from masakhane-preprocessor import Preprocessor

my_prep = Preprocessor(lang='ig')

We need to compile this into a package so that people can easily use it with a pip install or python setup.py

See here for more background on unicode and normalization

Note to Self

Find a way to do regex for these characters that still compose into characters + accent....like below

One idea:
Could do regex preprocessing in 2 steps using example ɛ̃:

1. map the double characters to a random integer or char like RANDOMCHAR1.
2. Do normal cleaning to remove punctuation and symbols.
3. Then map RANDOMCHAR1 back to its original value, thus preserving it

Language: Fon
Lang code: fon

allowed_symbols: ['à', 'á', 'è', 'é', 'ì', 'í', 'î', 'ï', 'ó', 'ù', 'ú', 'ā', 'ă', 'ē', 'ĕ', 'ŏ', 'ū', 'ŭ', 'ɔ', 'ɖ', 'ò', 'ε', 'έ', 'ɔ̀', 'ɔ̆', 'ὲ', 'ɔ́', 'ĭ', 'ɛ̆', 'ɛ̃']
diacritics: True