Basic Ruby package.

write a simple document to show how to contribute to open-tamil.

Python 3 support is missing. This goes to origin of sorted() working differently with new API in Python 3. Ref: [[https://docs.python.org/3/howto/sorting.html]], [[http://python3porting.com/preparing.html]]
for l in sorted(frequency.keys(), key=cmp_to_key(tamil.utf8.compare_words_lexicographic)):
File "/usr/local/lib/python3.4/dist-packages/tamil/utf8.py", line 390, in compare_words_lexicographic
return cmp(pos1, pos2)
NameError: name 'cmp' is not defined

Reverse transliterators

1. Reverse f/w tables of Azhagi, Tamil99e etc
2. See if Olingoa standard from KaReFo
   http://karky.in/labs/olingoa/help.jsp

Define solthiruthi command line interface, any combination of one or more of following,
$ python solthiruthi.py -dialects kovai,nellai,chennai -dict std,wikipedia,madurai,
$ python solthiruthi.py -help
$ python solthiruthi.py #to use the standard input and report errors at EOF
$ python solthiruthi.py -in -nalt 5 -replace #only 1 file at a time
$ python solthiruthi.py -auto # output in useful format for testing speller itself

options

1. Dialect related spell checking for Kovai, Nellai, Ceylong or Chennai and other future vattara mozhigal
2. Dict shows the allowed word dictionaries while spell checking
3. Nalt shows number of alternatives in the suggestions
4. Replace option replaces the text inplace
5. Auto mode is sutiable for testing -a non-Interactive mode

Solthiruthi - framework - iterable method for DTrie data structure (from #36 )

1. Iterable methods,
   obj,getWordsIterable()
   is very idiomatic python and has less memory footprint compared to passing large lists, and naturally maps to working with streamed data.

1. Saurashtra text is written in Tamil/Brahmi script
2. Provide tools for Saurashtra text validation based on Tamil/Brahmi script

Spell checking without context is inherently 'embarassingly parallel' problem.

In pypi release 0.2.8, under tamil module directory two modules / directories are missing.
txt2ipa and txt2unicode modules / directories are not present in https://pypi.python.org/pypi/Open-Tamil/0.2.8

Download and extract tarball
https://pypi.python.org/packages/source/O/Open-Tamil/Open-Tamil-0.2.8.tar.gz#md5=270be5290aa1796016e19653caa21440

You can see that those 2 directories are missing.

solthiruthi - add wikipedia, project madurai wordlist to data folder

1. solthiruthi/tests
   1.1 generate example documents with typos in words, and misspelled words.
   1.2 specify the words that are in error in text
   1.3 specify the corrected word/expected correct word

2. Items could be listed as XML file,
   <?xml ver="1.2"?>
   <urai>
   <pizhaigal>
   <sol>
   <thiruttham></thiruttham>
   </sol>

   .... more sorkal ....
   </pizhaigal>
   </urai>

3. Domain-specific language : alternative to 2 the urai items can be listed in a clear text format,

   urai

   .... tamil text goes here ...

   end urai

   pizhaigail

   $SOL | $THIRUTTHANGAL

   end pizhaigal

This issue is to prove for yourself the DTrie with counts measures the right frequency information form given corpus. It is a complementary to #53 an #51.

Acceptable solution includes a simple unittest.

Support Tamil Monolingual Encoding (TAM) Tamil read/write support

1. Define protocol/class outline interface for applying rules
2. Define rules for adjacent vowels, consonants
3. Define rules based on other Tamil language heuristics

The following url is not working.
https://github.com/arcturusannamalai/open-tamil/archive/latest.zip

Please change it into master.zip in setup.py or upload latest.zip in any of your release.

Thanks.

utf8.tamil_letters has agaram repeated twice.

1. part of agaram
2. part of uyir-mei sequence

this error can be fixed

data on proper nouns - part 1

1. cities
2. animals
3. capitals-n-countries

Users should upload files and get various statistics using the web application.

Build a Trie data structure with API

1. add words into Trie
2. check if word exists in Trie
3. list out all words in a Trie in alphabetical order
4. provide DTrie - dictionary based and In-Memory Trie.

Tries are useful to build indexes from dictionaries to aid in fast searches.
Ref: http://en.wikipedia.org/wiki/Trie

வணக்கம்,

எனது சிறிய வேண்டுகோள்/கருத்து.

உங்கள் முயற்சிகளுக்கு எனது மனமார்ந்த வாழ்த்துக்கள் நன்றிகள் !
எந்த தமிழ் மென்பொருளாகினும் ஆவணம் தமிழில் கிடைக்கிடைப்பது அறிது.
உங்களின் படைப்புகளுக்கான ஆவணங்களை(Readme etc..) முடிந்த வரை தமிழில் கொடுத்தால் பயனாக இருக்கும்.

வருகின்ற நிரலாலற்களும் இதைத்தொடர்ந்து செய்துடுவார்கள்.

நன்றி
அருண் குமார்

Take words from dictionary and form a text. Introduce known errors and form a test case following the format of #26

https://ezhillang.wordpress.com/2015/04/22/solthiruthi-multi-pass-spell-checker-for-tamil-language-draft-1/

Is it possible to explore these stemmers, adopt them to tamil and add to open-tamil?

http://tartarus.org/~martin/PorterStemmer/

http://tartarus.org/~martin/PorterStemmer/python.txt

http://text-processing.com/docs/stem.html

snowball stemmer seems working for tamil.

https://github.com/rdamodharan/tamil-stemmer
snowballstem/snowball-data#1

With data like n-gram stats for letters of Tamil language, we may provide services for,

1. detecting words in error
2. ranking suggestions based on their letter likelihood

Hello Arulalan,

Please move unit tests to right folders, for 'utils' module.

I think we have to merge all of that code into tamil module at some point. Maybe 'tamil.utils' class within 'tamil' module would be a nice setup.

Thanks

DTrie -add a 'count' attribute Trie

1. This is useful for n-gram word/letter data computation.
2. Has minimal overhead.

Ref: http://en.wikipedia.org/wiki/Tamil_script

ksha க்ஷ் series of Tamil grantha/sanskrit letters missing from tamil.utf8 module of open-tamil.
Ref: http://en.wikipedia.org/wiki/Tamil_script @ sec consonants of modern Tamil

Collect noun list from various sources.
Ask community to help for adding more verbs.

Following 'Solthiruthi - framework - Trie data structure' issue #36 we need to,

1. add pickle capability to save index to disk
2. load pickle file from disk into in-memory index
3. add unit tests

பாடத்தில் -> பாடத்த் (இல் நீக்கம்)
அவர்கள் -> அவர் (கள் நீக்கம்)

• Task: Build n-gram letter analysis tools to work on a given corpus, e.g. Wikipedia, Website, Blog etc.
• Requirements
  1. Letter frequency info
  2. Esp bi-, tri- gram data sets, with smoothing
  3. Word level bi-gram data.
  4. Write out the data in some common text/XML format as efficient.
• Note :
  This will be useful precursor to Bayesian analysis in Tamil

Solthiruthi - queues - document flow

1. Split words from document into entity/non-entity with line/col info attributes
2. Each entity has 'is_error' Boolean attribute, and a 'reason' list of strings attribute.
3. File read process puts this data into a queue.
4. Each pass/nilai will read words, process them and put them back in output queue

Goal is to reuse the Tamil grammar contributed by Elanjelian Venugopal.

1. Write XML parser to load data from file, Ref: https://github.com/languagetool-org/languagetool/blob/master/languagetool-language-modules/ta/src/main/resources/org/languagetool/rules/ta/grammar.xml
2. Write interpreter for the rules from 1

LanguageTool.org Tamil grammar file is currently distributed under LGPL.

Support TACE encoding on open-tamil. This will be a good enhancement for future Tamil encoding formats.

In the Solthiruthi framework, update the datastructure Trie with a method 'getAllWordsPrefix' to return all words starting with known prefix.

e.g. 

    >>>  trieObj.getAllWordsPrefix(u"நவீ")
           # should return, 
           [u"நவீனம்", u"நவீன",u"நவீ"]
   >>>  # etc

Parser for data sets on proper nouns.

Basic Encoding support for ISCII-Tamil
Ability to read/write letters
** Ref **

1. https://en.wikipedia.org/wiki/Indian_Script_Code_for_Information_Interchange
2. https://ta.wikipedia.org/wiki/%E0%AE%87%E0%AE%B8%E0%AF%8D%E0%AE%95%E0%AF%80

Reported by Shrini

open-tamil பயன்படுத்தி tscii - ஒருங்குறி மாற்றும் போது, 'ரி' எழுத்து
வருவதில்லை.

¢ மட்டுமே வருகிறது. ரி க்கு முன் உள்ள எழுத்துமே மாறி விருகிறது.

உதாரணக் கோப்புகளைக் காண்க.

இதே பிழை இங்கும் உள்ளது.

http://www.suratha.com/reader.htm

இது சரியாக வேலை செய்கிறது.
http://www.higopi.com/adhiyaman/

அருள்,
இந்த பிழையை சரி செய்ய வேண்டுகிறேன்.

Encoding TAB (Tamil Bilingual) Tamil read/write support

4000 கும் மேற்ப்பட்ட தமிழ் பெயர்கள் - Tamil Names
Ref: http://naayak.blogspot.in/p/blog-page.html

Create a standard library that can
1.Take any unicode input and try to return the codepoints for TACE-16
2. Interpret complex characters and prrovide TACE-16 codepoints
3. String Manipulation on TACE-16

tamil.utf8.istamil_alnum('---')
Traceback (most recent call last):
File "", line 1, in
File "build/bdist.linux-x86_64/egg/tamil/utf8.py", line 251, in istamil_alnum
AttributeError: 'str' object has no attribute 'istamil'

Create a standard library that can
1.Take any unicode input and try to return the codepoints
2. Provide a standard method for Level II rendering for each complex character( avoid canonical equivalence problem)
3. Provide a new index to all 247+ Character set.
4. Return a standard tamil character array
5. Provide common string manipulation functions that are agnostic to encoding
a. Char count
b. Concatenate strings
c. reverse the strings
d. provide an array and return a string
e. compare strings
f. split string

Add a word splitter algorithm for Tamil

பாரதமாதா -> பாரத + மாதா
தமிழ்நாடு
ஒருதலைகாதல் -> ஒரு + தலை + காதல்

பல்கலைகழகம் -> பல் + கலை + கழகம்

கணிதம் -> (same)

Ref: These will serve similar purpose like algorithm from German language, https://github.com/danielnaber/jwordsplitter

https://ezhillang.wordpress.com/2015/04/22/solthiruthi-multi-pass-spell-checker-for-tamil-language-draft-1/

https://ezhillang.wordpress.com/2015/04/22/solthiruthi-multi-pass-spell-checker-for-tamil-language-draft-1/

Quickly detect if given unicode character is in the Tamil unicode page.

1. Shrinivasan's code
   Ref: https://github.com/tshrinivasan/tamil-wikipedia-word-list/blob/master/find_tamil.py
2. Language Tool's code
   Ref: https://github.com/languagetool-org/languagetool/blob/4c49afc22b4e6dc7c41e0a727a822d7070dbece7/languagetool-office-extension/src/main/java/org/languagetool/openoffice/TamilDetector.java

Provide some key functions in Java and JavaScript like the ones found in tamil.utf8 python module.

Preferably make JavaScript library as toolkit independent.