libindic / normalizer Goto Github PK

• Use with style opening for automatically closing the file after reading
• python3 compatibiity
• Use for to loop through all lines
• Simplify ignoring comments
• Simplify rule syntax (is "" necessary?)

There should be a test folder with various tests that check that the normalization is correct.

We could consider integrating travis too, after that.

The error is possibly due to the fact that the dist/normalizer*.tar.gz has been prefixed by dist/libindic-normalizer*.tar.gz

I addded another line(line14) in normalizer_ml.rules as ൿ=ക്‍=ൿ=ക്‍ and raised an exception in LoadRules function of core.py

if(len(line.split("=")) != 2):
                raise Exception("[Error] Syntax Error in the Rules. Line number: %d"%line_number)
                # print(
                #     "[Error] Syntax Error in the Rules. Line number: ",
                #     line_number)
                # print("Line: " + text)

Output generated:

Exception: [Error] Syntax Error in the Rules. Line number: 27

Expected output:

Exception: [Error] Syntax Error in the Rules. Line number: 14

line_number is increamented twice within the loop, and hence the diiference in output

Like how https://github.com/libindic/unicode-conversion-maps has the rules file in root of the project, make the same here

/rules/ml.rules

There are cases where the normalized text is expected to retain the punctuations. As of now, the normalizer removes all punctuations. It will be good to make this a configurable option by passing an argument to the normalize method.

Atomic chillu should be converted to consonant + ് + zwj