jcrowgey / neolo Goto Github PK

This project should be installable as a package.

There is an off-by-one error in sent_split which cause lines to be dropped from the output. Whether or not a line is dropped depends on how many sentence splitting punctuation marks it contains.

This is a severe bug which causes inaccurate results. It is in the same function which has the bug reported in #4, but is not related to that bug.

Reported via email

There is one change that would be very desirable: I wonder how difficult
would it be if the last line of results Neolo gives us could be
moved up to group with the first results that come in the very first line.
This is the last line of a result:

Punctuation ratio: 0.0799721500682859                                                                               

Since it comes after all the
lists and dictionaries, very few people pay attention to it.

The way I think would be good, would be like this:

Punctuation ratio: 0.0799721500682859                                                                               
17021 sentences in your text.                                                                                       
Average sentence length: 10.969684507373245 tokens.                                                                 
                                                                                                                    
----------------- Neologism List ----------------                                                                   
abeakutic                                                                                                           
abovementioned                                                                                                      
abrines                                                                                                             
abulafia                                                                                                            
accep                                                                                                               
accidens                                                                                                            
achates,,,                                                                                                          

Maybe MLTD is undefined on certain texts, in that case neolo still shouldn't try to divide by zero.

$ neolo mary.txt --dicts /usr/share/dict/words --mltd
Neologism list:
,
.

Statistics:
-----------
Text size: 25 tokens in 20 types.
Number of hapax legomena: 15
TTR (type-token ratio): 0.8
HTR (hapax-token ratio): 0.6
HTyR (hapax-type ratio): 0.75
Neologisms:  2 types not found in 1 dictionaries
Dictionaries contained 160707 tokens in 72376 types.
MLTD...Traceback (most recent call last):
  File "/home/j/src/neolo/venv/bin/neolo", line 10, in <module>
    sys.exit(main())
  File "/home/j/src/neolo/venv/lib/python3.7/site-packages/neolo/neolo.py", line 436, in main
    print(mltd(clean_text))
  File "/home/j/src/neolo/venv/lib/python3.7/site-packages/neolo/neolo.py", line 203, in mltd
    return float(toks)/float(seg_n)
ZeroDivisionError: float division by zero

I found a bug yesterday that seems a serious problem. In the 'Wordtypes and frequencies (unigram model):' I cannot find '?' and '!' and thus 'the sentences in your text.', 'Average sentence length:' and
'Punctuation ratio:' results come wrong. To check the error follow
https://drive.google.com/file/d/1LQRNF4kTVk8AyebYf4Hj1KJZDsaaPbJJ/view

In this address one may find other results to check:

https://agrippaproject.wordpress.com/
Thanks

It's hard to make changes without having some basic testing to guard against regression

The readme is incomplete, and probably needs updating on the part that is complete.