This module should have the ability to understand and work with [[...;...]]^...$ constructions implemented.

convert existing tests to unittest and run in travis

@unhammer and @sushain97: decide upon the license as discussed in irc and add the license headers.

from a21b53a

keep escapes, want to be able to reconstruct the input
TODO: if one wants to remove them, there should be a function from lu's to lu's that does that

Hello!

Streamparser always fails while trying to parse this output of apertium's tagger:
'^*+*/*+*$'

Original string is '*+*'.

Traceback (most recent call last):
File "apertium2fastmorph.py", line 137, in parse_apertium
pos = mainpos(r)
File "streamparser/streamparser.py", line 90, in mainpos
return reading[-1].tags[0] # type: ignore
IndexError: list index out of range

I parsed the entire corpus consisting of hundreds of millions tokens, and this is the only part where it fails. If I change *+* to +* or *+ or ** or *** it works.

It seems that it is a bug.

Can we add setuptools for easier installation/building?

We should have some proper python unit tests instead of the bash script.

pypy3 makes it 3x faster (which is very handy when using it for testvoc over millions of lines of input), but doesn't have Enum at all, and doesn't allow changing the namedtuple.doc (for SReading).

Some nice people suggested at #debian-python that streamparser package could benefit from a namespace (because it being such a generic name, as @flammie already showed us). Would apertium be okay name? It would be then "apertium.streamparser" instead of just "streamparser".

p.s. i'm not really familiar with namespace stuff so correct me if I got everything wrong, thanks!

biltrans-output has tags on the "wordform" (stuff between ^ and first /)

https://coveralls.io/builds/16163088/source?filename=streamparser.py

there are a couple uncovered lines ;)

http://peterdowns.com/posts/first-time-with-pypi.html

As it is now, it's quite hard to use software with streamparser as a dependency (like apertium-apy)

Travis appears to not really be reporting status anymore

streamparser.py has this line: __copyright__ = "Copyright 2016, Apertium". Shouldn't it be you guys, @unhammer and @sushain97 ?

in streamparser.py there is the warning printing print("WARNING: Empty readings for {}".format(self.lexicalUnit), file=sys.stderr), could it be printed to STDERR instead?

and validate in travis

if possible with latest pypy

will require major (?) version bump

The given example in the readme doesn't work with current version. See this paste from @nuboro:

from streamparser import parse
lexicalUnits = parse('^hypercholesterolemia/*hypercholesterolemia$\[\]\^\$[^ignoreme/yesreally$]^a\/s/a\/s<n><nt>$^vino/vino<n><m><sg>/venir<vblex><ifi><p3><sg>$.eefe^dímelo/decir<vblex><imp><p2><sg>+me<prn><enc><p1><mf><sg>+lo<prn><enc><p3><nt>/decir<vblex><imp><p2><sg>+me<prn><enc><p1><mf><sg>+lo<prn><enc><p3><m><sg>$')
for lexicalUnit in lexicalUnits:
    print('%s (%s) → %s' % (lexicalUnit.wordform, lexicalUnit.knownness, lexicalUnit.readings))


====================================================================================

    textBuffer += next(stream)
TypeError: 'str' object is not an iterator