Presumably, this will require fixing a bunch of existing problems, heh.

As discussed in apertium's IRC channel it would be nice to have chained translations, for example Sardinian -> Italian -> Spanish. So some code needs to be done that creates a path which the source text needs to go trough in order to achieve the wanted translation. Also some flag etc. that enables these chained translations.

With a single request, we're getting up to three different responses.

This is the request we're using: http://beta.apertium.org/apy/perWord?lang=eng-fra&modes=biltrans&q=hi

Depending on APy's mood, these are the outputs we get:

Correct output:

[{"biltrans": ["salut<ij>"], "input": "hi"}]

Empty output:

[{"biltrans": [], "input": "hi"}]

Error output:

{"message": "Internal Server Error", "code": 500, "status": "error", "explanation": "Internal Server Error"}

Noticed by @avyayv.

Useful for testing: https://phabricator.wikimedia.org/T108798#2011440

More examples: http://petstore.swagger.io/ / http://petstore.swagger.io/v2/swagger.json

Document translation doesn't currently support the old Windows formats. Perhaps there's a quick Python plugin that we can use to perform a conversion on the fly back and forth. This would be best as an enable-able feature so that the library doesn't have to be installed

migrated from https://sourceforge.net/p/apertium/tickets/7/

e.g.

meow (en->es) *meow
*meow (es->fr) **meow

so

meow (en->fr) **meow

instead of

meow (en->fr) *meow

http://apy.projectjj.com/beta/translate?langpair=ita gives {"code": 400, "message": "Bad Request", "status": "error", "explanation": ""} which isn't much of an explanation

if getLocalizedLanguages reads from a damaged disk/db, it can block other requests. It should use the same non-blocking yield pattern as translation does.

/list?q=pairs return data differs from rest of the modes. It produces:

$ curl 'http://localhost:2737/list?q=pairs'

{"responseStatus": 200, "responseData": [
 {"sourceLanguage": "kaz", "targetLanguage": "tat"}, 
 {"sourceLanguage": "tat", "targetLanguage": "kaz"}, 
 {"sourceLanguage": "mk", "targetLanguage": "en"}
], "responseDetails": null}

and the rest are like:

$ curl 'http://localhost:2737/list?q=analyzers'
{"mk-en": "mk-en-morph", "en-es": "en-es-anmor", "kaz-tat": "kaz-tat-morph", 
 "tat-kaz": "tat-kaz-morph", "fin": "fin-morph", "es-en": "es-en-anmor", "kaz": "kaz-morph"}

Should we just remove the pairs mode from /list and have only /listPairs?

Currently APy outputs unicode characters using \u codes, e.g. '{"vblex": ["c\u00f3rrer"], "n": ["cursa"]}' chnages to '{"vblex": ["córrer"], "n": ["cursa"]}'.

It would be more convenient if it output unicode directly, e.g. '{"vblex": ["córrer"], "n": ["cursa"]}' chnages to '{"vblex": ["córrer"], "n": ["cursa"]}'.

• getCoverage may have to be replaced with calcCoverage

• the example on wiki page fails with 400- bad request error.

INFO:root:A pipe for pair sme-sme_spell has handled 200 requests, scheduling restart
INFO:root:sme-sme_spell not in pipelines of this process
INFO:root:Starting up a new pipeline for sme-sme_spell <E2><80><A6>
ERROR:tornado.application:Future <tornado.concurrent.Future object at 0x102670ef0> exception was never retrieved: Traceback (most recent call last):
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/wwserver1/divvun/apertium-apy/servlet.py", line 995, in get
    reformat=False)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/wwserver1/divvun/apertium-apy/servlet.py", line 503, in translateAndRespond
    translated = yield pipeline.translate(toTranslate, nosplit, deformat, reformat)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1063, in run
    yielded = self.gen.throw(*exc_info)
  File "/Users/wwserver1/divvun/apertium-apy/translation.py", line 80, in translate
    for part in all_split]
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1055, in run
    value = future.result()
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 828, in callback
    result_list.append(f.result())
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/concurrent.py", line 238, in result
    raise_exc_info(self._exc_info)
  File "<string>", line 4, in raise_exc_info
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/tornado/gen.py", line 1069, in run
    yielded = self.gen.send(value)
  File "/Users/wwserver1/divvun/apertium-apy/translation.py", line 285, in translateNULFlush
    proc_deformat = Popen(deformat, stdin=PIPE, stdout=PIPE)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/subprocess.py", line 676, in __init__
    restore_signals, start_new_session)
  File "/opt/local/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/subprocess.py", line 1185, in _execute_child
    errpipe_read, errpipe_write = os.pipe()
OSError: [Errno 24] Too many open files

This is on a server where language data is updated nightly, and pipes are restarted every 200 requests / on 3600 idle secs.

Maybe APy isn't correctly closing files when restarting pipelines?

APy should be able to install itself. It already has a Makefile, but there is no install target. Alternatively, add a setup.py.

Some way to run a single command to have APy put itself into $DESTDIR/$prefix in a fully finalized state. Basically, do what make + https://github.com/TinoDidriksen/apertium-packaging/blob/master/trunk/apertium-apy/debian/apertium-apy.install does.

The result should be that I can delete my install file entirely and just use the files APy puts into $DESTDIR/$prefix.

@unhammer:

ScaleMT support's multiple &q parameters. This would make it possible for clients to buffer up some requests, while ensuring each request does not interfere with the other (to the extent that apy itself ensures this through ensuring NUL flushing between value of q).

migrated from https://sourceforge.net/p/apertium/tickets/72/.

I tried with 2 requests through APy sandbox. Both of them work in localhost while both of them fail when the server is https://www.apertium.org/apy with 400 - Bad request error. Is it because APy is running some other version of code on the server?

Fails on APy URL

Works on localhost

I didn't see any mention that apertium-apy is gplv3+, does this mean were stuck with gplv3 forever? Maybe those ~12 people who have contributed could be ask individually but that is usually a difficult job.

Hi all,

I was trying to install apertium-apy from nightly debian repo at apertium.projectjj.com, it installs, but when you try to start service it shows:

Feb 06 11:31:26 softaragones python3[8195]: Traceback (most recent call last):
Feb 06 11:31:26 softaragones python3[8195]: File "servlet.py", line 58, in
Feb 06 11:31:26 softaragones python3[8195]: from streamparser.streamparser import parse, known
Feb 06 11:31:26 softaragones python3[8195]: ImportError: No module named 'streamparser'
Feb 06 11:31:26 softaragones systemd[1]: apertium-apy.service: main process exited, code=exited, status=1/FAILURE
Feb 06 11:31:26 softaragones systemd[1]: Unit apertium-apy.service entered failed state.
Feb 06 11:31:26 softaragones systemd[1]: apertium-apy.service holdoff time over, scheduling restart.
Feb 06 11:31:26 softaragones systemd[1]: Stopping Translation server and API for Apertium...
Feb 06 11:31:26 softaragones systemd[1]: Starting Translation server and API for Apertium...
Feb 06 11:31:26 softaragones systemd[1]: apertium-apy.service start request repeated too quickly, refusing to start.
Feb 06 11:31:26 softaragones systemd[1]: Failed to start Translation server and API for Apertium.
Feb 06 11:31:26 softaragones systemd[1]: Unit apertium-apy.service entered failed state.

In order to workaround that, I downloaded streamparser from https://github.com/goavki/streamparser/tree/cf67427283a5316f3a23dbc193e75603c5cbe34c and put at /usr/share/apertium-apy/ directory, after that, it seems to be started and running OK, so I supose this module is missing inside debian package an it should to be included.

Bye!

sqlite3 -header /home/apy/apertium-apy/missing.db <<EOF
.mode insert
select * from missingFreqs;
EOF

gives me e.g.

INSERT INTO table(pair,token,frequency) VALUES('sme-nob','redakt&oslash',24);
INSERT INTO table(pair,token,frequency) VALUES('sme-nob','Nollii&raquo',3);

– whenever we see an html-entity, it's cut off at the ; so we don't know what the rest of the word is.

https://docs.python.org/3/whatsnew/3.5.html#pep-479-change-stopiteration-handling-inside-generators

324a185 changed the return from generator into raise StopIteration to make it compatible with older python3's, but this makes it incompatible with 3.5 and newer, see https://travis-ci.org/goavki/apertium-apy/jobs/102092337 for the error message

We used to (?) support multiple q's per query (or maybe scalemt did). People are expecting us to, anyway:

Hello,

today I discovered that apertium is sending wrong response for batch interface (several q=).
For this:
https://www.apertium.org/apy/translate?format=html&markUnknown=no&langpair=en%7Ces&q=seconds+with&q=queries.+Memory+Usage%3A&q=8.12+MB

I get:
{"responseStatus": 200, "responseDetails": null, "responseData": {"translatedText": "8.12 MB"}}

ONLY LAST PART IS TRANSLATED!!! And according to this information:
http://wiki.apertium.org/wiki/Apertium_scalable_service
Each part should be translated.

Please fix it or inform ke does batch interface changed. Anyway please send me some info about sitiation :)

Cheers
Michał Podbielski

Could we just delete toro.py from apertium apy source code and add to the wiki instructions to install it with pip or via operating system's package manager? Or is it possible to move completely to tornado's locks so that we wouldn't even need toro?

From @TinoDidriksen:

Need the APy people to look at this.

APy yields 500 Internal Server Error:
https://www.apertium.org/apy/translate?q=250%20ans%20d%E2%80%99histoire,%20un%20mod%C3%A8le%20p%C3%A9dagogique%20unique,%20une%20ouverture%20sur%20le%20monde%20avec%20de%20nombreux%20partenariats%20avec%20les%20meilleures%20universit%C3%A9s,%20des%20institutions%20culturelles%20de%20premier%20plan%20et%20les%20entreprises%20les%20plus%20innovantes.&langpair=fra|cat

But this works:
echo "250 ans d'histoire, un modèle pédagogique unique, une ouverture sur le monde avec de nombreux partenariats avec les meilleures universités, des institutions culturelles de premier plan et les entreprises les plus innovantes." | sh ~apertium/tarballs-build/share/apertium/modes/fra-cat.mode -g
250 anys d'història, un model pedagògic únic, una obertura sobre el món amb moltes cooperacions amb les millors universitats, de les institucions culturals de primer plànol i les empreses les més *innovantes.

APy log: http://codepad.org/cjtbFI8g

It would be great if we could have a deb package for installing apertium apy.

Re: https://www.mediawiki.org/w/index.php?title=Topic:T13rue5gsz3gjqrx&topic_showPostId=t13w8lgm4bxsxk3w&fromnotif=1#flow-post-t13w8lgm4bxsxk3w
it'd be cool to support TMX-files as with the -m option in Apertium, and even cooler if we could integrate https://github.com/goavki/patcher

Requests to https://www.apertium.orf/apy/identifyLang produce code 400 errors.

When trying to translate this file from Norsk Bokmål (nob) to Nynorsk (nno), I only get part of the file back.

The section I get back begins and ends with ".(null)"

Any way to pre-process the file to get around this?

Using https://www.apertium.org/apy/translateDoc?langpair=nob|nno&markUnknown=no

Wikimedia Foundation is using apertium-apy in production environment (Thanks for all help!). It will be great to have ChangeLog or Summary on what has been changed between two releases.

Python 3.2's flake and mypy malfunction. This makes our build process and imports somewhat nasty (introduced in #69).

Should we stop supporting Py3.2?

cc @unhammer

It'd be nice to have some type checking. I just made a silly mistake when renaming a variable (sending the module named html into re.sub instead of the html-contents variable …), which a good type checker would've caught right away. Python can't do very good type checking, but at least some type checking is possible with mypy, and mypy --check-untyped-defs would've caught this.

We'd have to do any type annotations in comments instead of the nice inline PEP484 syntax, since we want to support older Python versions.

We might have to only enable it for newer python versions in travis.

We might have to include some stubs for modules (servlet.py:55: error: Cannot find module named 'cld2full'), or tell mypy to ignore some things (translation.py:10: error: Name 'locks' already defined because we fallback to toro.locks in a try-except) or fill in for some bugs in mypy (translation.py:12: error: Module 'select' has no attribute 'PIPE_BUF').

Keeping pipelines open when translating requires splitting requests into pieces that are <pipebuf in size, but this messes up HTML. Currently (e.g. in branch giellatekno 3dd7a76 ) we just start new pipelines per page request.

@unhammer:

We probably shouldn't do threading since it makes python eat all our swap :-) It should be simple enough to start the process using tornado.process.Subprocess in a @gen.coroutine using "yield", similar to translateSimple()

migrated from https://sourceforge.net/p/apertium/tickets/7/

Safari users report documents getting back all garbled

tull_test.txt

I'm new to apertium-apy. I install this way in ubuntu server xenial (just for english and spanish):

wget http://apertium.projectjj.com/apt/install-nightly.sh
sudo bash install-nightly.sh
sudo apt-get install apertium-apy
sudo apt-get install apertium-en-es
sudo systemctl start apertium-apy.service
sudo systemctl enable apertium-apy.service

The service is running, and with this
curl http://localhost:2737/listPairs
I get
{"responseStatus": 200, "responseDetails": null, "responseData": [{"targetLanguage": "eng_US", "sourceLanguage": "spa"}, {"targetLanguage": "eng", "sourceLanguage": "spa"}, {"targetLanguage": "spa", "sourceLanguage": "eng"}]}

But I cannot translate anything, for example,
curl 'http://localhost:2737/translate?langpair=eng|spa&q=this+is+a+test'
{"message": "Internal Server Error", "status": "error", "explanation": "Internal Server Error", "code": 500}

What I'm doing wrong? I'm trying to use in combination with ContentTranslation mediawiki extension, and I need to be sure that it works from command line. Thanks in advance

e.g. "https://www.avvir.no" gives Feb 01 15:34:44 gtweb.uit.no python3[14767]: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645) in the log, but simply "translation not available" in html-tools.

• better JSON response from apy
• have html-tools give that response instead of "translation not available"

In in ListLanguageNamesHandler in servlet.py should we change the localeArg = self.get_argument('locale') to be like localeArg = self.get_argument('locale', default=None)? Without the defaulting to None, I don't see why the code elif 'Accept-Language' in self.request.headers: is useful (in order to use the header field there still needs to be the "locale" parameter but with no value).

I think only some of the endpoints are currently tested.

• all the endpoints should be tested
• tests should use standard libraries (in Python)
• coverage stats should be produced
• Travis should enforce coverage
• should test non-blocking nature of the endpoints

(moved from https://sourceforge.net/p/apertium/tickets/78/ )

Currently, we only keep track of the pids of the "ends" of flushing pipelines (inpipe, outpipe), not the whole group.

If part of a language pair pipeline crashes on startup, the rest of the pipeline may simply hang (and leave the request hanging). Of course this might mean there's a serious bug in the core tools (or language data was compiled for the wrong version), but it'd be nice if APY were more robust here.

One possible way of mitigating this might be to store pids of all started procs in the FlushingPipeline object, and then in the cleanup function go through pipelines_holding and stop all procs of any pipeline which hasn't been used in over a minute.

@unhammer:

We might want to have some locks or a similar method of ensuring we don't start more than N processes at a time (not likely to happen in normal usage, more to protect against malicious users), could have a dict of doc_locks just like pipeline_locks.

migrated from https://sourceforge.net/p/apertium/tickets/7/

I am unsure of what other kind of inputs would fail. I tried http://facebook.com as query for identifyLang and it gives Internal Server error.

servlet.py has this function:

def setupHandler(
    port, pairs_path, nonpairs_path, langNames, missingFreqsPath, timeout,
    max_pipes_per_pair, min_pipes_per_pair, max_users_per_pipe, max_idle_secs,
    restart_pipe_after, max_doc_pipes, verbosity=0, scaleMtLogs=False, memory=1000
):

and there is the port parameter. It is unused so it could removed.

We might want to log all failing requests, e.g. 500's and such, to a file. The GET requests just go to output by tornado (collected by journald), but not POST.

An argument like --config=config.conf would be great, the commands become really long with all the settings.

e.g. to specify rehtml vs rehtml-noent

Local variable "serverlist" is possibly mixed up with class' data attribute on line 184 in gateway.py.

failing sanity-tests run about one hour apart give

exit code 1: .cat-fra: expected 'pour', got 'chaud' (for input: per)
exit code 1: ...................cat-fra: expected 'pour', got 'Éternels' (for input: per)
exit code 1: ............................................................cat-fra: expected 'pour', got '*carcater' (for input: per)
exit code 1:
exit code 1: ........................................................cat-fra: expected 'pour', got 'boulangerie' (for input: per)
exit code 1:
exit code 1: ..........................................cat-fra: expected 'pour', got 'relâcher' (for input: per)
exit code 1: .......................cat-fra: expected 'pour', got '*Cotilló' (for input: per)
exit code 1: ............cat-fra: expected 'pour', got 'rainure' (for input: per)
exit code 1: .................................................cat-fra: expected 'pour', got 'approuvez' (for input: per)

as if the output of sending "per" into apy's cat-fra was from the input of an earlier request.

(That seems quite bad)

http://stackoverflow.com/a/19682945/69663 gives php code to do it for docx; it'd be nice not having to shell out to mimetype/xdg-mime/file (which have had security issues). We would only need to check for the 3 or 4 types we accept, and disregard the rest.

Some normal pages take up to 30s to translate with the giellatekno pairs.

Maybe we should try having a little
{ pair : { url : translationOutput } }
? Just for the slashdot effect where one page gets requested again and again.

Seeing as caching is Hard, we could keep it simple and just empty it when sys.getsizeof > max_url_cache. But sys.getsizeof doesn't recursively follow the size of the object; also, news sites etc. change, so I added a stupid simple "empty if older than two hours" rule for the whole cache object.

TODO:

• have a time stamp per url instead
• maybe don't save to cache if we already have N pages cached for that pair?
• fix it for py32 https://travis-ci.org/goavki/apertium-apy/jobs/197297317