averkij / lingtrain-aligner-editor Goto Github PK

Visualizations should be paged for the large documents.

In addition to #7 it would be usefull to see a list of lines for both languages that were used more then once, i.e. - duplicated.

Duplicated lines are not shown as conflicts, at least I wasn't able to trigger that manually adding duplicates.

Due to the model inference alignment process can take a long time for the big (1000+ lines) documents. Parallel processing will decrease the calculation time dramatically.

The following points should be considered:

• batches should be independent
• multiprocessing instead of multithreading (it's a CPU bound task and Python code)
• tasks should update some state which will be fetched by UI during the processing

Requested by Alexandra Khramtsova.

For successfully aligned pair of sentences user can add some mark of completion (through the button or the checkbox).
This information can be used for calculation the overall progress and track unaligned sentence pairs.

Now API uses the file orders (1, 2, etc.) during the queries. It is inconvenient because of the excessive checks and inability to determine if the alignment was already started for selected documents or not.

We need

• to create the database and store the uploaded file names along with the IDs
• use them during the API queries
• use them also on Frontend

Requested by Alexandra Khramtsova.

Search sentences by line ID or by text.

Requested by Alexandra Khramtsova.

Select document after uploading.

(Issue described by averkij)
Multilingual pretrained models are currently used. A model for single language pair can be lighter (so that it's easier to use the model locally) and can possibly have better performance.

For reference: a paper on BERT for English and Arabic https://arxiv.org/abs/2004.14519v2

Requested by Alexandra Khramtsova.

Would be useful to have the undo last action functionality.

Requested by Alexandra Khramtsova.

In the alignment candidates list would be useful to have the ability to leave only the unused ones.

Requested by Alexandra Khramtsova.

Add the tooltips on element hover.

Detect the conflicts and highlight the corresponding lines.

Conflict — break in the alignment flow of any kind. More detailed classification of conflicts should be done before doing this issue.

Need to fix the duplication while uploading new files.
Check by filename of checksum.

After moving to the new storage mechanism (on DB) downloading was broken.

Consider the index ordering while implementing the new download mechanism.

Not sure if this is some kind of edge case, triggered by me, or just a problem in the current version, but having alignment done with no pending conflicts - when I go to Create tab and press "Generate preview" or "Download book" there's no result and an exception in console:

03-Jul-21 15:06:36 [ERROR] - 24: Exception on /items/Alex/create/fr/en/89f8ad0e6f154cd1896d34e971153600/preview [POST]
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/local/lib/python3.8/site-packages/flask_cors/extension.py", line 161, in wrapped_function
    return cors_after_request(app.make_response(f(*args, **kwargs)))
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/local/lib/python3.8/site-packages/flask/_compat.py", line 39, in reraise
    raise value
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/local/lib/python3.8/site-packages/flask/app.py", line 1936, in dispatch_request
    return self.view_functionsrule.endpoint
  File "./main.py", line 773, in get_book_preview
    paragraphs, delimeters, metas = reader.get_paragraphs_polybook(
  File "/usr/local/lib/python3.8/site-packages/lingtrain_aligner/reader.py", line 180, in get_paragraphs_polybook
    par_info = [(par_id, par_sent_ids_from, par_sent_ids_to)
  File "/usr/local/lib/python3.8/site-packages/lingtrain_aligner/reader.py", line 180, in <listcomp>
    par_info = [(par_id, par_sent_ids_from, par_sent_ids_to)
  File "/usr/local/lib/python3.8/site-packages/lingtrain_aligner/reader.py", line 446, in get_next_paragraph
    tid_min, tid_max = min(json.loads(item[0][3])), max(
ValueError: min() arg is an empty sequence
[pid: 24|app: 0|req: 1591/3777] 172.17.0.1 () {48 vars in 1012 bytes} [Sat Jul  3 15:06:36 2021] POST /items/Alex/create/fr/en/89f8ad0e6f154cd1896d34e971153600/preview => generated 290 bytes in 328 msecs (HTTP/1.1 500) 4 headers in 160 bytes (1 switches on core 0)
172.17.0.1 - - [03/Jul/2021:15:06:36 +0000] "POST /items/Alex/create/fr/en/89f8ad0e6f154cd1896d34e971153600/preview HTTP/1.1" 500 290 "http://localhost/user/Alex/create/fr/en" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:89.0) Gecko/20100101 Firefox/89.0" "-"