averkij / lingtrain-aligner Goto Github PK

I am using the "%%%%%." term to force separate lines into individual paragraphs. Here is an example:

Neil%%%%%h5.
We’ll discuss that more in a moment and find out if chatbots really think for themselves. But first I have a question for you, Rob. The first computer program that allowed some kind of plausible conversation between humans and machines was invented in 1966, but what was it called? Was it:

a) ALEXA %%%%%.

b) ELIZA %%%%%.

c) PARRY %%%%%.

While this process successfully separates the items into different paragraphs, it doesn't remove the "%%%%%" term, which is consequently retained in the final document.

Делаю
docker pull lingtrain/aligner:v4
Загружаю текстовый файл и...

После вот такого предупреждения ничего не происходит
Причём оно вылазит на любой текстовый файл

when I use “splitted_from = splitter.split_by_sentences_wrapper(text1_prepared, lang_from)” return list，

But I see that there will be a conflict when insert sqlite ，specific error：

File "ling_test.py", line 36, in
aligner.fill_db(db_path, splitted_from, splitted_to)
File "lingtrain_aligner/aligner.py", line 498, in fill_db
db.executemany("insert into languages(key, val) values(?,?)", [("from", lang_from), ("to", lang_to)])
sqlite3.InterfaceError: Error binding parameter 1 - probably unsupported type.

The current version ignores the H1-H5 headers that were added by user. But when book was translate text from chapter 1 will be translate as a chapter 1 text into another language.
You can use this fact and split a big text to small parts.

Next idea - try split a big text to small blocks automatically:
Select a few sentences from original text(for example 10 sentences) and using loop try to find translate block in the thanslated text.

You can use the next psedocode:

left_array = original_sentences[100:110]
sum=[]
for i=50;i<150 do:
   right_array_candidate=translated_sentences[i:i+10]
   sum[i]=sum(cosunuse_distance(left_array,right_array_candidate))

rigth_array=get_index_with_max_value(sum)

left_text_split_index=left_array[0]
rigth_text_split_index=rigth_array[0]