How many iterations does it run and ETA ? about fast_align HOT 3 CLOSED

Kinda figured it out going through the code base, I think it runs 5 iterations by default. Takes about 20 min /iteration on a t2.xlarge machine.

Mohammed Ayub

from fast_align.

I have problem with larger dataset while using ./fast_align
I found that I could not deal with the data more than 1000 lines maybe. Have you ever met the same problem or how do you usually run the script?
For me it is:
./fast_align -i ~/sockeye_autopilot/systems/wmt14_en_de/data/bpe/new_combined_srctrgaa -d -o -v > forward.align_endea
Thanks a lot!

from fast_align.

@Ryanight I have no problem running files with more than 1000 lines. Maybe the input file has some problems. I have created the input file and the running the commands like below:

create the input file:

pr -mtJS' ||| ' /home/ubuntu/mayub/datasets/train_tokenized_bpe_applied.en /home/ubuntu/mayub/datasets/train_tokenized_bpe_applied.es > /home/ubuntu/mayub/datasets/fastalign_input_en_es.txt

forward train:
fast_align/build/fast_align -i /home/ubuntu/mayub/datasets/fastalign_input_en_es.txt -d -o -v > /home/ubuntu/mayub/datasets/forward_align_en_es.align

reverse train:
fast_align/build/fast_align -i /home/ubuntu/mayub/datasets/fastalign_input_en_es.txt -d -o -v -r > /home/ubuntu/mayub/datasets/reverse_align_en_es.align

symmetrize:
fast_align/build/atools -i /home/ubuntu/mayub/datasets/forward_align_en_es.align -j /home/ubuntu/mayub/datasets/reverse_align_en_es.align -c grow-diag-final-and > /home/ubuntu/mayub/datasets/corpus_en_es.gdfa

Unfortunately i have not found a good tool that can detect errors or troubleshoot the above process if the files contain error in alignments, which may in some cases. Open to suggestions.

from fast_align.