Bugs: Fail to calulate BLEU when reference corpus is large about fast-bleu HOT 11 CLOSED

Could you please give more information about the dataset you are using and put some sample of it here? Is it publicly available?
I duplicated my dataset to have 1.5 million samples, each of them having 10 words on average and didn't have problem.

from fast-bleu.

For example, reference_corpus = [[str(i) + str(j) for i in range(100)] for j in range(70000)].

from fast-bleu.

I guess maybe it is caused by vocabulary size.
If using reference_corpus = [['a' for i in range(100)] for j in range(70000)], it works.

from fast-bleu.

Yes your guess is close.
I tested your example without issue but the big vocabulary caused large amount of memory consumption.

Does your memory have enough capacity?

from fast-bleu.

Yes, I am running this code on a server with 256GB memory. And the above code occupied about 3GB memory when failing. I guess maybe this issue due to the limit memory of local variables in functions.

from fast-bleu.

Excuse me, waht is your OS , python and fast_bleu version?

As I see in you stack trace, there is something about threading.
Is the code you are running exactly like this?

from fast_bleu import BLEU

bleu = BLEU(reference_corpus, weights)

If your python code is in mult-ithreaded mode, could you please test it in standard and single-thread mode?

from fast-bleu.

I have tested in three severs, and met the same error.
OS: Ubuntu 16.04.1 LTS (GNU/Linux 4.4.0-193-generic x86_64), Ubuntu 16.04.5 LTS (GNU/Linux 4.15.0-120-generic x86_64), Ubuntu 18.04.2 LTS (GNU/Linux 4.15.0-121-generic x86_64)
Memory: 256GB, 128GB, 64GB
Python: 3.7.4, 3.7.4, 3.7.7
fast_bleu: 0.0.86, 0.0.86, 0.0.86

The entire code is shown as follows, without using multi-threading:

from fast_bleu import BLEU

reference_corpus = [[str(i) + str(j) for i in range(100)] for j in range(70000)]
weight = {'4': (.25, .25, .25, .25)}
BLEU(reference_corpus, weight)

from fast-bleu.

Thank you for the extra information.
Could you please install the 0.0.87 version and test it?

from fast-bleu.

Thanks for your help. Now it can run successfully on my server with GCC version 7.5.0.
However, when I installed it on the server with GCC version 5.4.0, it failed to install. Could you lower the barrier to version for more widely usage.

from fast-bleu.

I created a new branch gcc-downgrade and lowered compilation flag to c++11.
I tested it and it seems there is no issue.

Could you please clone the repo and and test whether the gcc-downgrade branch works well or not?
Please uninstall the currently installed version and install the new version with python setup.py install command .

from fast-bleu.

I can run it successfully with GCC version 5.4.0. Thank you very much for your help.

Best wishes!

from fast-bleu.