notai-tech / anuvaad Goto Github PK

Are these models bidirectional? I can easily translate English to Hindi for example, but I don't see how I can translate Hindi back to English.

For example

from anuvaad import Anuvaad
anv = Anuvaad("english-hindi")
anu.anuvaad('नमस्ते।', target_lang="EN", source_lang="HI")
# returns 'नमस्ते।'

Is it not possible with these models to go the other way?

Hi,

I read your point of view here
But still in-depth visual inspection and evaluation is not possible .. as with language there are many possibilities & variation.
Hence benchmarking should be done regardless.

Google, Facebook. HuggingFace do benchmarking on Tatoeba .. its a small sentence dataset and it won't give very drastically different translations. I suggest you guys should post these BLEU Scores with your models.
It will add more trust in users who are going to use it and invest time in.

Thanks.

I've created an MR to add your repository to indic.page. knadh/indic.page#12 is the pull request. Please do let me know if there's any concern

Hey,

I have installed the anuvaad package using pip and it is taking too long for translating the sentences. Is there any solution for this, like how can we optimize the inference time?

from anuvaad import Anuvaad
anu = Anuvaad("english-kannada")
input_text = 'Welcome back home'
output = anu.anuvaad(input_text)

Hi, great work! Can you share the checkpoints and training scripts for further fine tuning on custom dataset.
Thanks.

I am getting that AttributeError when I try on my local terminal. Could you give me requirements.txt ?
Thank you

Hi,

When translating long sentences I am getting part of the translation and other part is missing. It is happening in mostly long sentences.

Some examples given below.

In the early days of artificial intelligence , the field rapidly tackled and solved problems that are intellectually difficult for human beings but relatively straight - forward for computers

कृत्रिम बुद्धिमत्ता के प्रारंभिक दिनों में, क्षेत्र ने तेजी से निपटकर समस्याओं का समाधान किया जो मनुष्यों के लिए बौद्धिक रूप से मुश्किल हैं लेकिन अपेक्षाकृत

Many of the early successes of AI took place in relatively sterile and formal environments and did not require computers to have much knowledge about the world .

एआई की कई प्रारंभिक सफलताएं अपेक्षाकृत स्वस्त और औपचारिक वातावरण में हुईं और दुनिया के बारे में ज्यादा जानकारी रखने के लिए कंप्यूटरों की

Chess can be completely described by a very brief list of completely formal rules , easily provided ahead of time by the programmer .

पूरी तरह से औपचारिक नियमों की एक बहुत संक्षिप्त सूची से शतरंज को पूरी तरह से वर्णित किया जा सकता है, जिसे प्रोग्रामर द्वारा आसानी से समय से आगे

I wanted to know , is there any length restrictions in the current model ?

Thanks,
Sriram

anu = anuvaad("english-malayalam")
Traceback (most recent call last):
File "", line 1, in
TypeError: 'module' object is not callable
anuvaad
<module 'anuvaad.anuvaad' from '/opt/conda/envs/translate/lib/python3.6/site-packages/anuvaad/anuvaad.py'>
anu = anuvaad("english-telugu")
Traceback (most recent call last):
File "", line 1, in
TypeError: 'module' object is not callable

When I run anu = Anuvaad("english-telugu"), I am running into the following error. I have tried to set the flag "from_tf=True" in all the mentioned places, but it did not resolve nor change the error.

Traceback (most recent call last):
File "/home/harsha/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 950, in from_pretrained
state_dict = torch.load(resolved_archive_file, map_location="cpu")
File "/home/harsha/.local/lib/python3.6/site-packages/torch/serialization.py", line 587, in load
with _open_zipfile_reader(opened_file) as opened_zipfile:
File "/home/harsha/.local/lib/python3.6/site-packages/torch/serialization.py", line 242, in init
super(_open_zipfile_reader, self).init(torch._C.PyTorchFileReader(name_or_buffer))
RuntimeError: [enforce fail at inline_container.cc:145] . PytorchStreamReader failed reading zip archive: failed finding central directory

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "", line 1, in
File "/home/harsha/.local/lib/python3.6/site-packages/anuvaad/anuvaad.py", line 96, in init
lang_path, return_dict=True, from_tf=True
File "/home/harsha/.local/lib/python3.6/site-packages/transformers/modeling_utils.py", line 953, in from_pretrained
f"Unable to load weights from pytorch checkpoint file for '{pretrained_model_name_or_path}' "
OSError: Unable to load weights from pytorch checkpoint file for '/home/harsha/.Anuvaad_en-te' at '/home/harsha/.Anuvaad_en-te/pytorch_model.bin'If you tried to load a PyTorch model from a TF 2.0 checkpoint, please set from_tf=True.

Do suggest on how to resolve this. Thanking you !