Add CI/CD to this repository.

The translation system with whatever 5k-50k pairs need to go live. This is an issue to prepare the list of tasks to be done for going live.

Start the parallel corpus for En-Or by taking help of the Wikipedia dumps.

• Do word to word and phrase to phrase separate files
• Divide multiple sentences into smaller chunks, may be sentence-wise ?
• Convert the Jupyter notebook code into pure Python code ?
• Lemmatization, Stemming, Inflection and derivation handling

Hello,

We've tried to activate or update your repository on Depfu and couldn't find any supported dependency files. If we were to guess, we would say that this is not actually a project Depfu supports and has probably been activated by error.

Monorepos

Please note that Depfu currently only searches for your dependency files in the root folder. We do support monorepos and non-root files, but don't auto-detect them. If that's the case with this repo, please send us a quick email with the folder you want Depfu to work on and we'll set it up right away!

How to deactivate the project

• Go to the Settings page of either your own account or the organization you've used
• Go to "Installed Integrations"
• Click the "Configure" button on the Depfu integration
• Remove this repo (soumendrak/MTEnglish2Odia) from the list of accessible repos.

Please note that using the "All Repositories" setting doesn't make a lot of sense with Depfu.

If you think that this is a mistake

Please let us know by sending an email to [email protected].

This is an automated issue by Depfu. You're getting it because someone configured Depfu to automatically update dependencies on this project.

Follow TDD or write up unit test cases

• Write unit test cases if possible before code writing

What?
The finalized input pairs need to be further split into Train-Test pairs.
May be we will start with 80% train and 20% test pairs.

Why?
This is needed to evaluate the translation accuracy by the model itself before going to live. To let us know where do we stand.

Blocked by: Issue #9

Figure out which translation framework/library will be used to build the NMT model.
Our options are:

• Keras
• Tensorflow
• PyTorch
• OpenNMT

Mostly we will go with seq2seq model.

Whether to go ahead with Jupyter Notebook or Python files?

Jupyter Notebook

Pros

• Good to work and developer friendly
• Odia fonts, Yuktakhyars (ଯୁକ୍ତାକ୍ଷର) and Unicode characters friendly

Cons

• Difficult to implement
• May create a problem in future while we will be production ready
• In Github I hardly see any serious projects solely on Jupyter Notebook

Python files

Pros

• Production ready
• Can be easily added to PyPi in future
• Unit testing will be easier
• Easy for others to follow

Cons

• Depending on the IDE, some IDEs do not support the Yuktakhyars (ଯୁକ୍ତାକ୍ଷର)
• Analysing and visualizing large amount of data might be a problem

We have to choose which server needs to be chosen to build and host the model. The following things the server should have the following features:

Must required

• Good free tier membership to begin with
• Exporting the data after the free tier expires
• A DB, MongoDB to store the pairs up to 500MB size
• A server which can handle at least 5 queries per second
• Python 3.6+ support

Good to have

• If the system can be trained and online supported then and there
• Later on, we will look into Active learning
• CI/CD with GitHub

On radar

• Google Cloud Platform
• Microsoft Azure
• Amazon Web Services
• IBM Cloud
• Heroku

As users will use the system, it is necessary that we take feedback from them. The model should learn from this feedback.

Also, we need to check all the feedback may not be constructive, so review mechanism needs to be there as well to validate the user feedback.

There will be three kinds of primary feedbacks received from users:

1. Thumbs Up i.e. positive, the user is happy with the translated text
2. Thumbs Down i.e. negative, the user is unhappy with the translation
3. No feedback i.e. user will not give any feedback at all.

The review process should start from the top-most query text and topmost feedbacked text.
It's fine if the review process can not be automated on this release.

The input English-Odia pairs need to be finalized.
With how many pairs we are going to start.

We should go ahead with 5k curated high-quality pairs.
It's fine if the pairs are sentences, phrases or words.

The pairs need to be retrieved from all the Individual files and the Combined file.

The final dataset will be added into this repository.

The UI of the system need to be developed which will have the following features:

Must required

• Two text boxes. One box to type/paste English text and another to display the Odia translation.
• One button to click when the user is ready to translate English to text.
• The button needs to call an API hosted in the cloud server.

Required

• Thumbs up and down buttons to know the feedback of the user on the translated Odia response.
• In case of Thumbs down ---> The user should be asked to edit and provide the correct translation

Good to have

• Uncluttered webpage
• SSO based login ---> [Backend Support] followed by the number of users tried our translation
• [Backend Support] Number of translation pairs trained so far
• [Backend Support] Number of pairs translated by users through the website