deep-diver / en-fr-mlt-tensorflow Goto Github PK

English-French Machine Language Translation in Tensorflow

Jupyter Notebook 14.79% HTML 85.21%

machine-translation deep-learning english-to-french nlp tensorflow

en-fr-mlt-tensorflow's Introduction

MLT (EN to FR ) TensorFlow

In this project, I am going to build language translation model called seq2seq model or encoder-decoder model in TensorFlow. The objective of the model is translating English sentences to French sentences. I am going to show the detailed steps, and they will answer to the questions like how to preprocess the dataset, how to define inputs, how to define encoder model, how to define decoder model, how to build the entire seq2seq model, how to calculate the loss and clip gradients, and how to train and get prediction. Please open the IPython notebook file to see the full workflow and detailed descriptions.

This is a part of Udacity's Deep Learning Nanodegree. Some codes/functions (save, load, measuring accuracy, etc) are provided by Udacity. However, majority part is implemented by myself along with much richer explanations and references on each section.

You can find only the model part explained in my medium post. https://medium.com/@parkchansung/seq2seq-model-in-tensorflow-ec0c557e560f

Brief Overview of the Contents

Data preprocessing

In this section, you will see how to get the data, how to create lookup table, and how to convert raw text to index based array with the lookup table.

Build model

In short, this section will show how to define the Seq2Seq model in TensorFlow. The below steps (implementation) will be covered.

(1) define input parameters to the encoder model
- enc_dec_model_inputs
(2) build encoder model
- encoding_layer
(3) define input parameters to the decoder model
- enc_dec_model_inputs, process_decoder_input, decoding_layer
(4) build decoder model for training
- decoding_layer_train
(5) build decoder model for inference
- decoding_layer_infer
(6) put (4) and (5) together
- decoding_layer
(7) connect encoder and decoder models
- seq2seq_model
(8) train and estimate loss and accuracy

Training

This section is about putting previously defined functions together to build an actual instance of the model. Furthermore, it will show how to define cost function, how to apply optimizer to the cost function, and how to modify the value of the gradients in the TensorFlow's optimizer module to perform gradient clipping.

Prediction

Nothing special but showing the prediction result.

en-fr-mlt-tensorflow's People

Contributors

Stargazers

Watchers

Forkers

lekhapatil cumberbatch08 anshprakash popnem77 headbangedneehar mysqlsc chensvm songkaisong ayanbasak13 chen372 adarsh415 manolo20 ramdhanoriya chenyang-chenyang prisawant lakshya0002 anwarsk jeonggwanlee bcui6611 bransonlee94 xuyannus siddharth67d pne-chahal tvanh512 gracexiaoya kelvinson 13716010896 foundation-models aakash11shukla faraz13 samanjamalabbasi sethips ibrahimkaya754 kibautista bennio behnam-samadi kunal-05 rkassana gupta-anit zafershamim depprasad jajabenit250 scottspace marcosvppfernandes mkeshita vinhsinhnb95 javiloria andylao-ai codejockie asifahsaan

en-fr-mlt-tensorflow's Issues

purpose of stride_slice

In process_decoder_input, what's the purpose of the code below:

after_slice = tf.strided_slice(target_data, [0, 0], [batch_size, -1], [1, 1])

It seems that it just remove the last word from each sentence(a punctuation actually).

Purpose masks

Hello,

Thanks for your repository, it helps a lot !
I am trying to understand the purpose of the masks in your code. I assume the goal is to remove the padding from the loss function, but looking at the get_batches function, it seems that the array of lenghts returned has always the same value as you compute the lengths of the target pad sequence instead of the target sequence. Is it a mistake ? Or the masks has another goal ?

Using Datasets and Estimators

Would be interesting to convert this example to use Datasets and Estimators as it would provide many advantages including TensforBoard, Serving capabilities, etc.

I changed your code to py version, but the experiment was much worse.

I need you help! I converted your dlnd_language_translation v2.ipynb into seq2seq_RNN_model_batch_size.py. On the verification set, the effect of the model is much worse than yours. seq2seq_RNN_model.py I also wrote a seq2seq_RNN_model.py, which differs from seq2seq_RNN_model_batch_size.py in that it does not require a specified batch size.Seq2seq_RNN_model.py is very ineffective on the verification set, basically no change. I record them in the analysis_log file. I hope you can help me find out why. This is a link to my code.

How to adapt the model to Arabic ?

Hello,

Thank you very much for this simple implementation of seq2seq model. however, i would like to know how to adapt decoder to translate English to Arabic, should I reverse the skip connection of the decoder or not ?

Thanks

Exporting model using SavedModel

Firstly, thanks for an amazing tutorial. I wanted to know how simple it would be to export the model using Tensorflow's SavedModel (currently the tutorial uses tf.train.Saver()) after training so it can be uploaded to GCP: ML Engine or used with a TensorFlow Serving server.

Not sure if SavedModel would work with a seq2seq model. Any assistance would be greatly appreciated.

Thank you