Natural Language Processing

Some models I have implemented from scratch in PyTorch using various tutorials and papers.

List of models

Seq2Seq
Seq2Seq with attention (work in progress)
Transformer architecture (based on the `Attention is all you need' paper [1])
GPT-like model from scratch

Model performance

Seq2Seq

Training params:

epochs: 20
learning rate: 1e-3
batch size: 64
input language: german (de)
output language: english (en)
shuffle batches: true
encoder embedding size: 300
decoder embedding size: 300
hidden size: 1024
num layers: 2
encoder dropout: 0.5
decoder dropout: 0.5
teacher forcing percentage: 0.5

Some examples (I came up with) after 20 epochs using the Multi30K dataset:

Input	Model output	Correct translation
Ein Mann läuft entlang eine Straße.	A man walking down a street.	A man walking down a street.
Eine Frau läuft entlang eine Straße.	A woman walking down a street.	A woman walking down a street.
Er ist groß.	A is a man.	He is large.
Ein Mann und zwei pferden ziehen einen Auto aus dem Meer.	A man and two men are a out of a large body of water	A man and two horses are pulling a car out of the sea.
Ein Skateboarder macht einen coolen Trick.	A skateboarder makes a trick trick.	A skateboader does a cool trick.
Hunderte Kinder besturmen einen Mann.	A group of people looking at a man.	Hundreds of children storming a man.
Ein sehr langer Satz der das Netzwerk wahrscheinlich nicht gut verarbeiten kann.	A very very very colorful the the price Angels for the basket.	A very long sentence which the network probably can't handle very well.

The network is actually performing quite well for such a simple architecture! Longer sentences are harder for the network. I also think the dataset is biased towards sentences which are descriptions of things (like "A man walking down a street" or "two dogs taking a bath") so inputs not of this form behave a lot worse (like "Er ist groß").

Nontheless a nice result.

References

[1] - Attention is all you need - A. Vaswani et al. - (12 Jun 2017) - DOI: https://doi.org/10.48550/arXiv.1706.03762

ljbkusters / natural-language-processing-ml Goto Github PK