archyyu / gpt-from-mlp-to-rnn-to-transformer Goto Github PK

This repository contains a series of models for generating text, ranging from basic MLP to RNN, GRU, and LSTM. Perhaps in the future, I will also add an attention model.

I created this repo to further my studies in deep learning. If everything goes smoothly, I hope to pursue a PhD position. Wish me the best!

At the moment, I have added a transformer model to generate new characters. The only issue with this repo is that the performance of the deep RNN model I added is not satisfactory. I finally realized that it was my first time attempting to train a deep neural network, so there are likely some tricks I need to learn. I will continue to work on improving the performance of this model.

I finnally tuned the deep RNN to perform better. please check if you are instested.

The other superise is adding attention machesnims to the rnn, so in the output of every time step, it will focus differently on previous step. the performance is better than I thought.