mpiza / mathtransformer Goto Github PK

Implement a deep neural network model that learns to expand single variable polynomials. Model input is factorized sequence and output is predicted expanded sequence.

• (7-3*z)*(-5*z-9)=15*z**2-8*z-63
• (7-3*z)*(-5*z-9) is the factorized input
• 15*z**2-8*z-63 is the expanded target

For the expanded form, only the form provided is considered as correct.

• The directory ./data contains train.txt, validation.txt and test.txt
• The source and target sequence vocabulary is stored in the directory ./vocab
• The trained model (best_model.pt) is present in the directory ./model
• All predictions made by the model on the test is stored in the file ./output/predictions.txt
• Summary for the model and it's trainable parameters is stored in network.txt
• The classes for the transformer model are in - backbone.py and transformer.py
• data.py splits the dataset into train,val and train datasets randomly based on the input split ratio (already split dataset is provided in the repo)
• train.py trains the model using the defined configurations
• test.py runs the trained model on the test data to generate predictions and calculates the accuracy
• text_EDA.ipynb contains the preliminary exploratory data analysis of the dataset
• requirements.txt contains dependencies

Model was trained on a single NVIDIA RTX 3090 GPU with CUDA 10.2 and torch == 1.11.0 (You might have to change the torch version depending upon the GPU and CUDA version of your machine)

• Set up a new conda virtual environment

conda create --name <env_name> python=3.9.2

• Activate the environment

conda activate <env_name>

• Install the dependencies (Run this command in the /Attention directory)

pip install -r requirements.txt

This solution uses the sacred library for logging, running, configuring and organizing the code.

All the commands should be run only from the parent directory (i.e. /Attention)

1. Split data into train, val and test set:

python data.py with 'split_ratio=0.8'

2. Train the model (All configurations can be observed in train.py and they can also be passed from command line as shown):

python train.py with 'hyperparameters.n_iters=20'

Evaluate model on test set (This will utilize GPU if available):

python test.py

Evaluate model on test set (Using CPU only):

python test.py with 'device="cpu"'

The model is evaluated against a strict equality between the predicted target sequence and the groud truth target sequence of the test dataset. The model achieved an accuracy of 98.63% (trained for 20 epochs for 45 minutes on a single GPU).

For a more comprehensive description of the solution, parameter choices and loss plots, please refer to the file Solution-Report.pdf