Arithmo-Mistral-7B is trained to reason and answer mathematical problems and is also capable of writing a Python program that upon execution prints answer to the question. We used Mistral-7B as a base model and used QLoRA to fine-tune it on a single RTX 4090 GPU.

Arithmo-Mistral-7B outperforms existing 7B and 13B state-of-the-art Mathematical Reasoning models. Refer to Comparing Arithmo-Mistral-7B with other LLM models section for more details.

• Zero-Shot CoT: On providing a question as prompt, model generates reasoning steps to solve the question along with answer. We check if answer matches with ground-truth.
• Zero-Shot PoT: We prompt the model to generate a Python program for the given question. During inference, we execute the Python program generated by the model and check if the program output matches with ground-truth answer. Visit Model Card to see few PoT examples.

pip install transformers >=4.34.0
pip install accelerate
pip install sentencepiece
pip install protobuf

# If you are GPU poor like me
pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/cpu

# If you have a GPU.
pip install --pre torch --index-url https://download.pytorch.org/whl/nightly/cu118
pip install scipy
pip install bitsandbytes

# Set `run_model_on_gpu` to `False` if you are running on CPU. Model will generate reasoning steps with answer for your question. If you want to generate Python program, uncomment line-69 that adds a Python prompt.
# This script automatically does formatting for you, so you just need to type question (eg: `What is 2+2?`) without any prefix like `Question:`, etc.**

$ python query_model.py

Note: Above script automatically does formatting for you, so you just need to type question (eg: What is 2+2?) without any prefix like Question:, etc. Checkout query_model.py for more details.

Question: There are total 10 children. I have to give 1 apple to first child, 2 apples to second child, 3 apples to third child, and so on. How many apples do I need?

Answer: The total number of apples needed is the sum of the first 10 positive integers.
This can be calculated using the formula for the sum of an arithmetic series:
\[S = \frac{n}{2}(a_1 + a_n),\]
where $S$ is the sum, $n$ is the number of terms, $a_1$ is the first term, and $a_n$ is the last term.
In this case, $n = 10$, $a_1 = 1$, and $a_n = 10$.
Plugging these values into the formula, we get:
\[S = \frac{10}{2}(1 + 10) = 5(11) = \boxed{55}.\]
The answer is: 55

Arithmo-Mistral-7B is trained with the following format:

Question: <question>

Answer:

Question: <question> <python_prompt>

Answer:

It will perform best if queried in this way with your own script.

Due to limited compute budget, Mistral-7B model is fine-tuned with QLoRA using Single RTX 4090 GPU. We plan to do a full finetuning of Mistral-7B model on this dataset to further improve performance.

P.S.: Please reach out to Ashvini Jindal if you would be interested in supporting compute need. We are looking for small-scale support so we'd appreciate any kind of help! :)

Model training data is prepared by combining MetaMathQA (train split), lila OOD (train, validation, and test splits), and MathInstruct (train split) datasets. We have verified that our training data has no overlap with GSM8K and MATH test set. Further post-processing steps are applied such as 1) deduplication, 2) randomly lower-casing x% inputs, 3) adding diverse set of Python prompts for PoT, and 4) standardizing answer format. Final dataset is of size ~540,000.

# This script generates train and eval sets.
$ python data_prep/prepare_model_traininig_data.py

Here is Huggingface link for our dataset.

# This script saves output to `data/predictions/gsm8k/Arithmo-Mistral-7B/predictions_Arithmo_gsm8k_zero_shot_CoT.json` path.
$ python eval/gsm8k/gsm8k_generate_response_zero_shot_CoT.py

# This script saves output to `data/predictions/gsm8k/Arithmo-Mistral-7B/predictions_Arithmo_gsm8k_zero_shot_PoT.json` path.
$ python eval/gsm8k/gsm8k_generate_response_zero_shot_PoT.py

# This script saves output to `data/predictions/gsm8k/Arithmo-Mistral-7B/predictions_Arithmo_MATH_zero_shot_CoT.json` path.
$ python eval/MATH/MATH_generate_response_zero_shot_CoT.py

Zero-Shot with PoT: Answers in MATH test set consist of expressions like (x+2)/5 instead of a numeric value. Currently, Arithmo-Mistral-7B's PoT training data doesn't contain expressions as answers. Hence, we don't run PoT based inference on MATH dataset.

$ python eval/gsm8k/gsm8k_compute_metric_zero_shot_CoT.py

Expected output: Total Instances: 1319, Correct Count: 985, Accuracy (Correct Count/Total Instances): 0.7467

# Step-1: This script executes generated python programs and saves results into a file.
$ python eval/gsm8k/gsm8k_write_zero_shot_PoT_outputs.py > data/predictions/gsm8k/Arithmo-Mistral-7B/gsm8k_zero_shot_PoT_results.txt

# Step-2: This script computes accuracy by taking above file as input.
$ python eval/gsm8k/gsm8k_compute_metric_zero_shot_PoT.py

Expected output: Total Instances: 1309, Correct Count: 932, Accuracy: 0.7119

$ python eval/MATH/MATH_compute_metric_zero_shot_CoT.py

Script is borrowed from official math repository
Expected output: Total Instances: 5000, Correct Count: 1266, Accuracy (Correct Count/Total Instances): 0.2532

Results for all models except Arithmo-Mistral-7B are taken from MetaMath repository.