How to run LLM (mistral 7b) on Raspberry Pi 5

Step-by-Step Guide on how to run Large Language Model on a Raspberry Pi 5 (might work on 4 too, haven't tested it yet)

1. Prerequisite
2. Setup Raspberry Pi
3. Option 1: Run LLMs using Ollama
4. Option 2: Run LLMs using Llama.cpp
5. Extra Resoucres

• Raspberry Pi 5, 8GB RAM
• 32GB SD Card

You can also follow along this YouTube video instead.

1. Connect the SD card to your laptop
2. Download Raspberry Pi OS (bootloader): https://www.raspberrypi.com/software/
3. Run it, and you should see:
   • "Choose Device" - choose Raspberry Pi 5
   • OS, choose the latest (64bit is the recommended)
   • "Choose Storage" - choose the inserted SD card
4. Now click next, and it will ask you if you want to edit the settings, click "Edit settings"
5. Configure
   • enable hostname and set it to raspberrypi.local
   • Set username and password you will remember, we will use them shortly
   • Enable "Configure Wireless LAN" and add your wifi name and password
   • Click save, and contiue. it will take a few minutes to write everything to the SD
6. Insert the SD card to your raspberry pi, and connect it to the electricity
7. SSH into the Raspberry PI:

ssh ssh <YOUR_USERNAME>@raspberrypi.local

1. Install Ollama:

curl -fsSL https://ollama.com/install.sh | sh

2. Download & Run Mistral model:

ollama run mistral

That is it!

1. Install:

sudo apt update && sudo apt install git g++ wget build-essential

2. Download llama.cpp repo:

git clone https://github.com/ggerganov/llama.cpp
cd llama.cpp

3. Compile:

make -j

4. Download Mistral model:

cd models
wget https://huggingface.co/TheBloke/Mistral-7B-v0.1-GGUF/resolve/main/mistral-7b-v0.1.Q4_K_S.gguf)

5. Go back to repo root folder, and run:

cd ..
./main -m models/mistral-7b-v0.1.Q4_K_S.gguf -p "Whatsup?" -n 400 -e

That is it!

• Youtube video on how to run Ollama on Raspberry Pi
• Llama.cpp own example
• Youtube video on how to setup Raspberry PI headlessly
• Adam on Twitter