Utilize hashi-up
to create a nomad
/consul
(with optional vault
) cluster on 3+ Raspberry Pis.
- 3 Raspberry Pi 4B 8GB
- server
- Hostname: rpi-cl-srv1
- OS: Raspbian Lite x64
- agent 1
- Hostname: rpi-cl-ag1
- OS: Raspbian Lite x64
- agent 2
- Hostname: rpi-cl-ag2
- OS: Raspbian Lite x64
- server
- PDM
- If you're not using PDM, you can run
pip install -r requirements.txt
- (Optional, recommended) Create a virtual environment:
venv .venv
. venv/scripts/activate
pip install -r requirements.txt
- (Optional, recommended) Create a virtual environment:
- If you're not using PDM, you can run
- A
BASH
shell
- Requirements:
hashi-up
- SSH credentials
- On the server, generate a key with
ssh-keygen
, i.e.$ ssh-keygen -t rsa -b 4096 -f ~/.ssh/rpi_cl_id_rsa
- For some reason, even when the key is added to the remote during the
cloud-init
phase, I've found I still need to copy the ID withssh-copy-id -i ~/.ssh/<cluster_ssh_key> <user>@<rpi-host>
for each new server/agent
- On the server, generate a key with
- Choose 3 microSD cards of the same capacity & speed
- Use the Raspberry Pi Imager to write the
Ubuntu Server <major_ver>.<minor_ver> x64
image to each SD card- As of writing this guide, I used the
Ubuntu Server 23.04 (64-bit)
image underOther general-purpose OS > Ubuntu
category
- As of writing this guide, I used the
- Create/copy the
cloud-init
files (cmdline.txt
,meta-data
,network-config
,user-data
)- Examples of these files can be found in the
pi_cluser/reference_files
directory - Make sure to edit the following in each file:
cmdline.txt
- No edits
meta-data
instance_id
: Set to the chosen hostname for the Pi host
network-config
addresses
: Set an address in the line that starts withaddresses
- Example:
... eth0: ... addresses: - 192.168.1.50
user-data
hostname
- Set to the chosen hostname for the Pi host
fqdn
- Set the Pi's FQDN, i.e.
hostname.domain
- Set the Pi's FQDN, i.e.
users > ssh_authorized_keys
: Set thessh-rsa
key(s) to copy/create on setup- Beforehand, create an SSH key to share for management of the Pi cluster
- The private key (i.e.
id_rsa
) should exist on every server/computer that will connect to the cluster. DO NOT COPY THE PRIVATE KEY TO THE PIs!!! - The public key (i.e.
id_rsa.pub
) should be copied to each Pi in the cluster - You can
cat id_rsa.pub
and copy the output tossh_authorized_keys
in theusers
section
- The private key (i.e.
- Beforehand, create an SSH key to share for management of the Pi cluster
netplan
(section begins with- path: /etc/netplan/custom-cloud-init.yaml
)- In the
content
section, look for a line that starts withaddresses:
- Edit the address, i.e.
192.168.1.50/24
- Edit the address, i.e.
- In the
- Examples of these files can be found in the
- Once
cloud-init
is complete on the Pis, if you can't log in via SSH, manually copy the public key to each hostssh-copy-id -i /path/to/public_key.pub <pi-user>@<pi-ip/pi-hostname>
- For easy SSH by hostname (i.e.
ssh rpi-cluster-host1
), add a line to your~/.ssh/config
for each host:Host rpi-cluster-host1 HostName 192.168.1.xxx User <pi-user> IdentityFile ~/.ssh/private_key
This project uses Ansible to automate setup of the Pi cluster. The ansible/ directory stores:
-
- Inventory files are where the Raspberry Pi hosts are configured
- On a fresh clone, make live copies of:
- rpi-nomad-cluster's
inventory.example.yaml
->inventory.yaml
. - rpi-nomad-cluster/group_vars's
all.example.yml
->all.yml
- rpi-nomad-cluster's
-
- Store Ansible playbooks
- These playbooks organize Ansible Roles into repeatable automations
- Example:
base-setup.yml
combines multiple roles in a specific order to automate the initial setup of a new cluster host.- This configuration is the base "shared" configuration that all hosts in the cluster will share
- Any agent or server-specific configurations should be defined separately.
-
- Ansible Roles let you automatically load related vars, files, tasks, handlers, and other Ansible artifacts based on a known file structure. After you group your content in roles, you can easily reuse them and share them with other users.
- The roles defined in this repository serve to automate setup of packages, configurations, security (firewalls, etc), & more
- The roles here are custom, but Ansible has a "role repository" called Ansible Galaxy
-
Create inventory files
- Copy
onboard/group_vars/all.example.yml
->onboard/group_vars/all.yml
- To change Python interpreter on remote, edit
ansible_python_interpreter
- Set a user for connecting to the remote
- User must exist on remote already, and host must be configured to connect to that user by SSH.
- Check the Raspberry Pi Setup section for instructions on SSH setup.
- (Optional) Paste your
tailscale
pre-shared key, if cluster will be configured with Tailscale- You can create a pre-shared key from Tailscale's web management console
- (Optional) Configure git for the remote
- To change Python interpreter on remote, edit
- Copy
onboard/inventory.example.yml
->onboard/inventory.yml
- Edit the host IP address(es)
- Copy
rpi-nomad-cluster/group_vars/all.example.yml
->rpi-nomad-cluster/group_vars/all.yml
- Edit
ansible_ssh_private_key_file
- Create an SSH key beforehand, write path to the private key
- Edit
- Copy
rpi-nomad-cluster/inventory.example.yaml
->rpi-nomad-cluster/inventory.yaml
- Set host IP address(es)
- Copy
- Cloud-init docs
- Cloud-init reference
- Hashi-Up's creator demo of a nomad cluster
- Running Hashicorp Nomad, Consul, Pihole and Gitea on Raspberry Pi 3 B+
- Github: Ubuntu 20.04+ WiFi & cloud-init RPi4B
- Jorge Sancho Personal Blog: Raspberry cluster up and running
- Github: mjpitz/rpi-clout-init reference repo
- Github: timebertt/pi-cloud-init reference repo
- Building a Nomad Cluster on Raspberry Pi Running Ubuntu Server
- Running Hashicorp Nomad, Consul, Pihole, & Gitea on Raspberry Pi
- Deploying a highly-available Nomad cluster with hashi-up
- Traefik in Nomad using Consul & TLS