Memento is a tool used to load and save accounts from old Solana snapshots in Google Cloud Storage.
- No fighting solana versioning between your geyser plugin and the validator.
- No fighting the rust compiler between your geyser plugin and the validator.
- Easy programmatic access to bulk snapshot ingestion.
$ cargo b --releaseWorker
$ ./target/release/jito-workers --helpBackfill
The following is a backfill client that backfills to clickhouse.
$ ./target/release/jito-replayer-backfill --help- There's a server and client.
- When starting up the server, it will load in all snapshots available across all GCP buckets you provide on the CLI.
- The clients can connect to the server over GRPC to start backfills and request information on the snapshots available.
Note: During the development a machine with the following specs was used:
- CPU: AMD Epyc 7313P
- Cores/Threads: 16c / 32t
- RAM: 128 GB
- Drive 0: NVME0N1: 15.36 TB
- Drive 1: NVME1N1: 15.36 TB
- Drive 2: NVME2N1: 15.36 TB
Drive 1 and 2 were mounted as an NVME in RAID0 (directions below).
You can get away with NVMEs if you're dumping to an external database. Also, you should tune the number of workers based on your system memory.
Kernel modifications for validator. (Some might be unnecessary.)
$ sudo bash -c "cat >/etc/sysctl.d/21-solana-validator.conf <<EOF
# Increase UDP buffer sizes
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728
# Increase memory mapped files limit
vm.max_map_count = 10000000
EOF"
$ sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
# Increase process file descriptor count limit
* - nofile 1000000
EOF"
$ sudo sysctl -p /etc/sysctl.d/21-solana-validator.confInstall packages
$ sudo apt update
$ sudo apt install -y \
unzip \
mdadm \
libssl-dev \
libudev-dev \
pkg-config \
zlib1g-dev \
llvm \
clang \
cmake \
make \
libprotobuf-dev \
protobuf-compiler \
libpq-dev \
nload \
iotop \
apt-transport-https \
ca-certificates \
dirmngr \
--no-install-recommendsInstall protobufs
export PROTOC_VERSION=21.12 && \
export PROTOC_ZIP=protoc-$PROTOC_VERSION-linux-x86_64.zip && \
curl -Ss -OL https://github.com/google/protobuf/releases/download/v$PROTOC_VERSION/$PROTOC_ZIP \
&& sudo unzip -o $PROTOC_ZIP -d /usr/local bin/protoc \
&& sudo unzip -o $PROTOC_ZIP -d /usr/local include/* \
&& rm -f $PROTOC_ZIP
Install rust:
$ curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
$ source "$HOME/.cargo/env"Mount NVMEs and setup swap (your system may differ)
$ sudo mdadm --create /dev/md0 --level=0 --raid-devices=2 \
/dev/nvme1n1 \
/dev/nvme2n1
$ sudo mkfs.ext4 -F /dev/md0
$ sudo mkdir -p /mnt/disks/ledger
$ sudo mount /dev/md0 /mnt/disks/ledger
$ sudo chmod a+w /mnt/disks/ledger
$ sudo fallocate -l 512G /swapfile
$ sudo chmod 600 /swapfile
$ sudo mkswap /swapfile
$ sudo swapon /swapfile
$ sudo swapon --showSetup and install clickhouse.
$ GNUPGHOME=$(mktemp -d)
$ sudo GNUPGHOME="$GNUPGHOME" gpg --no-default-keyring --keyring /usr/share/keyrings/clickhouse-keyring.gpg --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 8919F6BD2B48D754
$ sudo rm -r "$GNUPGHOME"
$ sudo chmod +r /usr/share/keyrings/clickhouse-keyring.gpg
$ echo "deb [signed-by=/usr/share/keyrings/clickhouse-keyring.gpg] https://packages.clickhouse.com/deb stable main" | sudo tee \
/etc/apt/sources.list.d/clickhouse.list
$ sudo apt-get update
$ sudo apt-get install -y clickhouse-server clickhouse-client
$ sudo systemctl start clickhouse-server
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent