The Aerospike Backup Service provides a set of REST API endpoints to back up and restore a cluster. You can perform full and incremental backups and set different backup policies and schedules. There are also several monitoring endpoints to check backup information.

Use the OpenAPI generation script to generate an OpenAPI specification for the service. A pre-built OpenAPI specification is available in Swagger format here.

• Getting Started
• User Guide
• Build From Source
• Usage
• Monitoring
• FAQ

Aerospike Backup Service reads configurations from a YAML file provided when the service is launched. See Run for specific syntax. A sample configuration file and docker-compose script will help you get started testing the service. Follow the docker-compose instructions to set up your test environment.

Linux installation packages are available under releases.

Each entity defined in the API specification has endpoints for reading and writing backup configurations at general or granular levels.

For specifics and example values, see the OpenAPI docs.

The endpoints defined within the configuration section permit the user to view or modify the configuration file. Endpoints ending in /config permit reading and changing the entire file at once, while /config/cluster endpoints enable more granular changes.

Cluster configuration entities denote the configuration properties needed to establish connections to Aerospike clusters. These connections include the cluster IP address, port number, authentication information, and more. See POST: /config/clusters for the full specification.

⚠️ Use the Aerospike Secret Agent to avoid including secrets in your configuration.

This entity includes properties of connections to local or cloud storage, where the backup files are stored. You can get information about a specific configured storage option, for example to check the cloud storage location for a backup. You can also add, update, or remove a storage configuration. See the Storage entities under /config/storage for detailed information.

⚠️ ABS currently supports only AWS S3 cloud storage.

A backup policy is a set of rules that define how backups should be performed. It could include information about a backup schedule, criteria for what data is being backed up, and the storage destination. See GET: /config/policies for full details about what parameters are available to customize a backup policy.

You can save multiple policies with different configurations. When you run the POST: /config/policies command to create a policy, ensure that you give your policy a name that will let you quickly identify its characteristics.

Aerospike recommends defining at least one backup and restore policy.

A backup routine is a set of procedures that actually perform backups based on the predefined backup policy. Routines are individually named just as policies are. See the Routines section for command examples showing how to find all routines, get information about a specific named routine, and add, remove, or update an existing routine.

⚠️ Incremental backups are deleted if they are empty and after each full backup. System metadata is backed up only on full backups.

• List backups: Returns the details of available backups. A time filter can be added to the request.
• Restore from a file: Starts a restore operation from a specified backup file/folder.
• Restore from a timestamp: Given a routine name, searches for the closest full backup to the given timestamp and applies the backup in the following order: full backup first, then incremental backups up to the given point in time, if they exist.

% ./backup -h
Aerospike Backup Service

Usage:
  Use the following properties for service configuration [flags]

Flags:
  -c, --config string   configuration file path/URL
  -h, --help            help for Use
  -r, --remote          use remote config file
  -v, --version         version for Use

Set the configuration file path with -c.

Without the -r flag, the file specified after -c is the actual configuration file. With the -r flag, the file specified after -c contains the path or URL to the actual configuration file.

For example, you may store your configurations remotely, such as on AWS S3 storage. In this case, you could have a remote_config.yaml file containing S3 details, and you would run the server with -c remote_config.yaml -r.

Run as a binary using a configuration file:

./target/aerospike-backup-service -c config/config.yml

Run in a container with a custom configuration file:

docker run -d -p 8080:8080 -v custom_config.yml:/app/config.yml --name backup-service backup-service

Example configuration files can be found in the config folder.

The service exposes a wide variety of system metrics that Prometheus can scrape, including the following application metrics:

• /metrics exposes metrics for Prometheus to check performance of the backup service. See Prometheus documentation for instructions.
• /health allows monitoring systems to check the service health.
• /ready checks whether the service is able to handle requests.

See the Kubernetes documentation on liveness and readiness probes for more information.

The HTTP metrics endpoint can be found on the OpenAPI specification page.

• Go 1.22

make build-submodules

Read the official documentation for the library build instructions.

The following command generates a binary under the target directory.

make build

docker build --build-arg GOARCH=amd64 -t backup-service .

docker build --build-arg GOARCH=arm64 -t backup-service .

Run make deb or make rpm based on the desired package manager. This will generate a package in the target directory. See the quick guide on how to get started with the Linux packages.

The service will skip the next startup until the previous backup run is completed.

The service uses the asbackup shared library, which is not currently thread safe. Given this limitation, backup routines are performed in sequence.

The backup service supports AWS S3 or compatible (such as MinIO) and local storage.

• The service may crash if an invalid S3 backup key is provided during configuration.

This section details how to fetch configurations for clusters, policies, and storage options. This is useful for setting up or verifying the configuration of your system.

This endpoint returns the configurations of existing clusters, including the default cluster setup with seed nodes and credentials.

Request:

GET {{baseUrl}}/v1/config/clusters

{
  "absDefaultCluster": {
    "seed-nodes": [
      {
        "host-name": "host.docker.internal",
        "port": 3000
      }
    ],
    "credentials": {
      "user": "tester",
      "password": "psw"
    }
  }
}

Retrieves the configured backup routines.

Request:

GET {{baseUrl}}/v1/config/routines

{
  "routine1": {
    "backup-policy": "keepFilesPolicy",
    "source-cluster": "absDefaultCluster",
    "storage": "local",
    "interval-cron": "@yearly",
    "namespaces": ["source-ns7"]
  },
  "routine2": {
    "backup-policy": "removeFilesPolicy",
    "source-cluster": "absDefaultCluster",
    "storage": "local",
    "interval-cron": "@yearly",
    "namespaces": ["source-ns8"],
    "set-list": ["backupSet"],
    "bin-list": ["backupBin"]
  }
}

Returns all the configured storage endpoints, including, if applicable, cloud storage endpoint information such as region and path.

Request:

GET {{baseUrl}}/v1/config/storage

{
  "local": {
    "type": 0,
    "path": "./localStorage"
  },
  "minio": {
    "type": 1,
    "path": "s3://as-backup-bucket/storage1",
    "s3-region": "eu-central-1",
    "s3-profile": "minio",
    "s3-endpoint-override": "http://host.docker.internal:9000"
  }
}

Provides a list of backups for each configured routine, including details such as creation time, namespace, and storage location.

Request:

GET {{baseUrl}}/v1/backups/full

{
  "routine1": [
    {
      "created": "2024-03-14T13:13:28.96962301Z",
      "from": "0001-01-01T00:00:00Z",
      "namespace": "source-ns7",
      "byte-count": 48,
      "file-count": 1,
      "Key": "s3://as-backup-bucket/storage1/minio/backup/1710422008983/source-ns4"
    }
  ],
  "routine2": [
    {
      "created": "2024-03-14T13:13:29.105744927Z",
      "from": "0001-01-01T00:00:00Z",
      "namespace": "source-ns8",
      "byte-count": 48,
      "file-count": 1,
      "key": "localStorage/filterBySetAndBin/backup/source-ns8"
    }
  ]
}

Destination field says where to restore to. It can be one of the clusters we read in 1st section, or any other Aerospike cluster.

This request restores a backup from a specified path to a designated destination. The no-generation parameter allows overwriting of existing keys if set to true.

In the source section, path is the key value returned as a response in the Full Backup List example. The type parameter under source denotes S3 storage if set to 1 and local storage if set to 0.

Request:

POST {{baseUrl}}/v1/restore/full

Request body:

{
  "destination": {
    "seed-nodes": [
      {
        "host-name": "localhost",
        "port": 3000
      }
    ],
    "credentials": {
      "user": "tester",
      "password": "psw"
    }
  },
  "policy": {
    "no-generation": "true"
  },
  "source": {
    "path": "s3://as-backup-bucket/storage1/minio/backup/1710422008983/source-ns4",
    "type": 1,
    "s3-region": "eu-central-1"
  }
}

The response is a job ID. You can get job status with the endpoint GET {{baseUrl}}/v1/restore/status/:<jobId>.

Response:

123456789

This option restores the most recent full backup for the given timestamp and then applies all subsequent incremental backups up to that timestamp. In this example, the destination and policy fields are the same as in the previous example.

Request:

POST {{baseUrl}}/v1/restore/timestamp

Request body:

{
  "destination": {
    "seed-nodes": [
      {
        "host-name": "localhost",
        "port": 3000
      }
    ],
    "credentials": {
      "user": "tester",
      "password": "psw"
    }
  },
  "policy": {
    "no-generation": "true"
  },
  "routine": "routine1",
  "time": "1710671632452"
}

The response is a job ID. You can get job status with the endpoint GET {{baseUrl}}/v1/restore/status/:<jobId>.

Response:

123456789