Code Monkey home page Code Monkey logo

ansible-backup's Introduction

ansible-backup

GitHub release Travis status

Ansible role which manage backups. Support file backups, PostgreSQL, MySQL, MongoDB and Redis backups.

Redis backup is experimental and only works with AOF disabled.

Supports export of backup status to Prometheus. Metrics are written in a directory in order to be exposed by the Textfile collector. On first deploy, it will create a new export with backup_time = 0, until the first backup is run.

Initially forked from Stouts.backup.

Variables

The role variables and default values.

backup_enabled: yes             # Enable the role
backup_remove: no               # Set yes for uninstall the role from target system
backup_cron: yes                # Setup cron tasks for backup

backup_user: root               # Run backups as user
backup_group: "{{ backup_user }}"

backup_home: /etc/duply         # Backup configuration directory
backup_work: /var/duply         # Working directory
backup_temp_dir: /tmp           # Temporary directory (for restore)

backup_duplicity_ppa: false     # Set to yes to use Duplicity team PPA
backup_duplicity_pkg: duplicity
backup_duplicity_version:       # Set duplicity version

# Logging
backup_logdir: /var/log/duply   # Place where logs will be keepped
backup_logrotate: yes           # Setup logs rotation
backup_prometheus: no
backup_prometheus_dir: /var/lib/node_exporter
backup_node_exporter_group: "{{ node_exporter_system_group | default('node-exp') }}"
                            # Default is compatible with cloudalchemy.node-exporter ansible role.

# Posgresql
backup_postgres_user: postgres
backup_postgres_host: ""

# Mysql
backup_mysql_user: mysql
backup_mysql_pass: ""

# Redis
backup_redis_user: redis
backup_redis_group: "{{ backup_redis_user }}"

# MongoDB
backup_mongo_user: ""
backup_mongo_password: ""
backup_mongo_port: 27017

backup_profiles: []           # Setup backup profiles
                              # Ex. backup_profiles:
                              #       - name: www               # required param
                              #         schedule: 0 0 * * 0     # if defined enabled cronjob
                              #         source: /var/www
                              #         max_age: 10D
                              #         target: s3://my.bucket/www
                              #         params:
                              #           - "BEST_PASSWORD={{ best_password }}"
                              #         exclude:
                              #           - *.pyc
                              #       - name: postgresql
                              #         schedule: 0 4 * * *
                              #         action: backup_purge         # any duply command (read more : https://duply.net/wiki/index.php/Duply-documentation)
                              #         source: postgresql://db_name
                              #         target: s3://my.bucket/postgresql
                              #         work_dir: /var/profile_specific_workdir
                              #         temp_dir: /profile_specific_temp_dir
                              #       - name: mongodb
                              #         schedule: 0 4 * * *
                              #         source: mongo://
                              #         db: mydb # optional, default to backup all databases
                              #         exclude_collection: # optional, allow to exclude collections from backup
                              #          - sessions

# Default values (overide them in backup profiles bellow)
# =======================================================
# (every value can be replaced in jobs individually)

# GPG
backup_gpg_key: disabled
backup_gpg_pw: ""
backup_gpg_opts: ''

# TARGET
# syntax is
#   scheme://[user:password@]host[:port]/[/]path
# probably one out of
#   file://[/absolute_]path
#   ftp[s]://user[:password]@other.host[:port]/some_dir
#   hsi://user[:password]@other.host/some_dir
#   cf+http://container_name
#   imap[s]://user[:password]@host.com[/from_address_prefix]
#   rsync://user[:password]@other.host[:port]::/module/some_dir
#   rsync://[email protected][:port]/relative_path
#   rsync://[email protected][:port]//absolute_path
#   s3://host/bucket_name[/prefix]
#   ssh://user[:password]@other.host[:port]/some_dir
#   tahoe://alias/directory
#   webdav[s]://user[:password]@other.host/some_dir
backup_target: 'file:///var/backup'
# optionally the username/password can be defined as extra variables
backup_target_user:
backup_target_pass:

# Time frame for old backups to keep, Used for the "purge" command.  
# see duplicity man page, chapter TIME_FORMATS)
backup_max_age: 1M

# Number of full backups to keep. Used for the "purge-full" command.
# See duplicity man page, action "remove-all-but-n-full".
backup_max_full_backups: 1

# forces a full backup if last full backup reaches a specified age
backup_full_max_age: 1M

# set the size of backup chunks to VOLSIZE MB instead of the default 25MB.
backup_volsize: 50

# verbosity of output (error 0, warning 1-2, notice 3-4, info 5-8, debug 9)
backup_verbosity: 3

backup_exclude: [] # List of filemasks to exlude

Usage

Add lafranceinsoumise.backup to your roles and set variables in your playbook file.

Example:

- hosts: all

  roles:
    - lafranceinsoumise.backup

  vars:
    backup_env:
      AWS_ACCESS_KEY_ID: aws_access_key
      AWS_SECRET_ACCESS_KEY: aws_secret
    backup_profiles:

    # Backup file path
    - name: uploads                               # Required params
        schedule: 0 3 * * *                       # At 3am every day
        source: /usr/lib/project/uploads
        target: s3://s3-eu-west-1.amazonaws.com/backup.bucket/{{ inventory_hostname }}/uploads
  

    # Backup postgresql database
    - name: postgresql
        schedule: 0 4 * * *                       # At 4am every day
        source: postgresql://project              # Backup prefixes: postgresql://, mysql://, mongo://, redis://
        target: s3://s3-eu-west-1.amazonaws.com/backup.bucket/{{ inventory_hostname }}/postgresql
        user: postgres

S3, Azure, Cloudfiles...

Some backends do not support the user/pass auth scheme. In this case, you should provide the necessary environment variables through backup_env or profile env. The following is an incomplete list from duply documentation.

  • Azure: AZURE_ACCOUNT_NAME, AZURE_ACCOUNT_KEY
  • Cloudfiles: CLOUDFILES_USERNAME, CLOUDFILES_APIKEY, CLOUDFILES_AUTHURL
  • Google Cloud Storage: GS_ACCESS_KEY_ID, GS_SECRET_ACCESS_KEY
  • Pydrive: GOOGLE_DRIVE_ACCOUNT_KEY, GOOGLE_DRIVE_SETTINGS
  • S3: AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY
  • Swift: SWIFT_USERNAME, SWIFT_PASSWORD, SWIFT_AUTHURL, SWIFT_TENANTNAME OR SWIFT_PREAUTHURL, SWIFT_PREAUTHTOKEN

Manage backups manually

Run backup for profile uploads manually:

$ duply uploads backup

Load backup for profile postgresql from cloud:

$ duply postgresql restore /path/to/destination

In /etc/duply/profile/restore, you can find for each profile examples of commands to run to import your restored data in your database.

Also see duply usage

Difference 4.x 5.x

5.x versions are compatible with Ubuntu Focal, which use python3 as default python. 5.x also does not use duplicity team PPA by default, whereas 4.x does.

License

Licensed under the MIT License. See the LICENSE file for details.

ansible-backup's People

Contributors

ben-ji-l avatar evilhamsterman avatar gzoritchak avatar hordijk avatar jezdez avatar jillro avatar klaemo avatar klen avatar maeve-fpf avatar merqlove avatar mig5 avatar nickhammond avatar paulmaunders avatar reminec avatar rutger1140 avatar ties avatar viatoriche avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

ansible-backup's Issues

YAML indentation is wrong in readme and causes error

# Backup file path
    - name: uploads                               # Required params
        schedule: 0 3 * * *                       # At 3am every day
        source: /usr/lib/project/uploads
        target: s3://s3-eu-west-1.amazonaws.com/backup.bucket/{{ inventory_hostname }}/uploads

should be

# Backup file path
    - name: uploads                               # Required params
      schedule: 0 3 * * *                       # At 3am every day
      source: /usr/lib/project/uploads
      target: s3://s3-eu-west-1.amazonaws.com/backup.bucket/{{ inventory_hostname }}/uploads

dependancy missing for B2 target

When using B2 as a target I'm getting "BackendException: B2 backend requires B2 Python APIs (pip install b2)".

Can B2 dependancies be added?

how does backup_purge action work?

Hi,

I'm a bit confused of how to use the backup_purge action. With the example in the readme file, the module creates a separate duply config without regards for the actual backup config, e.g.

- name: create media backup for {{ site.name }}
  include_role:
    name: lafranceinsoumise.backup
  vars:
    backup_max_age: 1M  # keep backups for X months
    backup_max_full_backups: 4  # keep X full backups
    backup_full_max_age: 7D  # max age till a new full backup is created
    backup_profiles:
      - name: "media_{{ site.name }}"
        schedule: "0 2,14 * * *"
        source: "{{ site.path_prefix }}"
        target: "file:///backup/{{ inventory_hostname }}/{{ site.name }}/media_{{ site.name }}"
        user: "{{ site.user }}"
      - name: "purge_media_{{ site.name }}"
        action: "backup_purge"
        schedule: "0 4 * * *"
        source: "{{ site.path_prefix }}"
        target: "file:///backup/{{ inventory_hostname }}/{{ site.name }}/media_{{ site.name }}"
        user: "{{ site.user }}"

will create a media_foo and purge_media_foo config files and to cron tasks:

#Ansible: media_foo
0 2,14 * * * foo_user /usr/bin/duply /etc/duply/media_foo backup >> /var/log/duply/media_foo 2>&1
#Ansible: purge_media_foo
0 4 * * * foo_user /usr/bin/duply /etc/duply/purge_media_foo backup_purge >> /var/log/duply/purge_media_foo 2>&1

Is this the expected behavior? Do I need source & target in the purge config?

mysql password issues in "pre"

Have run into two issues using this playbook with the trellis-backup playbook by @guilro (which I very much appreciate you putting together, filled a need we had). Both issues concern the backup_mysql_password parameter in the pre.j2 template. Since solving them probably involves some overlapping work, reporting them as one single issue here.

First, there is a space inserted between the -p flag and the password variable (line 23). The result is that duply prompts for the password instead of passing it automatically, which causes database backups to fail. That should be simple enough to fix (just need to drop the extra space). I haven't rewritten the template, but I have experimented with manually removing the space on the server and this solves the problem right away, so I'm fairly confident that's the issue.

Second (and the part I'm less sure how to solve), there is an issue if the password contains any special characters (see this stack exchange answer for more documentation). Basically, the presence of a special character without the password being passed in quotes results in an "unexpected token" error. However, I'm less clear in the syntax of this playbook how we get around that, especially while also trying to eliminate the space that causes the first issue.

Debian Support

Hi,
The role looks really great, thanks for publishing it. Since I am using Debian, I would like to add support for it. Would this be of interest? If so, I could work on a pull request.

Cheers!

No package matching 'python-boto' is available

Hi there,

I've been using Xilonz's trellis-backup-role successfully in all my (Trellis) projects, which uses this role as dependency. Backups are uploaded to a Digital Ocean Space using S3.

In my latest project, which uses a Ubuntu 22.04 droplet for the first time (instead of 20.04), I'm not able to successfully provision my droplet with the exact same setup/credentials. I already created an issue in the trellis-backup-role repo for this, but I think the issue originates in this package.

So the error I'm getting is:

"No package matching 'python-boto' is available"

Which is defined here as:

python{{ system_python_version }}-boto

This system_python_version variable is defined here like this:

system_python_version: "{{ (ansible_distribution_release == 'focal') | ternary('3', '') }}"

So when I try to debug this {{ ansible_distribution_release }} variable in my server.yml playbook as a pre_task:

  pre_tasks:
    - debug:
        var: ansible_distribution_release
    - debug:
        msg: "System python version: {{ (ansible_distribution_release == 'focal') | ternary('3', '') }}"

It outputs jammy:

TASK [debug] *******************************************************************
ok: [146.190.29.96] => {
    "ansible_distribution_release": "jammy"
}

TASK [debug] *******************************************************************
ok: [146.190.29.96] => {
    "msg": "System python version: "
}

While it should be focal in order to use python3 correct?
I used the standard Ubuntu 22.04 (LTS) x64 image in Digital Ocean.

When I ssh into my droplet both python --version as well as python3 --version return:

Python 3.10.12

So it's trying to install the python(2x) version of boto while it should be the python3 version?
I tried installing python3-boto manually by adding it to trellis like this:

apt_packages_custom:
  python3-boto: "{{ apt_package_state }}"

But it still fails with the same error, is there anything else I can do to fix this?

Thanks!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.