pleiszenburg / abgleich Goto Github PK

View Code? Open in Web Editor NEW

31.0 4.0 2.0 1.47 MB

zfs sync tool

License: GNU Lesser General Public License v2.1

Makefile 0.60% Python 99.40%

zfs zfsonlinux zfs-backup zfs-snapshots zfs-storage backup backup-utility backup-tool python3 python-cli

abgleich's People

Contributors

Stargazers

Watchers

Forkers

bhohbaum aland-zhang

abgleich's Issues

Importing CLoader (and CDumper) from pyyaml causes crash if not present

Allow custom configurable transfer compressions via pipes, e.g. bzip

Relevant for low-bandwidth connections.

Make SSH-port on source and target configurable

abgleich does not play nice with zfs-auto-snapshot on target

abgleich assumes that its backups on the target side are left alone. zfs-auto-snapshot shoud be made aware of that (if it is running on the target side).

"abgleich" does not fail gracefully if remote hosts are not up

`abgleich` should work in its own isolated namespace for snapshots and ignore snapshots of other tools ...

... as far as possible / reasonable.

target cmpatibility options create tasks, not task lists

If set to yes (and with activated type checking), the following two options will make the backup fail:

target_samba_noshare: yes
target_autosnapshot_ignore: yes

Fix: Tasks need to be wrapped in task lists.

Add an option to skip failed snaps or backups and continue with next dataset/subvol

To run automated, snap and backup should not be stopped after one transaction has failed, if it's not caused by connection issues.
If one snapshot fails, the program should log this and continue with the remaining datasets / subvols when skip parameter is set.

"abgleich restore" for restoring a backup from target back to source in the event of a catastrophic failure

Improve shell-escape for ssh'ed commands

Current implementation is not robust enough, should use shlex.quote instead. ZFS is rather flexible in terms of what it allows in dataset and snapshot names.

Define an option to autocreate prefix on target zpool with subcommand backup

The subcommand backup stops, if the target prefix is not present on the zpool.
By default the program should print a message like The configured prefix was not found on the target zpool. Please ensure you created the prefix and have sufficient permissions..
The workflow with the config option activated could be:

#check if prefix present on zpool
if not exists:
    try:
        # create prefix
    except:
        print (f"Auto-creation of prefix {prefix} on zpool {zpool} on target failed. Please ensure you have sufficient permissions.")

"mountpoint" property assumed to be always present ...

... while it may be unset or set to "none"/"-".

ZVOLs do not always expose a mountpoint property AT ALL. If it is missing, abgleich crashes.

Python 3.6 compatibility broken

Inconsistency in the Python's standard library's typing module: OrderedDict missing.

Versions 0.0.2 to 0.0.5

"ableich [tree|compare|backup|cleanup]" fail to initialize if remote tree is completely empty

Backups of the prefix itself are not yet optional

samba may automatically share/expose new datasets on the target side

It's actually more of a bug in samba, but abgleich could label its backups on the target side in a way that would tell samba to ignore them.

Add recursive 'ignore'

abgleich should support 2 different types of excludes/ignores to be able to build one configuration for a complete zpool with multiple hierachical levels of subvols and datasets.

exclude a single dataset, but include all subsets (seems to bee the current feature)
exclude recursively, docker manages it's volumes itself, creates and snapshots many datasets for the container image layers and I don't want to snapshot or backup them. This could be a second config option or support for wildcards (e.g. var/lib/docker*).

The subcommands tree and compare should only list the included datasets and subvols. For displaying the full data there should be an additional parameter (e.g. --full or --show-excludes)

Config: Not only validate data - also provide defaults

This would make the configuration format extensible while maintaining (some) backwards compatibility.

Config is currently derived from dict. Remove inheritance from dict and make the datatype immutable.

Add `--force` parameter to snap, backup and cleanup subcommand for automation use cases

To allow unattended runs of abgleich [snap, backup, clenaup], it needs a parmeter like --force.

I've already commited code changes to my local clone of the repository:
Added --force parameter to snap.py
Added --force parameter to backup.py
Added --force parameter to cleanup.py
Don't know if the code the best solution, but it's an idea.

Developer install via `make install` failes

... due to import order issue. Regular deployments are not affected.

Version 0.0.5

Some way to backup properties / configurations of datasets

A custom property namespace on the target side could hold this information.

Add optional config key 'port' to connect ssh on non-default port

On public servers I am using a non-default port for the ssh server.
The config file should be extended with an optional 'port' parameter.

On my git server I already made to following change:
Added config value 'port' for non-default ssh port

`abgleich` crashes if pyyaml is installed without C-library support

It actually also works without the libyaml C-library. The import must be fixed.

`tree` and `compare` could produce HTML output

`abgleich` ignores ZVOLs completely

Present in v0.0.2 and v0.0.3.

Nothing works on FreeBSD 12.1

peter@peter-freebsd ~ % sudo abgleich --version
0.0.7

peter@peter-freebsd ~ % uname -a
FreeBSD peter-freebsd.domain.local 12.1-RELEASE-p10 FreeBSD 12.1-RELEASE-p10 GENERIC  amd64

zfs-backup.yaml :

source:
    zpool: tanksys
    prefix: data/homedirs
    host: localhost
    user:
target:
    zpool: zbackup
    prefix: data
    host: localhost
    user:
include_root: yes
keep_snapshots: 2
always_changed: yes
written_threshold: 1048576
check_diff: no
suffix: _backup
digits: 2
ignore: []
ssh:
    compression: no
    cipher: [email protected]

zfs list:

NAME                                                USED  AVAIL  REFER  MOUNTPOINT
tanksys                                             202G  42.7G    19K  none
tanksys/ROOT                                        158G  42.7G    19K  none
tanksys/ROOT/update20200921                         544K  42.7G   991M  /
tanksys/ROOT/update20200921/usr                      76K  42.7G  2.54G  /usr
tanksys/ROOT/update20200921/usr/local                76K  42.7G   111G  /usr/local
tanksys/ROOT/update20200921/usr/local/frpc             0  42.7G  64.2M  /usr/local/jails/cbsd/jails-data/frpc-data
tanksys/ROOT/update20200921/usr/local/frps             0  42.7G  66.8M  /usr/local/jails/cbsd/jails-data/frps-data
tanksys/ROOT/update20200921/usr/local/npm              0  42.7G   191M  /usr/local/jails/cbsd/jails-data/npm-data
tanksys/ROOT/update20200921/usr/local/oauth            0  42.7G  55.1M  /usr/local/jails/cbsd/jails-data/oauth-data
tanksys/ROOT/update20200921/usr/local/php              0  42.7G   179M  /usr/local/jails/cbsd/jails-data/php-data
tanksys/ROOT/update20200921/usr/local/ports-build      0  42.7G  42.8M  /usr/local/jails/cbsd/jails-data/ports-build-data
tanksys/ROOT/update20200921/usr/obj                    0  42.7G    19K  /usr/obj
tanksys/ROOT/update20200921/usr/src                    0  42.7G   335M  /usr/src
tanksys/ROOT/update20200921/var                     188K  42.7G  38.0G  /var
tanksys/ROOT/update20200921/var/log                  85K  42.7G  8.86M  /var/log
tanksys/ROOT/update20200921/var/tmp                   1K  42.7G    20K  /var/tmp
tanksys/ROOT/update20201021                         158G  42.7G   939M  /
tanksys/ROOT/update20201021/usr                     118G  42.7G  2.54G  /usr
tanksys/ROOT/update20201021/usr/local               115G  42.7G   111G  /usr/local
tanksys/ROOT/update20201021/usr/local/frpc         64.2M  42.7G  64.2M  /usr/local/jails/cbsd/jails-data/frpc-data
tanksys/ROOT/update20201021/usr/local/frps         66.8M  42.7G  66.8M  /usr/local/jails/cbsd/jails-data/frps-data
tanksys/ROOT/update20201021/usr/local/npm           191M  42.7G   191M  /usr/local/jails/cbsd/jails-data/npm-data
tanksys/ROOT/update20201021/usr/local/oauth        55.1M  42.7G  55.1M  /usr/local/jails/cbsd/jails-data/oauth-data
tanksys/ROOT/update20201021/usr/local/php           179M  42.7G   179M  /usr/local/jails/cbsd/jails-data/php-data
tanksys/ROOT/update20201021/usr/local/ports-build  42.8M  42.7G  42.8M  /usr/local/jails/cbsd/jails-data/ports-build-data
tanksys/ROOT/update20201021/usr/obj                  19K  42.7G    19K  /usr/obj
tanksys/ROOT/update20201021/usr/src                 336M  42.7G   335M  /usr/src
tanksys/ROOT/update20201021/var                    39.4G  42.7G  39.3G  /var
tanksys/ROOT/update20201021/var/log                9.03M  42.7G  8.89M  /var/log
tanksys/ROOT/update20201021/var/tmp                31.5K  42.7G  21.5K  /var/tmp
tanksys/data                                       41.3G  42.7G    19K  none
tanksys/data/TabSessionManagerBackups              5.17G  42.7G  5.17G  /home/peter/Downloads/TabSessionManagerBackups
tanksys/data/homedirs                              36.1G  42.7G  34.0G  /usr/home
tanksys/swap                                       2.06G  44.0G   729M  -
zbackup                                             162K  1.76T    23K  none
zbackup/data                                         23K  1.76T    23K  /zbackup-data

peter@peter-freebsd ~ % sudo abgleich tree zfs-backup.yaml source
(empty)

In three-machine setup (control/source/target): Option for forcing pull (from target) or push (to target)

(lack of) ability to configure the behavior of `abgleich snap` ...

... i.e. when precisely a snapshot is taken.

Current solution: See here. I perform the following checks on each dataset in exactly this order:

If there is no snapshot, I create one.
If the dataset's "written" property equals zero, nothing has changed. I leave.
If "written" is larger than a megabyte, then something has definitely changed, i.e. I create one.
If the dataset type equals "volume" (and because I know I have not left in check number 2), I create one.
Last, I perform a diff. If it is not empty, I create one.

The above logic was developed with performance in mind. It ignores minor changes like updated "atime" values.

Check (3) should become optional. The "one megabyte" needs to be configurable.

Non-UTF-8 binary output from zfs commands is unhandled

See here.

Define behaviour for unmounted datasets on source side

zfs diff does not work if a data set is not mounted.

Add JSON-Format output parameter

Add a parameter to print output in json format.

Move documentation to RTD

As the project grows, having all documentation in the README file is not helpful at all. Time to move it to RTD.

Three machine setup (control/source/target): Allow source and target to be on the same host

Add documentation ...

... via Sphinx and RTD. Shorten README.md.

"abgleich tree" does only perform host-up check for source side - target missing

Colorized CLI output can upset the typechecker

This issue appears with the current version in master branch:

root@abgleich:~# abgleich snap /etc/abgleich/frequent.yaml
| Typ      |   Geschrieben | Name des Datensatzes   | Snapshot-Name       |
|----------|---------------|------------------------|---------------------|
| Snapshot |      96.0 KiB | ROOT                   | 2021031301_frequent |
| Snapshot |       8.2 GiB | ROOT/pve-1             | 2021031301_frequent |
| Snapshot |      96.0 KiB | data                   | 2021031301_frequent |
Möchten Sie fortfahren? [y/N]: y
Traceback (most recent call last):
  File "/usr/local/bin/abgleich", line 8, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.9/dist-packages/abgleich/cli/snap.py", line 67, in snap
    transactions.run()
  File "/usr/local/lib/python3.9/dist-packages/abgleich/core/transaction.py", line 286, in run
    f'{colorize(" | ".join([str(command) for command in transaction.commands]), "yellow"):s}'
  File "/usr/local/lib/python3.9/dist-packages/typeguard/__init__.py", line 916, in wrapper
    raise TypeError(*exc.args) from None
TypeError: type of the return value must be a tuple; got list instead

Tested with Python3.9 on Manjaro Linux and Debian 11 (bullseye/testing) against a Proxmox 6.3.6 system.
My config:

source:
    zpool: rpool
    prefix:
    host: px
    user: root
target:
    zpool: backup
    prefix: px/prod
    host: px
    user: root
include_root: no
keep_snapshots: 8
always_changed: no
written_threshold: 1048576
check_diff: yes
suffix: _frequent
digits: 3
ignore: []
ssh:
    compression: no
    cipher: [email protected]

Python module versions of my pipenv:

$ pipenv run pip list
Package    Version
---------- -------
abgleich   0.0.7
click      7.1.2
pip        21.0.1
PyYAML     5.4.1
setuptools 53.0.0
tabulate   0.8.9
typeguard  2.11.1
wheel      0.36.2