pleiszenburg / abgleich Goto Github PK
View Code? Open in Web Editor NEWzfs sync tool
License: GNU Lesser General Public License v2.1
zfs sync tool
License: GNU Lesser General Public License v2.1
Relevant for low-bandwidth connections.
abgleich assumes that its backups on the target side are left alone. zfs-auto-snapshot
shoud be made aware of that (if it is running on the target side).
... as far as possible / reasonable.
If set to yes (and with activated type checking), the following two options will make the backup fail:
target_samba_noshare: yes
target_autosnapshot_ignore: yes
Fix: Tasks need to be wrapped in task lists.
To run automated, snap
and backup
should not be stopped after one transaction has failed, if it's not caused by connection issues.
If one snapshot fails, the program should log this and continue with the remaining datasets / subvols when skip parameter is set.
Current implementation is not robust enough, should use shlex.quote instead. ZFS is rather flexible in terms of what it allows in dataset and snapshot names.
The subcommand backup
stops, if the target prefix
is not present on the zpool
.
By default the program should print a message like The configured prefix was not found on the target zpool. Please ensure you created the prefix and have sufficient permissions.
.
The workflow with the config option activated could be:
#check if prefix present on zpool
if not exists:
try:
# create prefix
except:
print (f"Auto-creation of prefix {prefix} on zpool {zpool} on target failed. Please ensure you have sufficient permissions.")
... while it may be unset or set to "none"/"-".
ZVOLs do not always expose a mountpoint
property AT ALL. If it is missing, abgleich
crashes.
Inconsistency in the Python's standard library's typing
module: OrderedDict missing.
Versions 0.0.2 to 0.0.5
It's actually more of a bug in samba, but abgleich
could label its backups on the target side in a way that would tell samba to ignore them.
abgleich
should support 2 different types of excludes/ignores to be able to build one configuration for a complete zpool with multiple hierachical levels of subvols and datasets.
docker
manages it's volumes itself, creates and snapshots many datasets for the container image layers and I don't want to snapshot or backup them. This could be a second config option or support for wildcards (e.g. var/lib/docker*
).The subcommands tree
and compare
should only list the included datasets and subvols. For displaying the full data there should be an additional parameter (e.g. --full
or --show-excludes
)
This would make the configuration format extensible while maintaining (some) backwards compatibility.
Config
is currently derived from dict
. Remove inheritance from dict
and make the datatype immutable.
To allow unattended runs of abgleich [snap, backup, clenaup]
, it needs a parmeter like --force
.
I've already commited code changes to my local clone of the repository:
Added --force parameter to snap.py
Added --force parameter to backup.py
Added --force parameter to cleanup.py
Don't know if the code the best solution, but it's an idea.
... due to import order issue. Regular deployments are not affected.
Version 0.0.5
A custom property namespace on the target side could hold this information.
On public servers I am using a non-default port for the ssh server.
The config file should be extended with an optional 'port' parameter.
On my git server I already made to following change:
Added config value 'port' for non-default ssh port
It actually also works without the libyaml C-library. The import must be fixed.
Present in v0.0.2 and v0.0.3.
peter@peter-freebsd ~ % sudo abgleich --version
0.0.7
peter@peter-freebsd ~ % uname -a
FreeBSD peter-freebsd.domain.local 12.1-RELEASE-p10 FreeBSD 12.1-RELEASE-p10 GENERIC amd64
zfs-backup.yaml :
source:
zpool: tanksys
prefix: data/homedirs
host: localhost
user:
target:
zpool: zbackup
prefix: data
host: localhost
user:
include_root: yes
keep_snapshots: 2
always_changed: yes
written_threshold: 1048576
check_diff: no
suffix: _backup
digits: 2
ignore: []
ssh:
compression: no
cipher: [email protected]
zfs list:
NAME USED AVAIL REFER MOUNTPOINT
tanksys 202G 42.7G 19K none
tanksys/ROOT 158G 42.7G 19K none
tanksys/ROOT/update20200921 544K 42.7G 991M /
tanksys/ROOT/update20200921/usr 76K 42.7G 2.54G /usr
tanksys/ROOT/update20200921/usr/local 76K 42.7G 111G /usr/local
tanksys/ROOT/update20200921/usr/local/frpc 0 42.7G 64.2M /usr/local/jails/cbsd/jails-data/frpc-data
tanksys/ROOT/update20200921/usr/local/frps 0 42.7G 66.8M /usr/local/jails/cbsd/jails-data/frps-data
tanksys/ROOT/update20200921/usr/local/npm 0 42.7G 191M /usr/local/jails/cbsd/jails-data/npm-data
tanksys/ROOT/update20200921/usr/local/oauth 0 42.7G 55.1M /usr/local/jails/cbsd/jails-data/oauth-data
tanksys/ROOT/update20200921/usr/local/php 0 42.7G 179M /usr/local/jails/cbsd/jails-data/php-data
tanksys/ROOT/update20200921/usr/local/ports-build 0 42.7G 42.8M /usr/local/jails/cbsd/jails-data/ports-build-data
tanksys/ROOT/update20200921/usr/obj 0 42.7G 19K /usr/obj
tanksys/ROOT/update20200921/usr/src 0 42.7G 335M /usr/src
tanksys/ROOT/update20200921/var 188K 42.7G 38.0G /var
tanksys/ROOT/update20200921/var/log 85K 42.7G 8.86M /var/log
tanksys/ROOT/update20200921/var/tmp 1K 42.7G 20K /var/tmp
tanksys/ROOT/update20201021 158G 42.7G 939M /
tanksys/ROOT/update20201021/usr 118G 42.7G 2.54G /usr
tanksys/ROOT/update20201021/usr/local 115G 42.7G 111G /usr/local
tanksys/ROOT/update20201021/usr/local/frpc 64.2M 42.7G 64.2M /usr/local/jails/cbsd/jails-data/frpc-data
tanksys/ROOT/update20201021/usr/local/frps 66.8M 42.7G 66.8M /usr/local/jails/cbsd/jails-data/frps-data
tanksys/ROOT/update20201021/usr/local/npm 191M 42.7G 191M /usr/local/jails/cbsd/jails-data/npm-data
tanksys/ROOT/update20201021/usr/local/oauth 55.1M 42.7G 55.1M /usr/local/jails/cbsd/jails-data/oauth-data
tanksys/ROOT/update20201021/usr/local/php 179M 42.7G 179M /usr/local/jails/cbsd/jails-data/php-data
tanksys/ROOT/update20201021/usr/local/ports-build 42.8M 42.7G 42.8M /usr/local/jails/cbsd/jails-data/ports-build-data
tanksys/ROOT/update20201021/usr/obj 19K 42.7G 19K /usr/obj
tanksys/ROOT/update20201021/usr/src 336M 42.7G 335M /usr/src
tanksys/ROOT/update20201021/var 39.4G 42.7G 39.3G /var
tanksys/ROOT/update20201021/var/log 9.03M 42.7G 8.89M /var/log
tanksys/ROOT/update20201021/var/tmp 31.5K 42.7G 21.5K /var/tmp
tanksys/data 41.3G 42.7G 19K none
tanksys/data/TabSessionManagerBackups 5.17G 42.7G 5.17G /home/peter/Downloads/TabSessionManagerBackups
tanksys/data/homedirs 36.1G 42.7G 34.0G /usr/home
tanksys/swap 2.06G 44.0G 729M -
zbackup 162K 1.76T 23K none
zbackup/data 23K 1.76T 23K /zbackup-data
peter@peter-freebsd ~ % sudo abgleich tree zfs-backup.yaml source
(empty)
... i.e. when precisely a snapshot is taken.
Current solution: See here. I perform the following checks on each dataset in exactly this order:
The above logic was developed with performance in mind. It ignores minor changes like updated "atime" values.
Check (3) should become optional. The "one megabyte" needs to be configurable.
See here.
zfs diff
does not work if a data set is not mounted.
Add a parameter to print output in json format.
As the project grows, having all documentation in the README file is not helpful at all. Time to move it to RTD.
... via Sphinx and RTD. Shorten README.md
.
This issue appears with the current version in master
branch:
root@abgleich:~# abgleich snap /etc/abgleich/frequent.yaml
| Typ | Geschrieben | Name des Datensatzes | Snapshot-Name |
|----------|---------------|------------------------|---------------------|
| Snapshot | 96.0 KiB | ROOT | 2021031301_frequent |
| Snapshot | 8.2 GiB | ROOT/pve-1 | 2021031301_frequent |
| Snapshot | 96.0 KiB | data | 2021031301_frequent |
Möchten Sie fortfahren? [y/N]: y
Traceback (most recent call last):
File "/usr/local/bin/abgleich", line 8, in <module>
sys.exit(cli())
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 829, in __call__
return self.main(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1259, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/usr/local/lib/python3.9/dist-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/usr/local/lib/python3.9/dist-packages/abgleich/cli/snap.py", line 67, in snap
transactions.run()
File "/usr/local/lib/python3.9/dist-packages/abgleich/core/transaction.py", line 286, in run
f'{colorize(" | ".join([str(command) for command in transaction.commands]), "yellow"):s}'
File "/usr/local/lib/python3.9/dist-packages/typeguard/__init__.py", line 916, in wrapper
raise TypeError(*exc.args) from None
TypeError: type of the return value must be a tuple; got list instead
Tested with Python3.9
on Manjaro Linux and Debian 11 (bullseye/testing) against a Proxmox 6.3.6 system.
My config:
source:
zpool: rpool
prefix:
host: px
user: root
target:
zpool: backup
prefix: px/prod
host: px
user: root
include_root: no
keep_snapshots: 8
always_changed: no
written_threshold: 1048576
check_diff: yes
suffix: _frequent
digits: 3
ignore: []
ssh:
compression: no
cipher: [email protected]
Python module versions of my pipenv:
$ pipenv run pip list
Package Version
---------- -------
abgleich 0.0.7
click 7.1.2
pip 21.0.1
PyYAML 5.4.1
setuptools 53.0.0
tabulate 0.8.9
typeguard 2.11.1
wheel 0.36.2
This may become optional so that only stuff below {zpool}{/{prefix}} is looked at by "abgleich".
Actually, the data should flow directly from source to target.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.