qmsk / backup Goto Github PK

The qmsk.backup-zfs --zfs-source=... option executes the equivalent of ssh ... zfs send -i #X --bookmark=Y --purge-bookmark=X ... | zfs recv ... to maintain the ZFS bookmark used for incremental sends on the source ZFS. The custom qmsk.zfs-ssh-command wrapper will perform the zfs send, and if it succeeds, create the new Y bookmark and destroy the old X bookmark.

If the zfs recv command on the backup machine gets stuck (due to e.g. openzfs/zfs#11893), then the zfs send may still complete, but the snapshot data will be stuck in SSH/TCP/pipe buffers, and may never end up in the ZFS pool. The next time qmsk.backup-zfs runs, it will attempt to send from the old bookmark X corresponding to its last X snapshot, which no longer exists.

This means that incremental snapshot sends are no longer possible, and full send is required to sync up the backup pool again. The qmsk.backup-zfs command does not provide any option to force a full send, so the backup dataset needs to be replaced using e.g. zfs rename and qmsk.backup-zfs --setup-create 😕

Workaround would be to add some qmsk.backup-zfs --full-send option to perform a full snapshot send, creating a new incremental send bookmark. This could possibly even happen automatically if the incremental send happens with a incremental source (...) does not exist error? Possibly the zfs send wrapper could even fallback to a full send if the incremental source is missing?

Fix would be to separate the --purge-bookmark operation into a separate step. It could probably be done during the next send-recv cycle - then there would generally be two bookmarks present on the source ZFS.

Using USER@HOST:lvm:vgFoo/lvBar as an rsync source fails with an rsync protocol incompatibility error if the rsync client version includes the fixes for CVE-2022-29154:

ERROR: rejecting unrequested file-list name: <FILENAME>
rsync error: protocol incompatibility (code 2) at flist.c(911) [Receiver=3.1.2]

However, the command actually executed by the rsync wrapper looks like this:

/usr/bin/rsync --server --sender -vnlHogDtprxe.iLsfxC --numeric-ids . /tmp/lvm_lvFoo-backup_cbeytaai.mnt/

Note the additional trailing / on the rsync server source, which was NOT present in the original USER@HOST:lvm:vgFoo/lvBar rsync source. This triggers the protocol error, and using USER@HOST:lvm:vgFoo/lvBar/ (with trailing /) works as an rsync source.

The extra trailing / is added by the path logic in the rsync wrapper's LVMSource - due to the qmsk.rsync-ssh-command wrapper's use of temporary mountpoints for the LVM snapshots, it doesn't make any sense to send the mountpoint directory as-is, only its contents:

The mismatch between the client and server trailing slash worked just fine in earlier versions of rsync, but breaks the increased validation of file names required by the CVE-2022-29154 fix. This is not an rsync regression, but rather a qmsk-backup bug and resulting rsync incompatibility.

The rsync wrapper logic must be fixed to require the client to supply the trailing / for any send from an LVM snapshot source, with a clear error message if this is missing. The rsync wrapper must never attempt to insert any additional trailing /, as this will fail with the fixed rsync validation.