redmop / sanoid Goto Github PK

Weekly_day
Weekly_time
Daily_time
Monthly_day (allow last day of the month)
Monthly_time

If no snapshots of a specific time exist, still take immediately.
If snapshots are older than the time interval, take immediately, then again at the requested time

It's easier to see what is making the snapshots if you're using more than one script.

This will allow programs to request snapshots.
I will mainly be using this for apt pre and post installing.
I'm not sure how to do this yet, maybe with trigger files.
The script writes a file to the disk, then waits for it to be removed before continuing (with timeout)
A program (probably incron) watches for this trigger file, and takes the snapshot

Don't forget to expire the snapshots.

Defaults for pv are "-p -t -e -r -b". They all need included if average is used. Let's spell them out "--progress --timer --eta --rate --bytes"
Size for pv is "-s " Let's spell it out "--size "

Average for pv is "-a" Let's spell them out "--average-rate"

Rate limiting for pv is "-L " Lets spell it out withg "--rate-limit "

Syncoid snapshots should have syncoid_source_destination type names. This will help with cleanup scripts when removing snapshots which have nothing to do with the host they are on. In other words, syncoid_hosta_hostb
syncoid_hosta_hostc

On hostb, hostc snapshots are not needed, and vice versa, so they can be deleted after X time. Maybe look to see if they exist on hosta, and delete them if they do not.

Frequent_interval (minutes)
Frequent_time (round to the even amount? If interval is 15, round to 0, 15, 30, 45?)

Many other scripts use a 15 min interval. Let's add that.

use mbuffer
allow configurable port

lbzip2
pbzip2
pigz

That way, you can do cleanup after hours.

Allow amounts by 1G, 1M, 1K etc
Allow percents with minimum (10% min 128M)

If no snapshots of a type, make one immediately.
If the time comes to make a specific snapshot (as in weeky_day, weekly_time or frequent_interval), take snapshot
Otherwise, don't take a snapshot unless enough time has passed to take another.

The problem:

Snapshots are not taken atomicly. Each object in the config file is taking individually.

Example:

DatasetA has a database
DatasetB has a file repository

A snapshot of DatasetA is taken
Both datasets are updated (database update, file added to repository)
A snapshot of DatasetB is taken
Your datasets are now out of sync. The database update is not saved, so the file added to the repo is orphaned.

Example 2:

DatasetA has a database
DatasetB has a file repository

A snapshot of DatasetB is taken
Both datasets are updated (database update, file added to repository)
A snapshot of DatasetA is taken
Your datasets are now out of sync. The database update is saved, but the file repo update is not, leaving orphaned data in the database. This is worse than the above.

Possible fix:

Take all the snapshots in a single command. As all the snapshots are pushed to an array n sanoid, this should need only minor changes near system($zfs, "snapshot", "$snap"); This is also needs fixing in syncoid, something around my $snapcmd = "$rhost $mysudocmd $zfscmd snapshot $fs\@$snapname\n"; This should not change expiring.

This will allow updates to have their own snapshots with their own expiration. Include expiration