fritz-hh / scripts_nas4free Goto Github PK

Message from linmag7 on NAS4Free forum (http://forums.nas4free.org/viewtopic.php?f=66&t=2750&p=14917#p14917):

I've set up a cron job now on my NAS4Free and it works fine so far. However, in the config.sh file I had to append a "-i /path/to/my/id_rsa" to the "BIN_SSH="/usr/bin/ssh" line. Otherwise ssh failed to log in to the remote backup server. This is not necessary if I just run the script from the command line since then it uses the default path to the id_rsa file in the users .ssh directory.

The script shall support a debug mode that can be selected in config.sh.
Selecting the debug mode shall lead to additional log verbosity.

In summary, the data echoed if no logged data are available, is not formatted correctly

The backupData script only works if the snapshots where generated using the naming conventions used in manage snapshots.

Some script create lock files when starting (e.g. backup.sh)
If the script stop abnormally, the lock file is not deleted. As a consequense, the script will fail to start the next time it is called.

The reports generated by the scripts should make clear which version of the script package is installed

It should be possible to set the compression of the destination fs as argument.
The compression argument should not only be used when creating the fs (compression algorithm of the fs should be updated when called the script with a different compression argument)

Idea: allow both:

• to define one compression algorithm for all fs to be replicated
• to define one compression algorithm for each fs to be replicated

Error mails are sent "randomly" without any apparent reason. (the log files do not contain any error or warning message)

This was observed in various scripts: backupData.sh, checkSpace.sh

The script should not accept that the user defines that "-" snapshots should be kept.

The script do not support providing as argument an fs name that contain more than one "/"
e.g.:

• "tank/data" is supported
• "tank/data/user1" is not supported

It is not possible to run the scripts from another working directory (e.g. with cron) if the configuration variables "CFG_TMP_FOLDER" and "CFG_LOG_FOLDER" contain relative paths

Check if the right number of mandatorry arguments are provided to the respective shell scripts

It would be neat a add a feature in manageSnapshots that would (optionnally) delete a snapshot if it does not taken any change compared to the previous one

If the scripts is killed by the user, or if the server is shutdown manually by the user, the scripts is no longer able to track the ACPI state of the NAS. These untracked time spans shall be recorded in the log, so that statistics can take them into account.
Ideally, it would be possible to distinguish between the script being killed from the NAS being shutdown.

Please update the scripts to make them more robust.
They should also behave correctly if the pool level property "listsnaps" is ON
(e.g. manageSnapshots.sh fails currently)

Note: If listsnaps is ON, then "zfs list" returns not only a list of fs and volumes but also the list of all snapshots

Currently, the destination must be a ZFS pool.
It would be more flexible to allow any ZFS FS as destination
This would allow to have backups of several NAS on a common destination NAS Pool, but on different ZFS FS.

Example:
(this example assumes that backupData.sh allready supports remote replication: see issue #1 )

NAS1: NAS owned by user1 (has 1 pool: tank_user1)
NAS2: NAS owned by user2 (has 1 pool: tank_user2)
NAS3: NAS owned by user1 and 2 and user for backups (has 1 pool: tank_backup)

in the latter case it would be good to be able to specify the FS "tank_backup/user1" as destination pool

It should be possible to configure the script to create 0 snapshot for any type (hourly, daily...)

The script sets the whole destination pool read only. This is not a good idea if other fs of the pool are used for other purposes as backup (it would be better to set only the resp. Destination fs of the backup to RO)

The current version of backup.sh only support replication on another pool on the same computer.
It would be helpful if backup.sh would support remote replication (i.e. to another computer on the network)

If the lock folder did not exist, and if two scripts are started at the same time, they may both try to create it. This leads to an error and the script exits.
The function create-lock-folder should be corrected

Hi,

thanks for your scripts.. & effort ..

i bump into small error with checkSpace.sh regarding quota filed of volumes.
it seems that the value return is not a number but the minus sign ("-"):
(this create an expression error in your calculation)

> zfs list -t volume
NAME USED AVAIL REFER MOUNTPOINT
tank/tstis 51.6G 6.74T 13.3G -
> zfs get -o value -Hp quota tank/tstis
-
>

i added small check about this (i am not a *nix guy at all so you might want to do it differently):

    getQuota() {
            local fs q
            fs="$1"

            # compute the quota of the filesystem in byte
            q=`$BIN_ZFS get -o value -Hp quota $fs`
            # check if quota is not a number, for volumes it migh be "-"
            ! [ $q -eq $q 2>/dev/null ] && q=0   
            echo $q
    }

all the best,
hagai

In case data corruption occur in a pool, it would be helpful to get more verbose of the actually affected files. Currently this more verbose data are not reported. E.g.:

20181103_110000	INFO	-------------------------------------
20181103_110000	INFO	Starting checking of pools
20181103_110000	INFO	pool 'tank' is healthy
20181103_110000	ERROR	  pool: tank_backup
20181103_110000	ERROR	 state: ONLINE
20181103_110000	ERROR	status: One or more devices has experienced an error resulting in data
20181103_110000	ERROR		corruption.  Applications may be affected.
20181103_110000	ERROR	action: Restore the file in question if possible.  Otherwise restore the
20181103_110000	ERROR		entire pool from backup.
20181103_110000	ERROR	   see: http://illumos.org/msg/ZFS-8000-8A
20181103_110000	ERROR	  scan: scrub repaired 41.5K in 4h21m with 2 errors on Thu Nov  1 16:26:07 2018
20181103_110000	ERROR	config:
20181103_110000	ERROR		NAME        STATE     READ WRITE CKSUM
20181103_110000	ERROR		tank_backup  ONLINE       0     0     2
20181103_110000	ERROR		  ada1      ONLINE       0     0     5
20181103_110000	ERROR	errors: 1 data errors, use '-v' for a list

The log file path is not assined correctly in checkSpace.sh if none if the options -f, -t are set

Hello Fritz,

Thank you for these scripts! They appear to be exactly what I was looking for (manageSnapshots.sh and backupData.sh in particular). I've just got one issue ... I'm trying to backup datasets on my primary storage server to a backup server. The servers are very similar in configuration, right down to the names of the pools/datasets. When I invoke backupData.sh from the command line:

./backupData.sh -s [email protected] Pool01/Misc Pool01/Misc

Nothing much happens, and I get the following in backupData.sh.log:

20171006_165920	INFO	-------------------------------------
20171006_165920	ERROR	SSH connection test successful.
20171006_165920	ERROR	The source filesystem "Pool01/Misc" is in the same pool than the destination filesystem "Pool01/Misc"

Am I doing something wrong? It also occurs to me that perhaps the script requires the storage pool to have a different name, even when located on another server. Any comment?

Thanks again!
Chris

In case a remote duplication (backup) is done from a NAS4Free server to another NAS4Free server, the server that is the destination of the backup should be prevented from shuting down.
One solution would be have an option to prevent shutdown if there is an active ssh connection to that server.
A way to detect such a connection is:

sockstat -c -p22 | grep sshd

The validity of the configuration parameters in config.sh (especially the pathes) are not all checked.
This may lead to unwanted behaviour in the scripts