cagdasbas / glacier-rsync Goto Github PK

Rsync like utility to back up files and folders to AWS Glacier. Utility can compress files and store on Glacier. Archive ids will be stored in an sqlite database.

You have to log in to aws with aws cli and create a glacier vault beforehand.

Run params:

$ grsync --help
usage: grsync version 0.3.5 [-h] [--loglevel {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}] [--db db] --vault vault --region region [--compress COMPRESS] [--part-size PART_SIZE] [--desc desc] src

Rsync like glacier backup util

positional arguments:
  src                   file or folder to generate archive from

optional arguments:
  -h, --help            show this help message and exit
  --loglevel {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}
                        log level (default: INFO)
  --db db               database file to store sync info (default: glacier.db)
  --vault vault         Glacier vault name (default: None)
  --region region       Glacier region name (default: None)
  --compress COMPRESS   Enable compression. Only zstd is supported (default: False)
  --part-size PART_SIZE
                        Part size for compression (default: 1048576)
  --desc desc           A description for the archive that will be stored in Amazon Glacier (default: None)

If compression is enabled, file will be read and compressed on the fly and uploaded to glacier multipart.

Sqlite database scheme:

CREATE TABLE 
    sync_history
(id          integer primary key,
 path        text,		/* full path of the backed up file */
 file_size   integer,	/* size of the file */
 mtime       float,		/* modification time */
 archive_id  text, /* archive id generated by glacier */
 location    text, /* archive url generated by glacier */
 checksum    text, /* checksum of the archive generated by glacier*/
 compression text, /* compression algorithm used. NULL if none */
 timestamp   text /* backup timestamp */
);

Currently, there is no way to rebuild it from aws inventory.

• Glacier supports 1024 bytes of description, and I'm currently putting a description in this format:

grsync|abs_file_path|size|mtime|user_desc

Which is not posix compatible since there is no limit to the filename or full path. I can put a metadata in front of every archive but this means that the data can be recovered only with the same tool

• If the absolute file path changes, grsync will treat it as a different file and re-upload
• Currently, there is no way to recover the local database, but you can download the inventory with aws cli and download individual files with the help of description. I maybe create a tool to re-create the local db with inventory retrieval, but the first issue has to be addressed before.