jff-bohdan / yabtool Goto Github PK

Add support of logging into file. File path need to be specified in command.

Should be supported two types of log files:

• big log file that will be appended;
• log file for each execution.

Big log file need to be rotated with some rule (by size, by age).

Small file should be used in future to send as packed attachment in email notification.

Add time (now only date) for objects uploaded to S3

Add ability to send emails after flow execution finished.

Log need to be added as attachment, body should contain statistical information.

Show information (for example in table format) with statistical information about:

• time required for each step in flow;
• which steps voted for flow skipping and why (detailed explanation);
• individual statistics for each step (if step want to show some data). S3 backup can show: how many objects were uploaded (names + size), copied between buckets (where and from), deleted (names), time required for object upload.

Add simple garbage collector for session logs that added using key --add-session-log. This collector should:

• start in folder for session logs;
• find possible candidates for session logs by analyzing file names by mask;
• remove outdated files (by date when file was created).

Calculate MD5 for each file that will be uploaded to S3. After uploading validate uploaded objects for MD5 sum and for size of object.

Step s3_multipart_upload should support adding storage class for objects uploaded to S3

Add ability to configure --add-main-log. Add into configuration file (with ability to overwrite from secrets file) options for:

• file retention policy;
• file rotation policy;
• flag that compression enabled.

Step s3_multipart_upload should support adding life cycle transitions for objects uploaded to S3

Add support of PostgreSQL backups

Automatically remove outdated weekly backups. Optionally take maximal amount of weekly backups and removed outdated objects from S3

Add flow step to save result files on local FS with rotation support similar to s3_multipart_upload

Improve dry run for S3 upload step to check that all required permissions given.

This step should:

• check that bucket exists;
• list keys in bucket;
• create non empty object (with dummy data) for each upload rule;
• tag object with some tags;
• copy object from one key to another.

In future dry run should perform also:

• set life-cycle for object;
• set end-of-life for object.

Compress session log with zip before attachment

Add ability to override parameters for specific steps in flow. For example, step firebird_backup contains parameter command_template which is useful to override for debugging purpose by adding -metadata. This parameter allows to dramatically decrease time required for one iteration

Add application version as jinja variable for emails generation

Step s3_multipart_upload should support adding of custom tags specified in file with secrets

Add ability to override global parameters (section parameters in configuration file) by using specific section in secrets file

Add ability to passthrough command line arguments to the flow steps. Yabtool flow orchestrator should be able to passthrough unknown arguments to flow steps. argparse.ArgumentParser::parse_known_args should be used to perform this operation, unknown arguments should be passed for further parsing by flow steps

Some flows may be skipped in case if required artifacts already available in target storage.

For example: entire fb7zs3-flow may be skipped if step s3_multipart_upload will decide that all artifacts required for this execution already available in target S3 bucket.

In some cases more than one step may vote for skipping and/or flow execution. Yabtool should support multi-vote scenario

Command line argument --force-flow should allow to disable voting for flow skipping

Improve email template - notify if flow was skipped

Add ability to add and use additional variables for parameters rendering values.

For example: secrets file may define additional values that may be used in secrets file.

Send plain text report as attachment (without compression)

Add new flow - single run S3 backup (ET approaches in work).

Flow should execute FB backup and perform regular steps and then upload result object to the S3 bucket with specific key. No object rotations should be applied.

Flow should:

• perform backup;
• compress and encrypt file, calculate hashed;
• generate name folder (in S3) with current timestamp and optional suffix provided by user (from command line). Example key 20191119T200304_new_backup;
• upload backup file to S3 with generated key.