cluetec / lifeboat Goto Github PK

If the user wants to backup a folder from the filesystem, we should provide him the possibility to exclude certain files.
A good practice would be a list of regex paths which should be ignored.

We could also think of a .lifeboatignore file. This file wouldn't make sense in terms of backing up the files of another application, but e.g. if a user wants to backup his local PC it would be a great feature.

Think also directly how a back of an directory to e.g. S3 could look like. I guess we maybe need to adjust our interfaces.

We need a way to provide the destination resource metadata/contextual data from the source resource.

Example use case

filesystem -> filesystem backup where the target file should be named just like the source file suffixed with a user specified string f.e. timestamp.

Rough idea for the implementation

Instead of running the io.Copy directly in the backup command we introduce a new struct (f.e. called BackupProcessor) which takes in the reader and writer for the io.Copy function and in addition to that arbitrary metadata (maybe a map[string]any or we even can decide on a fixed set of possible metadata for each destination type?) and populates the writer with that metadata before running the io.Copy

Some notes for later:

• https://stackoverflow.com/questions/49546567/how-do-you-encrypt-large-files-byte-streams-in-go

• Add basic support for HashiCorp Vault with token based auth #38
• Add support for kubernetes auth #51

We aim to provide support for various resource types (such as sources and destinations), each requiring distinct configurations. Nevertheless, we strive to keep the setup process effortless. Therefore, our proposal is to specify the resource type and the associated configuration at the same level.

For example, the configuration should look something like the following:

source:
  type: filesystem
  path: /take-my-backup-from-here

destination:
  type: filesystem
  path: /store-the-backup-here

The type field indicates the resource type (e.g. filesystem, http, postgresql, mongodb, hashicorp vault, ...). Everything else will be the configuration of the respective resource.

• Add Dockerfile #39
• Add test/build pipeline

For:

• general config
• filesystem source
• filesystem destination

When adding support for new systems, I also test the negative paths: for example, what happens if the HashiCorp Vault token doesn't have the required permissions or isn't valid at all.

During these tests, due to my oversight in changing the path, I often get an error message saying that the target file already exists.

This brings up the question of whether the destination file should be deleted if the backup process fails. In my opinion, there is no reason to keep it, but I am open to other suggestions.

Homebrew supports both macOS and Linux (https://docs.brew.sh/Homebrew-on-Linux), and therefore would be a very good first step towards distributing binaries.

Reference:

• https://docs.brew.sh/How-to-Create-and-Maintain-a-Tap

Especially for testing purposes it and as far as #19 isn't implemented it would be nice to have the possibility to enable the overwriting of existing backups.