These should really come from the .backup file so they are accurate. For now I think it's enough to remove them and them recommend to check the .backup file in the restore docs.

Check to be sure that the backup being resumed is the same type that is being attempted:

• If full, previous backup should have been full
• If differential, resumed backup should have been differential and full backup should be the same
• If incremental, resumed backup should have been incremental and previous backup should be the same.

ALSO: If the version number has changed do not resume. If there is no manifest or version file do not resume.

The backup type will need to be written into each manifest for this to work.

Allow any config option to be set/overridden on the command line. Even a full config could be built this way.

This is still contained in backup code and needs to be moved to manifest code. Once this is done, the rest of the synthetic restore unit tests can be written.

Since backrest maintains it's own log directory it should also expire logs automatically.

Default to 30 days and make it user configurable. [Per Russell's suggestion make the default = to retention-full?]

Write unit tests for expiration.

Some kind of info command to see when the last backup was done for each, this can be done by stanza or globally.
It would be nice to see:

1. Last backup
2. Number of backups of each type and date
3. What backups are currently running
4. Failed backups in temp

The internal format for this should be JSON so it can easily for formatted into text, Nagios alerts - whatever.

if db::path is specified on the command line then if tablespaces are not also on the command line then put them under the base pg_tblspc path without linking. If tablespaces are mapped in the config, then make sure a warning is thrown to show that they have been remapped.
This is to prevent an existing installation from being clobbered if the base path is remapped for the purpose of testing.

This will play nicer with backup software and prevent anyone from trying to restore a backup that is currently being deleted.

Maybe instead have a backup manifest that lists the valid backups.
Also add a feature to check the presence of all files before a restore.

Delete from backup.info before actually removing the directory. Move the directory deletion code to the end - scan the db path and delete anything that is not in backup.info.

Currently this is manual - need to make it automatic.

Actually changed this to info since it has never caused any issues - still, it should be fixed. From the code:
# If the dir is not empty check if the files are in the manifest
# If they are, then error - there has been some issue
# If not, they are new - continue processing without error - they'll be picked up on the next run

Process backup and restore files in reverse order. This is more efficient at the end since you're less likely to end up with a single thread processing the last big file.

Refactor code to allow manifest to be saved every minutes. This will reduce the number of files that need to be recopied on a resume.

Create a format settings that keeps track of when the on-disk/manifest format changes for backrest. This is will determine whether backups are compatible.

Who doesn't need help?

Getting one archive file at a time can be tedious if the cluster is very far behind. An async get with some sort of prefetch would speed the process a lot.

Should be able to specify how may archive logs to prefetch.

This should make development restores easier.
Perhaps a command line option --no-tablespace?

This should probably be done in html so there is more flexibility.

• Lots of work needed on restore.

Set backup dir name to date/time when backup completes (after stop backup). This indicates the time from which recovery is possible. Right now the timestamp is from the beginning of the backup which can be misleading.

Use link and symlink in Perl to make this function pure perl.
Make it work through remote protocol.

This is for cases when the cluster is already down:

1. Don't do start/stop backup
2. Preserve the contents of pg_xlog
3. Don't run if postmaster.pid is present

Current only has local operation.

Defaults for:
command::remote
command::psql
backup::user
backup::path
db::user
db::path
[more?]

Sort of a weird case, but perhaps it should be accounted for.
Just throw an error to let the user know this is not a good idea?

These would perform better with threading.

Leave this until I decide if threads will be replaced or implemented in a new way.

This will help identify issues where archive for different DBs has gotten mixed.

Make sure that archive::path != backup::path if backup is local.

Duplicate WALs can be logged if checksums are turned on (since the filenames do no collide). Do not allow this as a misconfiguration would render the archive repo unusable.

Make sure that nothing unexpected is output to STDOUT during a backup.

Just what it says.

Allow user-defined hook scripts to be run before and after backups.

When archive logs are copied to pg_xlog, be sure to add them to the manifest.
This might be required to make some of the new unit tests work.

This will eliminate some configuration and should increase portability. This should also eliminate IPC::System::Simple.

Add to BackupTest.

Create a link to the latest backup called latest. This makes automating restores off the most recent backup easier.

If an archive log is missing in the middle of an archive stream backrest will return a soft error (1), even though there is probably no chance of that archive log showing up.

If an archive log is missing then check to see if the next one is present - if so return a hard error.

This is tricky because there is a question of how long to wait. With parallel async push it's very possible that the WALs could arrive out of order.

Here's how to do it. BackRest on the database server knows the oldest WAL segment that is currently on the db server and not pushed. If this is reported to the backup server, then it can determine if a hole in the archive stream may be filled, or if it is a permanent condition.

Add a throttling feature to limit the amount of disk i/o and or network bandwidth being used by the backup.
To start, make this a per thread limitation. That simplifies the problem quite a bit and most users who are throttling will probably be single threaded.

Checksums are calculated during the backup process, but the delta is still done during diff/incr backups.
Add a new option checksum_delta (default n) that does the delta using checksums. Of course, if the timestamp or size has changed the checksum does not need to be calculated.

The backup works but this message should probably be suppressed. It seems this only happens when the source file is remote. Need to check the error code when ignore missing is set before throwing an error.

This is handy for backups that have failed. This may be tied to using DBD:Pg to maintain a connection during the backup for safety.

However, it seems that if a backup is started more than once the previous backup should fail. Might need to put in checking to make sure that the archive start position is the same when pg_stop_backup() is called.

The async archiver exits as soon as there are no files left to transfer. A configurable sleep would be good here because it would reduce the number of SSH connections made to the remote.

This feature can create duplicate WAL files - perhaps defer it to another time?

This will help with simple installations where only one stanza is configured.

Even with the thread refactor they are not reliable on all platforms. It's time to move to processes. Performance testing has also show that they are not very efficient.

Before starting:

1. Run four processes from the command line in parallel and make sure this is actually faster and make sure this is faster than running four threads. RESULTS: Processes are not measurably faster than threads, but it's still a good idea to replace them for portability reasons. Threads are not well supported and in fact are officially deprecated.

There will be a few steps to this effort:

1. Separate protocol from remote
2. Create local module
3. Create a ProcessGroup module to manage local helpers
4. Replace ThreadGroup with process group

Fix it!

Waits for a file or directory to exist with configurable retry and timeout. Replaces the wait_for_file function in the Utility module.