flyingcircusio / backy-extract Goto Github PK

After doing stuff on a FUSE mount for a while, backy-extract consumes all available memory and gets eventually killed by the OOM killer. A sure way to trigger this bug is to create a loop device and running xfs_repair on that.

This happens because backy-extract maintains a page cache in RAM which is filled with every page accessed at least twice in addition to pages dirtied by write operations.

Fix: Implement LRU cache eviction. Drop cached pages after consuming more than a specified fraction of the host's RAM. Keep dirty pages.

backy-extract fails when invoked from the backup dir as pwd,

# backy-extract JBs7pis8kJgtZiT2WPuD4W >/dev/null
Error:  Cannot lock backup dir `'
Detail: Resource temporarily unavailable (os error 11)

Currerntly, only a fixed amount of chunks (32) are kept in the cache. Use a better data structure (ChunkBucket?) to size the cache dynamically.

If the restore target is a block device, b-e should check that it has the right size and fail early.

We fixed the lock management to use shared locks. However, backy-extract is not ready to encounter incomplete revisions:

root@patty ~ # restore-single-files services03
* Starting FUSE
Error: Invalid data in .rev file '/srv/backy/services03/WAzpkLtRMghXJVzyfY8fuC.rev'

Caused by:
    stats: missing field `duration` at line 4 column 16
* Registering loop device
losetup: /mnt/backy-fuse/services03/last: failed to set up loop device: No such file or directory

Incomplete revision files should be silently ignored (that's how backy deals with them).

backy-extract should proceed even without being able to acquire a purge lock if invoked with --force.