aaronwebster / attic Goto Github PK

Is it safe to interrupt (siginterrupt) a backup?

This project seem abandoned as there is no communication from the owner. I would like to volunteer to maintain the project. Please submit pull requests or issues to either AaronWebster/attic or jmfernandez/attic (most recent contributor).

(html2pdf) tejas@g3:~$ pip install Attic
Collecting Attic
Downloading https://files.pythonhosted.org/packages/b5/a1/84a448f4c375a9c18d24626e212e289140083fd2ce3c6b4665cd139d92ba/Attic-0.16.tar.gz (232kB)
|████████████████████████████████| 235kB 7.3MB/s
Collecting msgpack-python
Downloading https://files.pythonhosted.org/packages/8a/20/6eca772d1a5830336f84aca1d8198e5a3f4715cd1c7fc36d3cc7f7185091/msgpack-python-0.5.6.tar.gz (138kB)
|████████████████████████████████| 143kB 11.0MB/s
Building wheels for collected packages: Attic, msgpack-python
Building wheel for Attic (setup.py) ... error
ERROR: Command errored out with exit status 1:
command: /home/tejas/anaconda3/envs/html2pdf/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"'; file='"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-ubjb6iwh --python-tag cp37
cwd: /tmp/pip-install-rno7f0t1/Attic/
Complete output (50 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/attic
copying attic/repository.py -> build/lib.linux-x86_64-3.7/attic
copying attic/remote.py -> build/lib.linux-x86_64-3.7/attic
copying attic/platform.py -> build/lib.linux-x86_64-3.7/attic
copying attic/_version.py -> build/lib.linux-x86_64-3.7/attic
copying attic/archive.py -> build/lib.linux-x86_64-3.7/attic
copying attic/init.py -> build/lib.linux-x86_64-3.7/attic
copying attic/fuse.py -> build/lib.linux-x86_64-3.7/attic
copying attic/key.py -> build/lib.linux-x86_64-3.7/attic
copying attic/cache.py -> build/lib.linux-x86_64-3.7/attic
copying attic/xattr.py -> build/lib.linux-x86_64-3.7/attic
copying attic/lrucache.py -> build/lib.linux-x86_64-3.7/attic
copying attic/archiver.py -> build/lib.linux-x86_64-3.7/attic
copying attic/helpers.py -> build/lib.linux-x86_64-3.7/attic
creating build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/repository.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/platform.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/archive.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/hashindex.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/init.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/mock.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/chunker.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/key.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/xattr.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/lrucache.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/archiver.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/run.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/helpers.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/crypto.py -> build/lib.linux-x86_64-3.7/attic/testsuite
running build_ext
skipping 'attic/crypto.c' Cython extension (up-to-date)
skipping 'attic/chunker.c' Cython extension (up-to-date)
skipping 'attic/hashindex.c' Cython extension (up-to-date)
skipping 'attic/platform_linux.c' Cython extension (up-to-date)
building 'attic.crypto' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/attic
/home/tejas/anaconda3/envs/html2pdf/bin/x86_64-conda_cos6-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/tejas/anaconda3/envs/html2pdf/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/tejas/anaconda3/envs/html2pdf/include -fPIC -I/usr/include -I/home/tejas/anaconda3/envs/html2pdf/include/python3.7m -c attic/crypto.c -o build/temp.linux-x86_64-3.7/attic/crypto.o
In file included from /home/tejas/anaconda3/envs/html2pdf/include/python3.7m/Python.h:11:0,
from attic/crypto.c:16:
/usr/include/limits.h:26:10: fatal error: bits/libc-header-start.h: No such file or directory
#include <bits/libc-header-start.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/home/tejas/anaconda3/envs/html2pdf/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1

ERROR: Failed building wheel for Attic
Running setup.py clean for Attic
Building wheel for msgpack-python (setup.py) ... done
Created wheel for msgpack-python: filename=msgpack_python-0.5.6-cp37-cp37m-linux_x86_64.whl size=99115 sha256=87cc81f0e042bd87aa89363aa8cd077291839730bd258bb85f616152e1cb334f
Stored in directory: /home/tejas/.cache/pip/wheels/d5/de/86/7fa56fda12511be47ea0808f3502bc879df4e63ab168ec0406
Successfully built msgpack-python
Failed to build Attic
Installing collected packages: msgpack-python, Attic
Running setup.py install for Attic ... error
ERROR: Command errored out with exit status 1:
command: /home/tejas/anaconda3/envs/html2pdf/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"'; file='"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-4258991t/install-record.txt --single-version-externally-managed --compile
cwd: /tmp/pip-install-rno7f0t1/Attic/
Complete output (50 lines):
running install
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.7
creating build/lib.linux-x86_64-3.7/attic
copying attic/repository.py -> build/lib.linux-x86_64-3.7/attic
copying attic/remote.py -> build/lib.linux-x86_64-3.7/attic
copying attic/platform.py -> build/lib.linux-x86_64-3.7/attic
copying attic/_version.py -> build/lib.linux-x86_64-3.7/attic
copying attic/archive.py -> build/lib.linux-x86_64-3.7/attic
copying attic/init.py -> build/lib.linux-x86_64-3.7/attic
copying attic/fuse.py -> build/lib.linux-x86_64-3.7/attic
copying attic/key.py -> build/lib.linux-x86_64-3.7/attic
copying attic/cache.py -> build/lib.linux-x86_64-3.7/attic
copying attic/xattr.py -> build/lib.linux-x86_64-3.7/attic
copying attic/lrucache.py -> build/lib.linux-x86_64-3.7/attic
copying attic/archiver.py -> build/lib.linux-x86_64-3.7/attic
copying attic/helpers.py -> build/lib.linux-x86_64-3.7/attic
creating build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/repository.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/platform.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/archive.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/hashindex.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/init.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/mock.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/chunker.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/key.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/xattr.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/lrucache.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/archiver.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/run.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/helpers.py -> build/lib.linux-x86_64-3.7/attic/testsuite
copying attic/testsuite/crypto.py -> build/lib.linux-x86_64-3.7/attic/testsuite
running build_ext
skipping 'attic/crypto.c' Cython extension (up-to-date)
skipping 'attic/chunker.c' Cython extension (up-to-date)
skipping 'attic/hashindex.c' Cython extension (up-to-date)
skipping 'attic/platform_linux.c' Cython extension (up-to-date)
building 'attic.crypto' extension
creating build/temp.linux-x86_64-3.7
creating build/temp.linux-x86_64-3.7/attic
/home/tejas/anaconda3/envs/html2pdf/bin/x86_64-conda_cos6-linux-gnu-cc -Wno-unused-result -Wsign-compare -DNDEBUG -fwrapv -O2 -Wall -Wstrict-prototypes -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -pipe -march=nocona -mtune=haswell -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O2 -ffunction-sections -pipe -isystem /home/tejas/anaconda3/envs/html2pdf/include -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem /home/tejas/anaconda3/envs/html2pdf/include -fPIC -I/usr/include -I/home/tejas/anaconda3/envs/html2pdf/include/python3.7m -c attic/crypto.c -o build/temp.linux-x86_64-3.7/attic/crypto.o
In file included from /home/tejas/anaconda3/envs/html2pdf/include/python3.7m/Python.h:11:0,
from attic/crypto.c:16:
/usr/include/limits.h:26:10: fatal error: bits/libc-header-start.h: No such file or directory
#include <bits/libc-header-start.h>
^~~~~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
error: command '/home/tejas/anaconda3/envs/html2pdf/bin/x86_64-conda_cos6-linux-gnu-cc' failed with exit status 1
----------------------------------------
ERROR: Command errored out with exit status 1: /home/tejas/anaconda3/envs/html2pdf/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"'; file='"'"'/tmp/pip-install-rno7f0t1/Attic/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' install --record /tmp/pip-record-4258991t/install-record.txt --single-version-externally-managed --compile Check the logs for full command output.

When I try to build attic on Debian/Testing box with python 3.5 (pip 9.0.1) and openssl 1.1.0c-2 I get the following error during build

attic/crypto.c:448:18: error: field 'ctx' has incomplete type
EVP_CIPHER_CTX ctx;

Name: attic-backup.org
Registry Domain ID: D171289523-LROR
Domain Status: pendingDelete, serverHold, redemptionPeriod
Registry Expiration: 2021-03-04 11:51:14 UTC

[nbecker@nbecker2 ~]$ attic mount -v nbecker@nbecker7:repository.attic::nbecker2-2017-01-03 mnt
[sits for about 5 seconds...]
Mounting filesystem

No error, but nothing happened
[nbecker@nbecker2 ~]$ ls mnt
[nbecker@nbecker2 ~]$ mount
...[nothing show that attic mounted anything]

This is attic-0.16 on fedora-25

There isn't any way to avoid the error when trying to initialized an already initialized repository

Use case: I use a script with attic to backup all of my mysql db's, and when a new db is created I must initialize the repository by the backup script. So by now the only way is to init the repository every time I'm going to create an archive in it. It works, because the init command doesn't make anything on an existing repository, but it's far from elegant

Also should work with at least OpenSSL 1.0.2

Just built on fedora. You need
libacl-devel
package.

When there is a inaccessible file (ex. "chmod 0 file"), attic does not stores the file. However, if there is a hard link to this file, it is stored as a hard link to an unexisting file. Example:

$ mkdir work
$ echo hello > work/forbidden.txt
$ chmod 0 work/forbidden.txt 
$ ln work/forbidden.txt work/link.txt
$ attic init repo
Initializing repository at "repo"
Encryption NOT enabled.
Use the "--encryption=passphrase|keyfile" to enable encryption.
Initializing cache...
$ attic create repo::1 work
attic: work/forbidden.txt: [Errno 13] Permission denied: 'work/forbidden.txt'
attic: Exiting with failure status due to previous errors
$ attic list repo::1
drwxr-xr-x cstrauss cstrauss        0 Jan 21 16:06 work
h--------- cstrauss cstrauss        0 Jan 21 16:06 work/link.txt link to work/forbidden.txt
$ mkdir mnt
$ attic mount repo::1 mnt
Traceback (most recent call last):
  File "/usr/bin/attic", line 3, in <module>
    main()
  File "/usr/lib/python3.6/site-packages/attic/archiver.py", line 730, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/lib/python3.6/site-packages/attic/archiver.py", line 720, in run
    return args.func(args)
  File "/usr/lib/python3.6/site-packages/attic/archiver.py", line 262, in do_mount
    operations = AtticOperations(key, repository, manifest, archive)
  File "/usr/lib/python3.6/site-packages/attic/fuse.py", line 48, in __init__
    self.process_archive(archive)
  File "/usr/lib/python3.6/site-packages/attic/fuse.py", line 82, in process_archive
    inode = self._find_inode(item[b'source'], prefix)
  File "/usr/lib/python3.6/site-packages/attic/fuse.py", line 127, in _find_inode
    inode = self.contents[inode][segment]
KeyError: b'forbidden.txt'

$ mkdir extract
$ cd extract/
$ attic extract ../repo::1
attic: work/link.txt: [Errno 2] No such file or directory: '/tmp/attic2/extract/work/forbidden.txt' -> '/tmp/attic2/extract/work/link.txt'
attic: Exiting with failure status due to previous errors

I'm using Attic 0.16.

Hi!
I'm trying to mount an encrypted repository for which I don't have the original key file. I have however the contents (text) of that file. I therefore placed that text in a new file in ~/.attic/keys, making sure there are no extra spaces or newlines. I named this file arbitrarily...

This does not work:

~$ attic mount ~/mnt/home.attic ~/Downloads/mnt
Traceback (most recent call last):
  File "/usr/bin/attic", line 3, in <module>
    main()
  File "/usr/lib/python3/dist-packages/attic/archiver.py", line 715, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/lib/python3/dist-packages/attic/archiver.py", line 705, in run
    return args.func(args)
  File "/usr/lib/python3/dist-packages/attic/archiver.py", line 245, in do_mount
    manifest, key = Manifest.load(repository)
  File "/usr/lib/python3/dist-packages/attic/helpers.py", line 98, in load
    key = key_factory(repository, cdata)
  File "/usr/lib/python3/dist-packages/attic/key.py", line 34, in key_factory
    return KeyfileKey.detect(repository, manifest_data)
  File "/usr/lib/python3/dist-packages/attic/key.py", line 201, in detect
    path = cls.find_key_file(repository)
  File "/usr/lib/python3/dist-packages/attic/key.py", line 220, in find_key_file
    raise Exception('Key file for repository with ID %s not found' % id)
Exception: Key file for repository with ID 7a578ef0d4e9dd7a7e4cc9e831e626b304ea2efd9ee7beca064c67c060c7b430 not found

Does the key file need to be named specifically?
Thanks in advance!

Has anyone tried to run in windows?
Seeing on python any reasons this wouldn't work?
Although Long term I'm aiming to move away from windows entirely my work has tried me to it so still need back up and this sounds awesome.

I cannot mount any repository
I used
attic mount -v -f /path/to/backup.attic::2016-04-16_14:39 backup-mount
And I'm getting this result:

Mounting filesystem
Traceback (most recent call last):
  File "/usr/bin/attic", line 3, in <module>
    main()
  File "/usr/lib/python3.5/site-packages/attic/archiver.py", line 730, in main
    exit_code = archiver.run(sys.argv[1:])
  File "/usr/lib/python3.5/site-packages/attic/archiver.py", line 720, in run
    return args.func(args)
  File "/usr/lib/python3.5/site-packages/attic/archiver.py", line 265, in do_mount
    operations.mount(args.mountpoint, args.options, args.foreground)
  File "/usr/lib/python3.5/site-packages/attic/fuse.py", line 230, in mount
    llfuse.main(single=True)
  File "src/fuse_api.pxi", line 273, in llfuse.main (src/llfuse.c:35175)
TypeError: main() got an unexpected keyword argument 'single'

Using Arch Linux with Attic 0.16

Our backup scheme is such that individual machines run attic to back up their files to a common NFS repo. I then want to back up the entire collection of backups offsite using tarsnap. Unfortunately, occasionally the attic backup kicks off after a tarnsnap backup has started. Often this causes tarsnap to error.

What I would like is a feature where if attic detects a certain file is will wait to begin it's back up until that file is gone. There would need to be a timeout on this behavior obviously. This would let be get reliable offsite backups while maintaining hourly backups via attic.

As a workaround I can wrap attic to create the behavior I want.

Hello! I have a specific need: I need to feed ZFS snapshots to attic (both full and incremental ZFS snapshots) from a replication stream, and have attic figure out what to store based on the content of the snapshots. Later on, I should be able to call upon attic to spit back those snapshots to me, so that I may apply them on the restore side.

The structure of a replication stream is explained here: http://open-zfs.org/wiki/Documentation/ZfsSend. For the purpose of understanding how snapshots work, it's useful to think of a snapshot as a series or a log of data modification orders (for the receiving side) to write a specific set of objects (the case of the full replication stream), and to partially rewrite those objects if they were present before (the case of the incremental replication stream). This log of orders can be naturally mergeable on the receiving side just by applying them in the right order.

I considered the naive implementation of just feeding the snapshots to attic and storing that, but doing so would eliminate my ability to do something like attic prune. My goal is to let attic know what has changed so that attic can reconstruct either the original snapshots or keep track of the data, not simply store the big files illegibly, such that I can recover space by deleting old backups without having to redo the base full backup. Given the state that attic keeps on what's been written previously, it should be possible for attic to use its diffing algorithm to figure out what later orders modify previously-stored orders in what way.

I think it should be possible, and I wonder what you think. The reason I'm contemplating this is quite simple: it's much faster to interpret a ZFS incremental send than it is to diff the modified files, especially when the modified files are gigabytes in size. My math says we're talking about 100X performance improvements for incremental backups on large data sizes.

Funding may be available depending on how long the project may run.

Dear Jonas,
Could you be so kind to generate .exe for the rest of us who use Windows w/o Python?
I’m told that py2exe might help you to achieve that.

adds retirement warning

See there: borgbackup/borg#2099

Not sure if the bug also happens with attic, but the suspicious code is also in attic:

https://github.com/jborg/attic/blob/0.16/attic/repository.py#L467

@jborg can you remember why you used "ab" (and not "wb") there?

At first glance, it seems to make no sense to append to an existing file, esp. if the file is a valid segment file that has already a MAGIC at the beginning and some entries in it, this would just add another MAGIC in the middle of the file (but that should not be there).

So, was that just a typo or is there some reason behind it?

On a pristine Ubuntu system, you also need libacl1-dev in order
for the installation (via pip install) to succeed.

A future commit should also update the fact that you need some
build tools (compiler, etc) setup, which can be installed on Ubuntu
via the build-essential pkg.

Hi,

Is it safe to move attic repos to another volume? We've started using attic long time ago and now as the size of the repos increases I'd like them moved to bigger volume, just wonder if that's safe to do?