different magics, accept/convert attic repos about borg HOT 13 CLOSED

so a few things about this:

1. i agree it's important to avoid blowing up stuff
2. if we're going to change the magic number, i think they should be versioned to cover for possible future changes in the data structures
3. in that spirit, which change actually warranted such a change in the magic numbers? surely there's a change that makes the ondisk format incompatible anymore?

i think this is an important scenario we should deal with properly: we are changing the on-disk format. fine. but what does that entail? do we just do random changes and not look back? i believe we should be really careful when that happen and provide a migration path. migrating from attic to borg isn't much different from migrating between borg 1.x and 2.x, assuming 2.x introduces on-disk changes as well.

so my suggestion would be to not break the on-disk format without a very good reason (i guess that still needs to be defined, but it seems unclear to me right now what the reason for the above change is).

when the on-disk format is changed:

1. it should be clearly documented as a change in internals.html
2. ... but also in a "changelog-style" file (this is currently CHANGES, but this lacks details on how the changes impact older repositories
3. the magic number needs to be incremented
4. a migration layer of some sort needs to be implemented to convert older repository formats

does that make sense?

is it the right place to discuss such issues for the future?

from borg.

@anarcat the magics are not for versioning, but to tell (detect) whether the files are from this software (borg). That's one of the reasons why they say BORG and not ATTIC. See above for others.

There's already other information in the on-disk files to version the API level. Which is btw. another reason not to tell ATTIC in the magics - who says that attic api v3 will be same as borg api v3?

from borg.

@ThomasWaldmann Right okay, that makes sense. Still doesn't invalidate my points i believe. ;)

from borg.

As borg 0.x.x is now targetted to be attic + conservative changes, a one-time conversion from current(!) attic repos seems reasonable and doable. The converter has to be careful not to convert attic repos (from past or from future) that are incompatible.

The different BORG magics must stay as they are, for reasons outlined above.

from borg.

Note: I just started a bounty on BountySource for this ticket. The bounty is about implementing the code needed for doing a one-time conversion from attic to borg repositories (so attic users have a migration path without losing their repository or having to begin from scratch).

from borg.

i may just do this at some point, but i'm in kind of a squeeze until at least october... so first person to grab this gets the bounty yay! :)

from borg.

no one started on this yet?

from borg.

Not me at least (and I don't know of anybody else working on this).

from borg.

so in short, there are a few places that needs to be rewritten:

• key file: s/ATTIC KEY/BORG_KEY/ in get_keys_dir(), that is $ATTIC_KEYS_DIR or $HOME/.(attic|borg)/keys, that is loaded by KeyfileKey.find_key_file() - that finds the keys with the right identifier for the repo, no need to decrypt to convert
• repository segments: s/ATTICSEG/BORG_SEG/ - luckily the segment length didn't change so we can just replace the 8 first bytes of all regular files in $ATTIC_REPO/data/**, the Repository.segment_iterator() can be used here
• hash indexes: s/ATTICIDX/BORG_IDX/ - and this is in a few locations:
  • the files and chunks cache (in $HOME/.cache/attic/<repoid>/), which we could just drop, but if we'd want to convert, we could open it with the Cache.open(), edit in place and then Cache.close() to make sure we have locking right
  • the repository index (in $ATTIC_REPO/index.%d, where %d is the Repository.get_index_transaction_id()), which we should probably update, with a lock, see Repository.open(), which i'm not sure we should use because it may write data on Repository.close()...

i think that about covers it.

from borg.

unless the latter wasn't clear, i'm working on this here. :)

from borg.

let's move this fun stuff to the PR in #231.

from borg.

PR #231 closes this. Thanks to @anarcat we have a migration path now! \o/

from borg.

you're welcome :)

from borg.