Go-ipfs now has a datastore plugin type, would be nice to support them

If I run the conversion to Badger Datastore twice the second time (from badgerds to badgerds) happens even though there is no need for any conversion. If I'm following the code correctly that case corresponds to this strategy case, the directories that should be checked (as mentioned in the TODO) are the path components of the spec file?

It links to https://github.com//ipfs-ds-convert - which won't work for obvious reasons. For what its worth, ipfs-see-all has the same problem. Also, why isn't js-ipfs listed there?

I'm trying to understand (with very little knowledge of the datastore specification) why is it necessary to manually update the repository configuration file in order to convert it to the Badger datastore, I would expect to be able to manage the entire operation from the command line, e.g.,

ipfs init
ipfs add . -r
ipfs-ds-convert convert <badger options>

@magik6k Could you provide me some context to better understand this? And maybe a few pointers that would help me submit a PR to add some command line options that would allow running the conversion tool without adjusting the configuration file (if that is feasible).

Currently, the latest version of ipfs-ds-convert isn't compatible with ipfs 0.6.0:

unsupported fsrepo version: 10

@Stebalien I think the release todo should cover a bump for the ds-convert compatibility as well. :)

I'm running the 13rc1 of ipfs and I got this message as I tried to convert the datastore from flatfs to badgerds:

./ipfs-ds-convert convert
convert 2022/05/16 13:24:18 unsupported fsrepo version: 12

We’re trying to make sure every repo has a description (see ipfs-inactive/docs#55) and this repo is missing one. I don’t have permissions to add it, but I’d suggest something like:

Command-line tool for convert a datastore (e.g. from FlatFS to Badger)

I'm attempting to convert a couple of my nodes from leveldb over to badger. And in each case I've run out of disk space before the conversion is complete.

While I'd expect the conversion to roughly double the size of the .ipfs directory during conversion, I'm seeing the conversion directory get up to several times bigger than the original datastore before the conversion even completes.

On one node I have roughly 400GB in IPFS and the conversion process filled up the filesystem overnight by consuming almost 2TB. On the other I have roughly 9GB, but I killed that conversion after the conversion directory got up to around 40GB; this node is a laptop with an SSD that isn't very large.

Currently, the latest version of ipfs-ds-convert isn't compatible with ipfs 0.5.0-rc2:

convert 2020/04/18 08:15:35 unsupported fsrepo version: 9

Right now the only way to install ipfs-ds-convert is via go get. Since go already has a great support for multiple platforms, it would be great and easy for users to directly wget/curl the pre-compiled binaries instead of first having to install go first and then perform go get... for this.

I read through the code and took some initial review notes. Since there is a lot of code here, i did not go through it as in depth as I would like.

• convert/convert.go line 89, switch should have default case that errors
• saveNewSpec, 'if backup' should probably all get extracted into its own function to make reading things a bit cleaner.
• In general, I'm starting to lean towards oneline single error checks, i.e. if err := foo(); err != nil {. It looks a bit cleaner, would love your thoughts on the pattern
• convert/copy.go:248, TODO still relevant?
• config/validate: Should we actually require that paths be strictly within the ipfs repo? I think this is fine to require for now, but likely want to reconsider moving forward

In general, It looks good. Great job @magik6k :) I will give it a try on my own ipfs repo soon.