myena / consul-backinator Goto Github PK

Consul api does provides this option through TLSConfig.

Hello,

Got another one (this one might be a little more complicated) - I've built the binary using the latest code (although this happens with 1.6.2 release as well). 'dump' seems to only work for KVs and not acls or queries. Consul is on version 0.9.3.

I perform a backup using the following:

# consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 03:42:14 [Success] Backed up 1 keys from / to kvs.bak
2017/11/03 03:42:14 [Success] Backed up 3 ACL tokens from  to acls.bak
2017/11/03 03:42:14 [Success] Backed up 1 query definitions from  to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.

Then dump the KVs (which works as expected)

# consul-backinator dump -file kvs.bak -key [SECRET]
[
  {
    "Key": "consul/backup/lock",
    "CreateIndex": 23,
    "ModifyIndex": 3123,
    "LockIndex": 7,
    "Flags": 0,
    "Value": "",
    "Session": ""
  }
]

Dumping acls complains that consul.bak does not exist (as I specifed a different filename during backup) and -nokv isn't implemented for dump

# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:00:24 [Error] Failed to dump data: open consul.bak: no such file or directory

# consul-backinator dump -nokv -acls acls.bak -key [SECRET]
flag provided but not defined: -nokv
Usage: consul-backinator dump [options]

	Dump the contents of a backup file to stdout.

Options:

	-file         Source filename (default: "consul.bak")
	-key          Passphrase for data encryption and signature validation (default: "password")
	-plain        Dump a reduced set of information
	-acls         Specified file is an ACL token backup file
	-queries      Specified file is a prepared query backup file (consider using plain for query files)

Please see documentation on GitHub for a detailed explanation of all options.
https://github.com/myENA/consul-backinator

If I backup kvs to consul.bak and then try to dump acls.bak the result seems to be dependent upon where I place the flags (placing the -acls flag at the end just causes it to dump consul.bak as KVs, seemingly ignoring that the flag is even there):

# consul-backinator backup -file consul.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 04:04:52 [Success] Backed up 1 keys from / to consul.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.

# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:05:56 [Error] Failed to dump data: gzip: invalid header

# consul-backinator dump -key [SECRET] -acls acls.bak
[
  {
    "Key": "consul/backup/lock",
    "CreateIndex": 23,
    "ModifyIndex": 3564,
    "LockIndex": 1,
    "Flags": 0,
    "Value": "",
    "Session": ""
  }
]

The result is the same when I try dumping queries.

Not sure if I'm doing something fundamentally wrong but any help would be appreciated.

Thank you,
Mike

New non-fatal error code introduced in newer aws-sdk-go packages. We shouldn't fail on this error.

Hi guys

I tried to add consul-backinator to a custom Docker image I'm building.

Here the Dockerfile

# base image
FROM python:3.8.8-slim

# install additional packages
RUN apt-get update
RUN apt-get install -y curl ca-certificates tzdata unzip

# set current dir
WORKDIR /usr/bin

# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz
RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz
RUN chmod +x ./consul-backinator

# ...

I have no idea why, but running the command tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz gives me this error:

...
 => [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz                                                                                                                                                                                                                               0.5s
 => ERROR [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz                                                                                                                                                                                                                                                                                            0.3s
------
 > [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz:
#14 0.236
#14 0.236 gzip: stdin: not in gzip format
#14 0.236 tar: Child returned status 1
#14 0.236 tar: Error is not recoverable: exiting now
------
executor failed running [/bin/sh -c tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz]: exit code: 2

I tried with zip archive

# ...

# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
RUN unzip consul-backinator-1.6.6-amd64-linux.zip
RUN chmod +x ./consul-backinator

# ...

but more or less the message is the same (command unzip consul-backinator-1.6.6-amd64-linux.zip):

...
 => [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
 => ERROR [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip                                                                                                                                                                                                                                                                                                  0.3s
------
 > [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip:
#15 0.225   End-of-central-directory signature not found.  Either this file is not
#15 0.225   a zipfile, or it constitutes one disk of a multi-part archive.  In the
#15 0.225   latter case the central directory and zipfile comment will be found on
#15 0.225   the last disk(s) of this archive.
#15 0.225 unzip:  cannot find zipfile directory in one of consul-backinator-1.6.6-amd64-linux.zip or
#15 0.225         consul-backinator-1.6.6-amd64-linux.zip.zip, and cannot find consul-backinator-1.6.6-amd64-linux.zip.ZIP, period.
#15 0.225 Archive:  consul-backinator-1.6.6-amd64-linux.zip
------
executor failed running [/bin/sh -c unzip consul-backinator-1.6.6-amd64-linux.zip]: exit code: 9

I tried to download and extract packages locally (macos) and the errors are the same!

It seems that archives are corrupted. Any idea why?
How can I solve this?

Hi! This is relevant to #3

By default, transform matches oldkey anywhere in the consul kv.

It would be great to have another option or similar (e.g. -tranform-mode=absolute|regex) to perform an absolute left to right match on the kv.

e.g.

kv:
x/y/z/foo/x/y -> ignore
foo/x/y -> match

command:
-transform foo/x/y,foo/x/z -transform-mode=absolute

kv:
x/y/z/foo/x/y -> match
foo/x/y -> match

command:
-transform foo/x/y,foo/x/z -transform-mode=regex

did docker run myena/consul-backinator backup -addr=10.0.1.50:8500 -file=s3://"access key id":"secret access key"@s3-bucket name/consul_backup?region=us-west-2

i am getting failed to backup data : invalid bucket name : the specified bucket is not valid,

but my nucket name is correct, can someone help ?

im using consul as a backend to store vault info.

all information in consul is stores under the vault/ tree in encrypted format.

1. We have keys with acls on them, Does your tool need the management acl in order to traverse and backup the key/values?
2. Will your tool support taking backup of encrypted values under the vault/ tree?
   Will the values be backed up in encrypted format? If yes, once the values are restored , will it still work with the original vault?

regards
meher

I am setting up backups from a Jenkins instance in EC2 that has an associated role allowing it to write to the consul-backups S3 bucket. I would like to have consul-backinator use the role rather than asking for credentials.

Since we have docker integration now, please consider integration with https://docs.docker.com/docker-hub/builds/ to make it helpful for people to use this utility.

Hi,
Is there a way that a frequency/interval option can be added? I am looking to run this in Docker on 10 min intervals.

Hi,

Thanks for the tool.
I've a feature request for S3 backup. Currently the S3 backup works by specifying AWS access/secret keys. Is it possible to add a feature that uses IAM instance profile where possible?

I can see data in my consul through http api but this tool returns with 0 keys.
How do I debug the issue?

// 20161007120442
// http://171.10.50.4:8500/v1/kv/?keys&token=access_token
[
  "test",
  "test1",
  "test2"
]

consul-backinator backup -file consul.bak -token access_token -addr 171.10.50.4:8500
2016/10/07 12:07:14 [Success] Backed up 0 keys from 171.10.50.4:8500/ to consul.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.

Error trace .

{"line":"2017-08-31T17:06:19Z [WARN] Error with docker; stopping container module="TaskEngine" task="ConsulBackup:3 arn:aws:ecs:us-west-2:776609208984:task/a03e35fe-2e1e-4a73-922a-e98f63a82b6f, Status: (CREATED-\u003eRUNNING) Containers: [ConsulBackup (RUNNING-\u003eRUNNING),]" container="ConsulBackup(myena/consul-backinator:latest) (RUNNING-\u003eRUNNING)" err="API error (404): oci runtime error: container_linux.go:247: starting container process caused "exec: \"backup -addr=10.0.1.50:8500 -file=s3://bakup-consul/consul_backup/first.bak?region=us-west-2\": stat backup -addr=10.0.1.50:8500 -file=s3://bakup-consul/consul_backup/first.bak?region=us-west-2: no such file or directory"","source":"stdout","tag":"ecs-agent/236df33fb65d"}

Can anyone please help me to resolve this error.

docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: "/usr/local/bin/consul-backinator backup -file=consul.bak": stat /usr/local/bin/consul-backinator backup -file=consul.bak: no such file or directory"

Hello! I know as per help menu, option prefix is intended only for backup operation. But, when used with restore, the result is misleading.

$ ./consul-backinator-master  --token token -backup -dc localhost -file foo.x.y -prefix foo/x/y
2016/05/09 14:39:31 [Success] Backed up 1 keys from foo/x/y to foo.x.y
Keep your backup (foo.x.y) and signature (foo.x.y.sig) files in a safe place.
You will need both to restore your data.

$ ./consul-backinator-master  --token token -restore -dc localhost -file foo.x.y -prefix foo/x/z
2016/05/09 14:40:04 [Success] Restored 1 keys from foo.x.y to foo/x/z

$ ./consul-backinator-master  --token token -backup -dc localhost -file foo.x.z -prefix foo/x/z
2016/05/09 14:43:31 [Success] Backed up 0 keys from foo/x/z to foo.x.z
Keep your backup (foo.x.z) and signature (foo.x.z.sig) files in a safe place.
You will need both to restore your data.

As shown above, restore result indicated restore of 1 Key to foo/x/z. But, nothing was restored. As you can see i tried to backup foo/x/z and there is nothing to backup. I can confirm there is no key foo/x/z.

I think prefix option should be ignored with a message or should return an error when used with restore.

Hello, I am trying to get this to work using Minio as the S3 backend but when I provide all the connection information I am getting an error. I have other things using this minio cluster which is working fine so it makes me think that there is either an issue with the s3 library or my connection string.

2018/07/03 18:21:43 [Error] Failed to backup key data: RequestError: send request failed
caused by: Put http://consul-backups.minio.endpoint.com/: read tcp 10.1.0.147:39184->10.160.7.199:80: read: connection reset by peer

Here is my s3 file connection string.

-file="s3://key:token@consul-backups/consul_kvs.bak?region=default&endpoint=http://minio.endpoint.com&secure=false"

Hello,

I am trying to pass an array of consul server but it gets failed, I think "-add" flag doesn't support array list.

Since I requested the docker integration, would you like some use case docs to link to?

Hello! This tool has saved me a lot of overhead, thank you for sharing this amazing tool.

I am having an issue with transform.

Currently transform matches the pattern anywhere in the path, e.g. --transform foo/x/y,foo/x/z will transform below mentioned key path:

a/b/c/foo/x/y/a/b/c -> a/b/c/foo/x/z/a/b/c
foo/x/y -> foo/x/z

But, i only intend to transform the foo/x/y to foo/x/z (which matches absolute path from left to right).

I know i can take the backup with prefix foo/x/y and then transform with the new key path as prefix matches absolute path from left to right. If this is the recommended way, i can submit a PR with this info for the transform option.

Is there a way to backup all the acls in a consul cluster?

to give users an extra sense of security that they have a real, untampered-with download.
Example: See the consul_0.6.4_SHA256SUMS file at https://releases.hashicorp.com/consul/0.6.4/
Would be great if you sign the file with a GPG key -- e.g.: consul_0.6.4_SHA256SUMS.sig

hello, i am trying to backup ACLS without key value and i got the permission denied error, could you please help me with this

docker run myena/consul-backinator backup -nokv -acls -token= -addr=10.18.1.82:8500 -file=s3://key id:secret@consul-bakup/backup-consul/$(date +%F_%R)_consul.bak?region=us-west-2

i run the same command without -nokv and -acls and it backed up keys but i am not able to backup ACL's, am i doing something wrong?

Hi,

Thanks for your job. I just tried it from a standalone container and it work perfectly.

I would like to integrate it to my docker compose. As cron jobs are not yet implemented in Swarm, I would like to know if there is a best practice for this?

Should I:

• Wrap your image with a crontab based container?
• Wrap your image with a scheduler script ?
• PR the project by implementing a daemon mode with a cron lib?
• stop trying to backup stuff and start praying god for keeping my data alive
• other ideas?

Many thanks by advance for your time.

Hello,

Consul 1.4 introduced a new ACL system where a token's permissions are tied to a policy (or multiple policies), rather than being baked into the token itself. I just recently upgraded my Consul cluster from version 1.3.1 to 1.4.5. After upgrade, all pre-existing non-management tokens were in "legacy" mode. At this point, consul-backinator was backing up the ACLs just fine.

However, after migrating the legacy tokens to the new format, consul-backinator now backs up only the management tokens and ignores anything with a policy tied to it. I confirmed this in two ways:

1. The consul-backinator log indicates that only a handful of ACL tokens were backed up, even though I have about 50 tokens in total

   2022/03/07 15:05:02 [Success] Backed up 10 ACL tokens from consul.service.example.com:8501 to /path/to/backup/my-acls

2. I restored the above backup to a fresh test cluster and confirmed that only 10 ACL tokens exist (all of which are management tokens)

Note that KV backups appear to be working as they did previously.

Can you please implement support for backing up ACLs/Policies in Consul 1.4+? If this already exists, please let me know how to enable it. I can provide more details if needed.

Thank you!

i am trying to restore keys to consul using

docker run myena/consul-backinator restore -addr=:8500 -file=s3://access key id:secret id@consul-bakup/stage-consul/2017-12-13_09:00_consul.bak?region=us-west-2

and i am getting error: rpc error permission denied

We are getting this error with consul:1.2.1 docker consul & backinator 1.6.5

panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
panic: standard_init_linux.go:178: exec user process caused "no such file or directory"

goroutine 1 [running, locked to thread]:
github.com/urfave/cli.HandleAction.func1(0xc4200a17a0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x23f
panic(0x6f0cc0, 0xc42011d5e0)
/usr/lib/golang/src/runtime/panic.go:489 +0x2cf
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc4200a1208, 0xc42000e0d0, 0xc4200a12a8)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0xc1
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc4200509b0, 0xaa88e0, 0xc42011d5e0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc42007c780, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main_unix.go:26 +0x66
reflect.Value.call(0x6d9be0, 0x750350, 0x13, 0x73b942, 0x4, 0xc4200a1760, 0x1, 0x1, 0xc4200a16f0, 0x731420, ...)
/usr/lib/golang/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x6d9be0, 0x750350, 0x13, 0xc4200a1760, 0x1, 0x1, 0x665b36, 0x73bace, 0x4)
/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6d9be0, 0x750350, 0xc42007c780, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x18f
github.com/urfave/cli.Command.Run(0x73baee, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d391, 0x51, 0x0, ...)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xac8
github.com/urfave/cli.(*App).Run(0xc4200aa000, 0xc42000c140, 0x2, 0x2, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x5d6
main.main()
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main.go:137 +0xbd2

Hi Team,

I have tried running consul-backinator to take the backup on S3 via k8s service account having enough IAM roles to perform consul backup directly to S3 bucket but it doesn't work.

S3 Policy has below:
"s3:GetObject",
"s3:DeleteObjectVersion",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject"

However when i run it to take backup locally it works and just to check if there is any issue with the role i tried installing aws cli and did successful upload via aws-cli.

On another note acl backup always fail complaining
[Error] Failed to backup ACL tokens: Unexpected response code: 403 (Permission denied)

It would be best if you can even mention what kind of S3 permissions and consul policies are required to take the acl backup.

Downloaded the container and run.
$ docker run myena/consul-backinator backup -addr consulhostname:8500 -dc consul-dc-01 -token xxxxx-xxxx-xxxx-xxxx -scheme http -file s3://mybucket-backup/consul

[Error] Failed to backup key data: Unexpected response code: 500

Hi guys !

What about a parameter uploading to/restoring from AWS S3, instead of -file ?

HI

Cuurently using 1.0 version

consul-backinator backup file=/tmp/bkupfile prefix=/
2017/02/16 22:09:39 [Success] Backed up 0 keys from / to consul.bak
Keep your backup (consul.bak) and signature (consul.bak.sig) files in a safe place.
You will need both to restore your data.

Trying to use 1.4 version

./consul-backinator-1.4 backup file=/tmp/bkupfile prefix=/
2017/02/16 22:04:43 [Error] Failed to backup key data: No keys found

Same command fails with 1.4 , what am i doing wrong here

Hi there,

I'm running into a TLS verification error using consul-backinator. When I run the following (with and without -tls-skip-verify):

# consul-backinator backup -addr=consul.dev.blahblah:8550 -scheme https -key=vault -ca-cert=/etc/consul/tls/ca.pem
2016/12/05 16:26:29 [Error] Failed to backup key data: Get https://consul.dev.blahblah:8550/v1/kv/?consistent=&recurse=: remote error: tls: bad certificate

What's interesting is that when I run:

# openssl verify -CAfile /etc/consul/tls/ca.pem /etc/consul/tls/cert.pem
/etc/consul/tls/cert.pem: OK

Additionally, when I run openssl s_client I do get "Verify return code: 0 (ok)." I feel like the -ca-path variable is being ignored, because if I leave the -CAfile option off openssl, I do get verification errors. Is there some way to get more verbose output out of consul-backinator?

Hello,

I'm able to successfully backup kvs/acls/queries but when I try to restore it seems to only restore the kvs and acls (ignoring the queries).

consul-backinator v1.6.2
consul v0.9.3

# consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:26:36 [Success] Backed up 1 keys from / to kvs.bak
2017/11/02 21:26:36 [Success] Backed up 3 ACL tokens from  to acls.bak
2017/11/02 21:26:36 [Success] Backed up 1 query definitions from  to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
# consul-backinator restore -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:26:59 [Success] Restored 1 keys from kvs.bak to /
2017/11/02 21:26:59 [Success] Restored 3 ACL tokens from acls.bak to

And if I try specifying only the queries to backup and restore, I'm able to successfully back them up but get an error regarding my command flags when trying to restore.

# consul-backinator backup -nokv -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:33:41 [Success] Backed up 1 query definitions from  to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
# consul-backinator restore -nokv -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:33:54 [Error] Passing 'nokv' and an empty 'acls' file doesn't make any sense.  You should specify an 'acls' file when using the 'nokv' option.

Am I missing something in my command syntax? Or has the restore queries functionality not been fully implemented yet?

Thank you,
Mike

I have a policy which allows the PutObject permission to the specific bucket where I store my backups. I am running version 1.6.5 and observe in my cloudtrail logs that backinator is trying to create the bucket; this fails for hopefully obvious reasons.

At this point I'm out of things to troubleshoot as by all rights this should work, and some months ago it did (I'm sure something changed since then, but right now I haven't bisected our change lists to find what it was).

docker run myena/consul-backinator backup -addr=10.0.1.50:8500 -file=s3://access key id:secretaccesskey@anilmytest/Backup?region=us-west-2
[Error] Failed to backup key data: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
status code: 409, request id: 3F231DD26FB7B637, host id: sQDegGUjvnzlT5YH7lcwZzLHmZPd1gNUBLKrB65VPKgiifX1rImHJPexGjwa/kT41/Xi6RTLlv8=

Hi guys,

What about a copy-before-delete feature that would move KV under a restore.bak/ prefix, prior to restore data ?!

I noticed that the -dump option provides values as base64 encoded.

Would be possible/preferable to display them as straight strings in the dump output?