myena / consul-backinator Goto Github PK
View Code? Open in Web Editor NEWCommand line Consul backup and restore utility supporting KVs, ACLs and Queries
License: Mozilla Public License 2.0
Command line Consul backup and restore utility supporting KVs, ACLs and Queries
License: Mozilla Public License 2.0
I can see data in my consul through http api but this tool returns with 0 keys.
How do I debug the issue?
// 20161007120442
// http://171.10.50.4:8500/v1/kv/?keys&token=access_token
[
"test",
"test1",
"test2"
]
consul-backinator backup -file consul.bak -token access_token -addr 171.10.50.4:8500
2016/10/07 12:07:14 [Success] Backed up 0 keys from 171.10.50.4:8500/ to consul.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
Hi guys
I tried to add consul-backinator to a custom Docker image I'm building.
Here the Dockerfile
# base image
FROM python:3.8.8-slim
# install additional packages
RUN apt-get update
RUN apt-get install -y curl ca-certificates tzdata unzip
# set current dir
WORKDIR /usr/bin
# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz
RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz
RUN chmod +x ./consul-backinator
# ...
I have no idea why, but running the command tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz
gives me this error:
...
=> [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.tar.gz 0.5s
=> ERROR [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz 0.3s
------
> [11/17] RUN tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz:
#14 0.236
#14 0.236 gzip: stdin: not in gzip format
#14 0.236 tar: Child returned status 1
#14 0.236 tar: Error is not recoverable: exiting now
------
executor failed running [/bin/sh -c tar -zxf consul-backinator-1.6.6-amd64-linux.tar.gz]: exit code: 2
I tried with zip archive
# ...
# install consul-backinator
RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
RUN unzip consul-backinator-1.6.6-amd64-linux.zip
RUN chmod +x ./consul-backinator
# ...
but more or less the message is the same (command unzip consul-backinator-1.6.6-amd64-linux.zip
):
...
=> [10/17] RUN curl -O https://github.com/myENA/consul-backinator/releases/download/v1.6.6/consul-backinator-1.6.6-amd64-linux.zip
=> ERROR [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip 0.3s
------
> [11/17] RUN unzip consul-backinator-1.6.6-amd64-linux.zip:
#15 0.225 End-of-central-directory signature not found. Either this file is not
#15 0.225 a zipfile, or it constitutes one disk of a multi-part archive. In the
#15 0.225 latter case the central directory and zipfile comment will be found on
#15 0.225 the last disk(s) of this archive.
#15 0.225 unzip: cannot find zipfile directory in one of consul-backinator-1.6.6-amd64-linux.zip or
#15 0.225 consul-backinator-1.6.6-amd64-linux.zip.zip, and cannot find consul-backinator-1.6.6-amd64-linux.zip.ZIP, period.
#15 0.225 Archive: consul-backinator-1.6.6-amd64-linux.zip
------
executor failed running [/bin/sh -c unzip consul-backinator-1.6.6-amd64-linux.zip]: exit code: 9
I tried to download and extract packages locally (macos) and the errors are the same!
It seems that archives are corrupted. Any idea why?
How can I solve this?
Hello, I am trying to get this to work using Minio as the S3 backend but when I provide all the connection information I am getting an error. I have other things using this minio cluster which is working fine so it makes me think that there is either an issue with the s3 library or my connection string.
2018/07/03 18:21:43 [Error] Failed to backup key data: RequestError: send request failed
caused by: Put http://consul-backups.minio.endpoint.com/: read tcp 10.1.0.147:39184->10.160.7.199:80: read: connection reset by peer
Here is my s3 file connection string.
-file="s3://key:token@consul-backups/consul_kvs.bak?region=default&endpoint=http://minio.endpoint.com&secure=false"
Hi,
Thanks for the tool.
I've a feature request for S3 backup. Currently the S3 backup works by specifying AWS access/secret keys. Is it possible to add a feature that uses IAM instance profile where possible?
docker run myena/consul-backinator backup -addr=10.0.1.50:8500 -file=s3://access key id:secretaccesskey@anilmytest/Backup?region=us-west-2
[Error] Failed to backup key data: BucketAlreadyOwnedByYou: Your previous request to create the named bucket succeeded and you already own it.
status code: 409, request id: 3F231DD26FB7B637, host id: sQDegGUjvnzlT5YH7lcwZzLHmZPd1gNUBLKrB65VPKgiifX1rImHJPexGjwa/kT41/Xi6RTLlv8=
I have a policy which allows the PutObject permission to the specific bucket where I store my backups. I am running version 1.6.5 and observe in my cloudtrail logs that backinator is trying to create the bucket; this fails for hopefully obvious reasons.
At this point I'm out of things to troubleshoot as by all rights this should work, and some months ago it did (I'm sure something changed since then, but right now I haven't bisected our change lists to find what it was).
Hi,
Is there a way that a frequency/interval option can be added? I am looking to run this in Docker on 10 min intervals.
im using consul as a backend to store vault info.
all information in consul is stores under the vault/ tree in encrypted format.
regards
meher
New non-fatal error code introduced in newer aws-sdk-go
packages. We shouldn't fail on this error.
HI
Cuurently using 1.0 version
consul-backinator backup file=/tmp/bkupfile prefix=/
2017/02/16 22:09:39 [Success] Backed up 0 keys from / to consul.bak
Keep your backup (consul.bak) and signature (consul.bak.sig) files in a safe place.
You will need both to restore your data.
Trying to use 1.4 version
./consul-backinator-1.4 backup file=/tmp/bkupfile prefix=/
2017/02/16 22:04:43 [Error] Failed to backup key data: No keys found
Same command fails with 1.4 , what am i doing wrong here
Hi,
Thanks for your job. I just tried it from a standalone container and it work perfectly.
I would like to integrate it to my docker compose. As cron jobs are not yet implemented in Swarm, I would like to know if there is a best practice for this?
Should I:
Many thanks by advance for your time.
Hello,
Consul 1.4 introduced a new ACL system where a token's permissions are tied to a policy (or multiple policies), rather than being baked into the token itself. I just recently upgraded my Consul cluster from version 1.3.1 to 1.4.5. After upgrade, all pre-existing non-management tokens were in "legacy" mode. At this point, consul-backinator was backing up the ACLs just fine.
However, after migrating the legacy tokens to the new format, consul-backinator now backs up only the management tokens and ignores anything with a policy tied to it. I confirmed this in two ways:
The consul-backinator log indicates that only a handful of ACL tokens were backed up, even though I have about 50 tokens in total
2022/03/07 15:05:02 [Success] Backed up 10 ACL tokens from consul.service.example.com:8501 to /path/to/backup/my-acls
I restored the above backup to a fresh test cluster and confirmed that only 10 ACL tokens exist (all of which are management tokens)
Note that KV backups appear to be working as they did previously.
Can you please implement support for backing up ACLs/Policies in Consul 1.4+? If this already exists, please let me know how to enable it. I can provide more details if needed.
Thank you!
Since we have docker integration now, please consider integration with https://docs.docker.com/docker-hub/builds/ to make it helpful for people to use this utility.
I noticed that the -dump
option provides values as base64 encoded.
Would be possible/preferable to display them as straight strings in the dump output?
Hello! I know as per help menu, option prefix
is intended only for backup
operation. But, when used with restore
, the result is misleading.
$ ./consul-backinator-master --token token -backup -dc localhost -file foo.x.y -prefix foo/x/y
2016/05/09 14:39:31 [Success] Backed up 1 keys from foo/x/y to foo.x.y
Keep your backup (foo.x.y) and signature (foo.x.y.sig) files in a safe place.
You will need both to restore your data.
$ ./consul-backinator-master --token token -restore -dc localhost -file foo.x.y -prefix foo/x/z
2016/05/09 14:40:04 [Success] Restored 1 keys from foo.x.y to foo/x/z
$ ./consul-backinator-master --token token -backup -dc localhost -file foo.x.z -prefix foo/x/z
2016/05/09 14:43:31 [Success] Backed up 0 keys from foo/x/z to foo.x.z
Keep your backup (foo.x.z) and signature (foo.x.z.sig) files in a safe place.
You will need both to restore your data.
As shown above, restore result indicated restore of 1 Key to foo/x/z
. But, nothing was restored. As you can see i tried to backup foo/x/z
and there is nothing to backup. I can confirm there is no key foo/x/z
.
I think prefix
option should be ignored with a message or should return an error when used with restore
.
Downloaded the container and run.
$ docker run myena/consul-backinator backup -addr consulhostname:8500 -dc consul-dc-01 -token xxxxx-xxxx-xxxx-xxxx -scheme http -file s3://mybucket-backup/consul
[Error] Failed to backup key data: Unexpected response code: 500
Hello! This tool has saved me a lot of overhead, thank you for sharing this amazing tool.
I am having an issue with transform
.
Currently transform
matches the pattern anywhere in the path, e.g. --transform foo/x/y,foo/x/z will transform below mentioned key path:
a/b/c/foo/x/y/a/b/c -> a/b/c/foo/x/z/a/b/c
foo/x/y -> foo/x/z
But, i only intend to transform the foo/x/y
to foo/x/z
(which matches absolute path from left to right).
I know i can take the backup with prefix foo/x/y
and then transform with the new key path as prefix
matches absolute path from left to right. If this is the recommended way, i can submit a PR with this info for the transform
option.
I am setting up backups from a Jenkins instance in EC2 that has an associated role allowing it to write to the consul-backups S3 bucket. I would like to have consul-backinator use the role rather than asking for credentials.
Hello,
Got another one (this one might be a little more complicated) - I've built the binary using the latest code (although this happens with 1.6.2 release as well). 'dump' seems to only work for KVs and not acls or queries. Consul is on version 0.9.3.
I perform a backup using the following:
# consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 03:42:14 [Success] Backed up 1 keys from / to kvs.bak
2017/11/03 03:42:14 [Success] Backed up 3 ACL tokens from to acls.bak
2017/11/03 03:42:14 [Success] Backed up 1 query definitions from to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
Then dump the KVs (which works as expected)
# consul-backinator dump -file kvs.bak -key [SECRET]
[
{
"Key": "consul/backup/lock",
"CreateIndex": 23,
"ModifyIndex": 3123,
"LockIndex": 7,
"Flags": 0,
"Value": "",
"Session": ""
}
]
Dumping acls complains that consul.bak does not exist (as I specifed a different filename during backup) and -nokv isn't implemented for dump
# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:00:24 [Error] Failed to dump data: open consul.bak: no such file or directory
# consul-backinator dump -nokv -acls acls.bak -key [SECRET]
flag provided but not defined: -nokv
Usage: consul-backinator dump [options]
Dump the contents of a backup file to stdout.
Options:
-file Source filename (default: "consul.bak")
-key Passphrase for data encryption and signature validation (default: "password")
-plain Dump a reduced set of information
-acls Specified file is an ACL token backup file
-queries Specified file is a prepared query backup file (consider using plain for query files)
Please see documentation on GitHub for a detailed explanation of all options.
https://github.com/myENA/consul-backinator
If I backup kvs to consul.bak and then try to dump acls.bak the result seems to be dependent upon where I place the flags (placing the -acls flag at the end just causes it to dump consul.bak as KVs, seemingly ignoring that the flag is even there):
# consul-backinator backup -file consul.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/03 04:04:52 [Success] Backed up 1 keys from / to consul.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
# consul-backinator dump -acls acls.bak -key [SECRET]
2017/11/03 04:05:56 [Error] Failed to dump data: gzip: invalid header
# consul-backinator dump -key [SECRET] -acls acls.bak
[
{
"Key": "consul/backup/lock",
"CreateIndex": 23,
"ModifyIndex": 3564,
"LockIndex": 1,
"Flags": 0,
"Value": "",
"Session": ""
}
]
The result is the same when I try dumping queries.
Not sure if I'm doing something fundamentally wrong but any help would be appreciated.
Thank you,
Mike
Hi guys,
What about a copy-before-delete feature that would move KV under a restore.bak/
prefix, prior to restore data ?!
Hi Team,
I have tried running consul-backinator to take the backup on S3 via k8s service account having enough IAM roles to perform consul backup directly to S3 bucket but it doesn't work.
S3 Policy has below:
"s3:GetObject",
"s3:DeleteObjectVersion",
"s3:AbortMultipartUpload",
"s3:DeleteObject",
"s3:PutObject"
However when i run it to take backup locally it works and just to check if there is any issue with the role i tried installing aws cli and did successful upload via aws-cli.
On another note acl backup always fail complaining
[Error] Failed to backup ACL tokens: Unexpected response code: 403 (Permission denied)
It would be best if you can even mention what kind of S3 permissions and consul policies are required to take the acl backup.
hello, i am trying to backup ACLS without key value and i got the permission denied error, could you please help me with this
docker run myena/consul-backinator backup -nokv -acls -token= -addr=10.18.1.82:8500 -file=s3://key id:secret@consul-bakup/backup-consul/$(date +%F_%R)_consul.bak?region=us-west-2
i run the same command without -nokv and -acls and it backed up keys but i am not able to backup ACL's, am i doing something wrong?
Consul api does provides this option through TLSConfig.
We are getting this error with consul:1.2.1 docker consul & backinator 1.6.5
panic: standard_init_linux.go:178: exec user process caused "no such file or directory" [recovered]
panic: standard_init_linux.go:178: exec user process caused "no such file or directory"
goroutine 1 [running, locked to thread]:
github.com/urfave/cli.HandleAction.func1(0xc4200a17a0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:478 +0x23f
panic(0x6f0cc0, 0xc42011d5e0)
/usr/lib/golang/src/runtime/panic.go:489 +0x2cf
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization.func1(0xc4200a1208, 0xc42000e0d0, 0xc4200a12a8)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:259 +0xc1
github.com/opencontainers/runc/libcontainer.(*LinuxFactory).StartInitialization(0xc4200509b0, 0xaa88e0, 0xc42011d5e0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/opencontainers/runc/libcontainer/factory_linux.go:277 +0x353
main.glob..func8(0xc42007c780, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main_unix.go:26 +0x66
reflect.Value.call(0x6d9be0, 0x750350, 0x13, 0x73b942, 0x4, 0xc4200a1760, 0x1, 0x1, 0xc4200a16f0, 0x731420, ...)
/usr/lib/golang/src/reflect/value.go:434 +0x91f
reflect.Value.Call(0x6d9be0, 0x750350, 0x13, 0xc4200a1760, 0x1, 0x1, 0x665b36, 0x73bace, 0x4)
/usr/lib/golang/src/reflect/value.go:302 +0xa4
github.com/urfave/cli.HandleAction(0x6d9be0, 0x750350, 0xc42007c780, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:487 +0x18f
github.com/urfave/cli.Command.Run(0x73baee, 0x4, 0x0, 0x0, 0x0, 0x0, 0x0, 0x74d391, 0x51, 0x0, ...)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/command.go:191 +0xac8
github.com/urfave/cli.(*App).Run(0xc4200aa000, 0xc42000c140, 0x2, 0x2, 0x0, 0x0)
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/Godeps/_workspace/src/github.com/urfave/cli/app.go:240 +0x5d6
main.main()
/builddir/build/BUILD/docker-ec8512ba42f349cbe5a63140c4d9c810f7afa527/runc-c5d311627d39439c5b1cc35c67a51c9c6ccda648/main.go:137 +0xbd2
to give users an extra sense of security that they have a real, untampered-with download.
Example: See the consul_0.6.4_SHA256SUMS file at https://releases.hashicorp.com/consul/0.6.4/
Would be great if you sign the file with a GPG key -- e.g.: consul_0.6.4_SHA256SUMS.sig
i am trying to restore keys to consul using
docker run myena/consul-backinator restore -addr=:8500 -file=s3://access key id:secret id@consul-bakup/stage-consul/2017-12-13_09:00_consul.bak?region=us-west-2
and i am getting error: rpc error permission denied
Hi there,
I'm running into a TLS verification error using consul-backinator. When I run the following (with and without -tls-skip-verify
):
# consul-backinator backup -addr=consul.dev.blahblah:8550 -scheme https -key=vault -ca-cert=/etc/consul/tls/ca.pem
2016/12/05 16:26:29 [Error] Failed to backup key data: Get https://consul.dev.blahblah:8550/v1/kv/?consistent=&recurse=: remote error: tls: bad certificate
What's interesting is that when I run:
# openssl verify -CAfile /etc/consul/tls/ca.pem /etc/consul/tls/cert.pem
/etc/consul/tls/cert.pem: OK
Additionally, when I run openssl s_client
I do get "Verify return code: 0 (ok)." I feel like the -ca-path
variable is being ignored, because if I leave the -CAfile
option off openssl, I do get verification errors. Is there some way to get more verbose output out of consul-backinator?
Since I requested the docker integration, would you like some use case docs to link to?
did docker run myena/consul-backinator backup -addr=10.0.1.50:8500 -file=s3://"access key id":"secret access key"@s3-bucket name/consul_backup?region=us-west-2
i am getting failed to backup data : invalid bucket name : the specified bucket is not valid,
but my nucket name is correct, can someone help ?
Hello,
I'm able to successfully backup kvs/acls/queries but when I try to restore it seems to only restore the kvs and acls (ignoring the queries).
consul-backinator v1.6.2
consul v0.9.3
# consul-backinator backup -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:26:36 [Success] Backed up 1 keys from / to kvs.bak
2017/11/02 21:26:36 [Success] Backed up 3 ACL tokens from to acls.bak
2017/11/02 21:26:36 [Success] Backed up 1 query definitions from to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
# consul-backinator restore -file kvs.bak -acls acls.bak -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:26:59 [Success] Restored 1 keys from kvs.bak to /
2017/11/02 21:26:59 [Success] Restored 3 ACL tokens from acls.bak to
And if I try specifying only the queries to backup and restore, I'm able to successfully back them up but get an error regarding my command flags when trying to restore.
# consul-backinator backup -nokv -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:33:41 [Success] Backed up 1 query definitions from to query.bak
Keep your backup and signature files in a safe place.
You will need both to restore your data.
# consul-backinator restore -nokv -queries query.bak -key [SECRET] -token [CONSUL_TOKEN]
2017/11/02 21:33:54 [Error] Passing 'nokv' and an empty 'acls' file doesn't make any sense. You should specify an 'acls' file when using the 'nokv' option.
Am I missing something in my command syntax? Or has the restore queries functionality not been fully implemented yet?
Thank you,
Mike
{"line":"2017-08-31T17:06:19Z [WARN] Error with docker; stopping container module="TaskEngine" task="ConsulBackup:3 arn:aws:ecs:us-west-2:776609208984:task/a03e35fe-2e1e-4a73-922a-e98f63a82b6f, Status: (CREATED-\u003eRUNNING) Containers: [ConsulBackup (RUNNING-\u003eRUNNING),]" container="ConsulBackup(myena/consul-backinator:latest) (RUNNING-\u003eRUNNING)" err="API error (404): oci runtime error: container_linux.go:247: starting container process caused "exec: \"backup -addr=10.0.1.50:8500 -file=s3://bakup-consul/consul_backup/first.bak?region=us-west-2\": stat backup -addr=10.0.1.50:8500 -file=s3://bakup-consul/consul_backup/first.bak?region=us-west-2: no such file or directory"","source":"stdout","tag":"ecs-agent/236df33fb65d"}
Can anyone please help me to resolve this error.
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: "/usr/local/bin/consul-backinator backup -file=consul.bak": stat /usr/local/bin/consul-backinator backup -file=consul.bak: no such file or directory"
Hi guys !
What about a parameter uploading to/restoring from AWS S3, instead of -file
?
Hello,
I am trying to pass an array of consul server but it gets failed, I think "-add" flag doesn't support array list.
Is there a way to backup all the acls in a consul cluster?
Hi! This is relevant to #3
By default, transform matches oldkey anywhere in the consul kv.
It would be great to have another option or similar (e.g. -tranform-mode=absolute|regex) to perform an absolute left to right match on the kv.
e.g.
kv:
x/y/z/foo/x/y -> ignore
foo/x/y -> match
command:
-transform foo/x/y,foo/x/z -transform-mode=absolute
kv:
x/y/z/foo/x/y -> match
foo/x/y -> match
command:
-transform foo/x/y,foo/x/z -transform-mode=regex
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.