How can I take backup for redis running on cluster mode using redis-dump-go?

Hi Yann,

Thanks for redis-dump-go, that's a nice tool. But it could be even nicer properly packaged in all of our favorites distributions :-)

Could you please create a tag or release a first version of the project ?

Best regards

Hello,

redis-server 5.0.9
redis-dump-go v0.7.0

Dump and restore all dbs doesn't work. All dbs (3,5,7) restoring to 0 db.

before restore:

Keyspace

db0:keys=280317,expires=279847,avg_ttl=208563507
db7:keys=8927,expires=0,avg_ttl=0

backup:
REDISDUMPGO_AUTH=pass redis-dump-go -port 6379 -host 127.0.0.1 -n 100 > redis.dump
REDISDUMPGO_AUTH=pass redis-cli -c -h 127.0.0.1 -p 6379 --pipe < redis.dump

after restore:

Keyspace

db0:keys=289432,expires=280032,avg_ttl=214048507

Is it normal behaviour now?

Thanks.

The "silent" flag was causing redis-dump-go to hang in v0.3.1, we should have some sort of simple integration tests guaranteeing that basic functionality works.

While trying to build the project, I encountered errors with golang.org/x/xerrors package

$ export GOPATH=$(mktemp -d)
$ go get github.com/yannh/redis-dump-go
# github.com/yannh/redis-dump-go/vendor/golang.org/x/xerrors
/tmp/tmp.H8Wn4abREk/src/github.com/yannh/redis-dump-go/vendor/golang.org/x/xerrors/adaptor_go1_13.go:16:14: undefined: errors.Frame
/tmp/tmp.H8Wn4abREk/src/github.com/yannh/redis-dump-go/vendor/golang.org/x/xerrors/format_go1_13.go:12:18: undefined: errors.Formatter

Steps to reproduce the bug

$ export GOPATH=$(mktemp -d)
$ go get github.com/yannh/redis-dump-go
...error...
$ cd ${GOPATH}/src/github.com/yannh/redis-dump-go
$ go build
go: downloading github.com/mediocregopher/radix/v3 v3.3.2
go: extracting github.com/mediocregopher/radix/v3 v3.3.2
go: downloading golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522
go: extracting golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522
go: finding github.com/mediocregopher/radix/v3 v3.3.2
go: finding golang.org/x/xerrors v0.0.0-20190513163551-3ee3066db522
# golang.org/x/xerrors
../../../../pkg/mod/golang.org/x/[email protected]/adaptor_go1_13.go:16:14: undefined: errors.Frame
../../../../pkg/mod/golang.org/x/[email protected]/format_go1_13.go:12:18: undefined: errors.Formatter

Fix

$ cd ${GOPATH}/src/github.com/yannh/redis-dump-go
$ go get -u ./...
go: finding github.com/mediocregopher/radix/v3 v3.4.1
go: finding golang.org/x/xerrors latest
go: downloading github.com/mediocregopher/radix/v3 v3.4.1
go: extracting github.com/mediocregopher/radix/v3 v3.4.1
go: downloading golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
go: extracting golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898
$ go build
$ go install

Thank you for the tool @yannh, any plan to support AUTH?

I want to view the data in json format ,but I found -output not support.
-output string
Output type - can be resp or commands (default "resp")

I am attempting to use this to backup a redis instance that I am using with lorawan-stack and when I run the tool I am getting the following output:

*2
$6
SELECT
$1
0
Database 0: 5 element dumpedError: Key ttn:v3:ns:application-uplinks:uid:halter:uplinks:invalidation is of unreconized type stream

For more context I am using AWS ElastiCache engine version 5.0.5.

Version: v0.7.0
Platform: Linux CentOS 7.9.2009 x86_64

Why its using url.QueryEscape for password?
I have a password with special characters. And result: ERR invalid password

Example Password for this issue (randomly generated, safe, not using anywhere):
,u[!m&bd3*+tQK,ZC#YS%=}Fn#qw)~$<

Script

#!/bin/sh
export REDISDUMPGO_AUTH=",u[!m&bd3*+tQK,ZC#YS%=}Fn#qw)~$<"
./redis-dump-go

I tried without QueryEscape and its worked for me.

for example:
-d DB dump DATABASE (0-N, default 0)

I have tried to take backup when Redis is running in Cluster Mode. The backup only contains data from one shard. To be precise, when the scan command is run, it only list the data from one node in the shard.

Is there any workaround using which I can take backup when Redis is running in Cluster Mode ?

When I dumped the data from one node and dumped it to other nodes, I found that the dumping of data with null value failed with the following error

using -output commands

ERR wrong number of arguments for 'set' command

The format of this type of data in the dump file is similar to the following

SET keyofnovalue 

It seems to work if I put it in double quote

SET keyofnovalue ""

But I'm not sure if it can be solved by double quotes for other types of data as well.

Is it possible to put double quotes for all types of data value if applicable

I'm getting authentication error with the latest build. It was working fine with version 0.5.1.
Seems like getDBIndexes function is not using the password. Can the auth error be because of that?

I am trying to use your application through importing it from golang.
The way the code works is that it requires a log.Logger then uses the log.Printf and log.Print function from this struct.
If we could replace the log.Logger with lets say a "Printer" interface which contains only Printf and Print then users can write their own and have the value saved to a struct or something else.

Please add a abilty to rename keys and prepend or append some string

redis version: 5.0.4
Sometimes it(redis-dump-go) hangs and waits for a long time.

redis version: 5.0.4
redis-dump: 0.4.6

[root@mtl-gslb-redis ~]# redis-cli info
# Keyspace
db0:keys=6890,expires=0,avg_ttl=0
db1:keys=86949,expires=0,avg_ttl=0             large keys is in db1
db2:keys=1217,expires=0,avg_ttl=0
db3:keys=8196,expires=0,avg_ttl=0
db4:keys=493653,expires=0,avg_ttl=0
db5:keys=1178,expires=0,avg_ttl=0
db6:keys=1,expires=0,avg_ttl=0
db8:keys=1,expires=0,avg_ttl=0
db12:keys=155,expires=0,avg_ttl=0

redis dump success
but import data error:

[root@mtl-gslb-redis ~]# redis-cli -n 2  --pipe < 2.resp    
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
All data transferred. Waiting for the last reply...
Last reply received from server.
errors: 0, replies: 1218
[root@mtl-gslb-redis ~]# redis-cli -n 1 --pipe < 1.resp   
Warning: Using a password with '-a' or '-u' option on the command line interface may not be safe.
ERR Protocol error: invalid multibulk length
Error reading from the server: Connection reset by peer

redis-cli -h platform-test-small-redis-redis-ha.platform-test-small.svc.cluster.local --pipe < redis-backup.resp
Error reading from the server: Connection reset by peer

Thanks @yannh You've created a very nice tool and it works perfectly well. I just have one query on keys filtering. How can I filter specific keys and dump those only?
eg: xyz:

Hello
looks like you need to update the build section of readme with adding env variable
GO111MODULE=off
overwise ${GOPATH}/src folder is empty after go get:
/bin/sh: cd: line 1: can't cd to /go/src/github.com/yannh/redis-dump-go: No such file or directory

Hi,

What's the reason for having this check:

		if dbIndex > 16 {
			return nil, fmt.Errorf("Error parsing INFO keyspace")
		}

We do backups with cronjob in kubernetes like this:

command: ['bash', '-c', '/redis-dump-go -host redis > /backup/redis.resp']

The docker-image you provide doesn't have a shell only the binary and you don't provide an option to specify output file..