Comments (12)
alexku-o
commented on June 18, 2024
1
@ben-nz - using -b like consulate kv backup -b worked for me (following what was suggested in #41)
from consulate.
alexku-o
commented on June 18, 2024
1
@tyx thanks. I actually tried restore with -b earlier but it was giving me some error. I ended up just using consul-backup because we have a simple backup/restore use case and it works perfectly well for us (and the fact that our test backup/restore worked with consul-backup but not with consulate).
edit: In case it came across that way, my comment here was certainly not in any way to discredit the incredible work that's been done on consulate. I appreciate a great deal of the work here, and am thankful.
from consulate.
ben-nz
commented on June 18, 2024
Looks like I'm striking the same issue after pointing vault to our consul store:
[centos@consul-vault-i-ca11b50e ~]$ consulate kv backup
Traceback (most recent call last):
File "/usr/bin/consulate", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/site-packages/consulate/cli.py", line 469, in main
KV_ACTIONS[args.action](consul, args)
File "/usr/lib/python2.7/site-packages/consulate/cli.py", line 216, in kv_backup
handle.write(json.dumps(records) + '\n')
File "/usr/lib64/python2.7/json/__init__.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib64/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib64/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xf9 in position 9: invalid start byte
Let me know if you'd like more info.
from consulate.
gmr
commented on June 18, 2024
Can you provide a sample consul DB dir for me to try and backup?
from consulate.
njsaunders
commented on June 18, 2024
Apologies for the delay - Consul DB containing Vault data attached.
consul-dir-vault-fail.tar.gz
from consulate.
alexku-o
commented on June 18, 2024
@ben-nz - also, in case it wasn't obvious to you (as it wasn't to me), seems like you'd not use the -b option when you restore the file that you backed up with -b option (I tested that and that worked).
So, -b for backup, and no -b for restore.
from consulate.
ben-nz
commented on June 18, 2024
@alexku-o Thanks mate I'll give that a try.
from consulate.
tyx
commented on June 18, 2024
@alexku-o On my side I had to use -b (for base64) also to restire. If not, restore will succeed but vault would not be able to access its data anymore.
from consulate.
isavcic
commented on June 18, 2024
I'm hitting the same error. Value is binary data populated by consul-template for deduplication purposes, residing under consul-template/dedup/$HASH.
# consulate kv backup
Traceback (most recent call last):
File "/usr/local/bin/consulate", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 469, in main
KV_ACTIONS[args.action](consul, args)
File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 216, in kv_backup
handle.write(json.dumps(records) + '\n')
File "/usr/lib/python2.7/json/__init__.py", line 243, in dumps
return _default_encoder.encode(obj)
File "/usr/lib/python2.7/json/encoder.py", line 207, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/lib/python2.7/json/encoder.py", line 270, in iterencode
return _iterencode(o, 0)
UnicodeDecodeError: 'utf8' codec can't decode byte 0xfe in position 1: invalid start byte
# consulate kv backup -b
Traceback (most recent call last):
File "/usr/local/bin/consulate", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 469, in main
KV_ACTIONS[args.action](consul, args)
File "/usr/local/lib/python2.7/dist-packages/consulate/cli.py", line 214, in kv_backup
records = [(k, f, base64.b64encode(v)) for k,f,v in records]
File "/usr/lib/python2.7/base64.py", line 53, in b64encode
encoded = binascii.b2a_base64(s)[:-1]
TypeError: must be string or buffer, not None
from consulate.
djenriquez
commented on June 18, 2024
Getting the same error as @isavcic when using -b, any progress for this?
from consulate.
djenriquez
commented on June 18, 2024
In the meantime, I created a dockerized solution for backing up Consul's datastore that works with Vault: Consulidator
Backup:
CONSUL_ADDRESS=10.0.1.2
CONSUL_PORT=8500
BACKUP_FILE=my_backup
docker run --rm \
djenriquez/consulidator:v0.1.0 \
--backup \
--port $CONSUL_PORT $CONSUL_ADDRESS > $BACKUP_FILERestore:
Assuming that you are running the restore in the working directory that contains $BACKUP_FILE
CONSUL_ADDRESS=10.0.1.2
CONSUL_PORT=8500
BACKUP_FILE=my_backup
docker run --rm \
-v `pwd`:/restore \
djenriquez/consulidator:v0.1.0 \
--restore $BACKUP_FILE \
--port $CONSUL_PORT $CONSUL_ADDRESSfrom consulate.
gmr
commented on June 18, 2024
Closing this and tracking the forward direction in #109 and #110
from consulate.
Related Issues (20)
- Add TCP check to service registration HOT 1
- Add DeregisterCriticalServiceAfter and status parameters to server ttl check HOT 1
- Use PUT to deregister services and checks HOT 1
- CLI: Change backup to dump
- CLI: Consider adding wrapper for snapshot
- CLI: Add support for manipulating individual ACLs HOT 1
- Consul kv race condition 'Error setting "$key" (200)'
- Release 0.6.0 not fully working with newest Consul version
- Support service metadata for service registration
- Bug is in api method deregister service. HOT 3
- 0.7 release? HOT 3
- consulate can not support DeregisterCriticalServiceAfter params?
- minor typo in readme
- Filter nodes out, with maintenance state
- Python 3.9 support
- json encode problem
- Services should be deactivated using the HTTP PUT method.
- AttributeError: 'bytes' object has no attribute 'get' when Consul in leaderless state
- Add version feature to cli HOT 1
- pkg_resources.DistributionNotFound: argparse on latest rc 7 HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from consulate.