Comments (6)
It seems that the Consul state isn't maintained anywhere persistently, so the next time the environment comes up, it starts fresh.
We write a key to Consul to let the preStart
know where to find the snapshot. If Consul's data was blown away then yeah, this won't work. You want to keep Consul running.
from mysql.
Is there a way to have the Consul instance write it's key/value pairs to a persistent storage area?
Could the manage.py script look into Manta to find the latest backup if Consul doesn't know?
from mysql.
Is there a way to have the Consul instance write it's key/value pairs to a persistent storage area?
That would be a function of how and where you deploy Consul. In an environment with attached network volumes (ex. NFS or soon-to-be-released Joyent/RFD26-style volumes) I imagine you could have your Consul containers use that. A better approach is to have a Consul raft for HA and then the failure of individual nodes won't delete data.
Could the manage.py script look into Manta to find the latest backup if Consul doesn't know?
Sure, in theory, but none of our locks can work without Consul. But maybe I'm not understanding the problem you're running into. If you blow away all the metadata for orchestration then none of your service discovery works. I think the right way to fix that is to not blow it away and make sure it's HA.
from mysql.
Sorry for the late reply, I was on vacation...
My concern is that if (for whatever reason) all of the Consul instances in HA somehow die, none of the metadata will be available when the environment is repaired. The databases will come backup fresh and clean, and all of the data previously stored is "lost". The last backups would stil exist, but the DB servers would not go and fetch them to restore.
Are there examples for Consul using a network volume to store the metadata persistently (e.g., into Manta)?
from mysql.
Are there examples for Consul using a network volume to store the metadata persistently (e.g., into Manta)?
Manta isn't a block store so it wouldn't be suitable except for backups. You're probably looking for NFS or soon-to-be-released Joyent/RFD26-style volumes. Alternately, you could add ContainerPilot to your Consul instances and give it a postStop
handler to snapshot the data to Manta.
from mysql.
Ok, so this is really a "how do I backup Consul?" question at this point, so if there aren't any objections I'm going to close it. Feel free to reopen if there's more to discuss @headw01
from mysql.
Related Issues (20)
- Document SNAPSHOT_BACKEND
- Health check should verify replication
- Simplify CNS search domain for consumers HOT 4
- MySQL not starting HOT 1
- Not starting cluster, can't find mysql-primary HOT 5
- Re-configure after a consul system failure. HOT 1
- This container has its own installation of consul? HOT 2
- Replication failure HOT 41
- Create snapshot when mysql-primary is restarted/recreated HOT 2
- consul rpc error: invalid session HOT 2
- Consul 2 private interfaces HOT 2
- Failover not reporting to consul HOT 4
- Switch to Percona 5.7 HOT 1
- Support for semisynchronous replication HOT 3
- Consider adding Percona Monitoring and Management agent
- Failover lost master HOT 1
- Snapshots not backing up HOT 2
- Minio restarting causes unhealthy mysql-primary
- InnoDB restore in the future
- Not able to run cluster. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mysql.