Comments (8)
This should be handled at the cluster level by set_backup_ttl
. The idea being that's it's not enough to make sure we're not running multiple backups on a node but that we want to make sure it's only running on a single node as well.
We're setting that TTL in Consul before we run the backup specifically for the reason you're saying here. But of course it's possible that there's a bug -- have you see this happen and can you share logs and/or configuration that reproduces it?
from mysql.
I was just able to confirm this is the case with what's currently in master. Not sure how/why that regressed. Let me look into it.
from mysql.
Yes, I've seen this happen. It actually created a bunch of create_backup
processes that turned into zombies.
I think the following execution happened: the master starts up, and executes its first health check. During this health check, it sees that LAST_BINLOG_KEY
does not exist. Hence the call to is_binlog_stale
in the health
method returns true, triggering a write_snapshot
. However, the LAST_BINLOG_KEY
is only written after the snapshot is taken. So if the process of spawning a new process to make the snapshot takes longer than the health interval, multiple of these snapshot creation jobs can run.
from mysql.
Ok, I can confirm that. It looks like my reasoning here was that is_binlog_stale
is getting called during the health check and that key will have been written. But the first time we call is_binlog_stale
we'll always consider the binlog stale and so each time the health check fires it's going to attempt the snapshot again. Not sure how this got missed in the first pass because it seems like an obvious bug. 😊
I'm going to spend a bit of time on this this morning.
from mysql.
@moretea I've dropped a fix into #20 and it's been merged. Want to give that a try?
from mysql.
Gah... I've had a chance to test that better in #21 and it won't work like that either.
from mysql.
#21 has been updated with this fix.
from mysql.
Fixed in #21, which has been merged.
from mysql.
Related Issues (20)
- Document SNAPSHOT_BACKEND
- Health check should verify replication
- Simplify CNS search domain for consumers HOT 4
- MySQL not starting HOT 1
- Not starting cluster, can't find mysql-primary HOT 5
- Re-configure after a consul system failure. HOT 1
- This container has its own installation of consul? HOT 2
- Replication failure HOT 41
- Create snapshot when mysql-primary is restarted/recreated HOT 2
- consul rpc error: invalid session HOT 2
- Consul 2 private interfaces HOT 2
- Failover not reporting to consul HOT 4
- Switch to Percona 5.7 HOT 1
- Support for semisynchronous replication HOT 3
- Consider adding Percona Monitoring and Management agent
- Failover lost master HOT 1
- Snapshots not backing up HOT 2
- Minio restarting causes unhealthy mysql-primary
- InnoDB restore in the future
- Not able to run cluster. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mysql.