No mutual exclusion enforced for MySQL backups about mysql HOT 8 CLOSED

This should be handled at the cluster level by set_backup_ttl. The idea being that's it's not enough to make sure we're not running multiple backups on a node but that we want to make sure it's only running on a single node as well.

We're setting that TTL in Consul before we run the backup specifically for the reason you're saying here. But of course it's possible that there's a bug -- have you see this happen and can you share logs and/or configuration that reproduces it?

from mysql.

I was just able to confirm this is the case with what's currently in master. Not sure how/why that regressed. Let me look into it.

from mysql.

Yes, I've seen this happen. It actually created a bunch of create_backup processes that turned into zombies.

I think the following execution happened: the master starts up, and executes its first health check. During this health check, it sees that LAST_BINLOG_KEY does not exist. Hence the call to is_binlog_stale in the health method returns true, triggering a write_snapshot. However, the LAST_BINLOG_KEY is only written after the snapshot is taken. So if the process of spawning a new process to make the snapshot takes longer than the health interval, multiple of these snapshot creation jobs can run.

from mysql.

Ok, I can confirm that. It looks like my reasoning here was that is_binlog_stale is getting called during the health check and that key will have been written. But the first time we call is_binlog_stale we'll always consider the binlog stale and so each time the health check fires it's going to attempt the snapshot again. Not sure how this got missed in the first pass because it seems like an obvious bug. 😊

I'm going to spend a bit of time on this this morning.

from mysql.

@moretea I've dropped a fix into #20 and it's been merged. Want to give that a try?

from mysql.

Gah... I've had a chance to test that better in #21 and it won't work like that either.

from mysql.

#21 has been updated with this fix.

from mysql.

Fixed in #21, which has been merged.

from mysql.