MySQL Master isn't starting about mysql HOT 10 CLOSED

You've got this error in the logs above:

mysql_1 | InnoDB: Error number 28 means 'No space left on device'.

Can you check the disk before we start digging into this further? If that looks good, can you share whether you're using the Compose file and image as-is or whether you've made changes?

from mysql.

I saw that as well but

df -h
Filesystem      Size   Used  Avail Capacity  iused    ifree %iused  Mounted on
/dev/disk1     465Gi  245Gi  220Gi    53% 64195080 57650454   53%   /
devfs          328Ki  328Ki    0Bi   100%     1136        0  100%   /dev
map -hosts       0Bi    0Bi    0Bi   100%        0        0  100%   /net
map auto_home    0Bi    0Bi    0Bi   100%        0        0  100%   /home

I'm using the images as is and the compose file. I added a link to the compose file for mysql to find consul

from mysql.

Interesting... what's the deployment environment? This looks like it's on Docker for Mac? Can you run docker run -it autopilotpattern/mysql df -h for me against wherever you're deploying?

from mysql.

It is docker4mac

Filesystem      Size  Used Avail Use% Mounted on
none             60G  2.4G   54G   5% /
tmpfs          1002M     0 1002M   0% /dev
tmpfs          1002M     0 1002M   0% /sys/fs/cgroup
/dev/vda2        60G  2.4G   54G   5% /etc/hosts
shm              64M     0   64M   0% /dev/shm

Funny I just deleted the whole sparse image (18G) and rebooted the machine. The max. should be at 64G.
The above mentioned prob seems to be gone, but now I get this:

mysql_1   | Version: '5.6.29-76.2-log'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Percona Server (GPL), Release 76.2, Revision ddf26fe
mysql_1   | 2016-05-09 13:22:22,817 ERROR manage.py mysql_tzinfo_to_sql returned error.
mysql_1   | 2016-05-09 13:22:22,818 INFO manage.py Generated root password: LrUbNKaQFpUrIHnhsAxT
mysql_1   | 2016-05-09 13:22:22 69 [Warning] 'proxies_priv' entry '@ root@e84c8306e2ee' ignored in --skip-name-resolve mode.
mysql_1   | 2016-05-09 13:22:22 69 [Warning] 'proxies_priv' entry '@ root@e84c8306e2ee' ignored in --skip-name-resolve mode.
mysql_1   | 2016-05-09 13:22:22,824 ERROR manage.py (1227, u'Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation')

from mysql.

Can you drop that whole log in a gist?

from mysql.

https://gist.github.com/muemich/51f5eaa92581bf60bdd034ed34813644

After changing the user/pwd the master came up. But the slaves still aren't working.

from mysql.

Just so that I'm clear, the logs you have here are from a totally separate run, right?

You've got this line in your log for mysql_2:

[32mmysql_2 |�[0m 2016-05-09 13:49:27,768 INFO manage.py Initializing database...

That log entry is in the initialize_db function, which gets call in the pre_start function only when there's no snapshot to fetch from the object store. Because we're initializing the DB in the replicas instead of getting the DB from the snapshot, the replicas don't have the user we need to set up replication.

So the thing we need to figure out is what happened to the snapshot. What jumps out at me in the log are the lines that we're missing. Somewhere in this section it looks like we died.

Here's where we should look next:

• Verify that the snapshot exists on Manta.
• Verify that the LAST_BINLOG_KEY and/or LAST_BACKUP_KEY have been written to Consul.
• If you create a replica with LOG_LEVEL=DEBUG, you'll see traces and exit values for most of the functions in this path.

from mysql.

Yes they are!

• there is a snapshot on Manta
• the values in consul are called mysql-last-backup and mysql-last-binlog
• Will try that

But there is a difference in the KEYS you mentioned and the KEYS in consul. So I deleted all backups and tried a new run. And it's working.

All of this is a bit strange. And I can't reproduce all the issues faced

from mysql.

But there is a difference in the KEYS you mentioned and the KEYS in consul.

Sorry, the ALL CAPS keys I mentioned are the consts in the code; they can be overridden in your _env file which is why I called them by the const name rather than the specific bit of text.

And I can't reproduce all the issues faced

Ugh... that's the worst. Not your fault, of course, just the trouble with replicating these kinds of failures. Turns out on second look that the log entires I was looking for in your logs were log.debug anyways, so as far as I can tell everything worked as expected from the primary and the failure was at the replicas. Maybe the replicas came up before the snapshot was done being written but once we check Manta/Consul everything is in place? That might be an area to improve (ref similar problem here: #22 (comment)).

It looks like there's a couple spots we could hook in more logging too. The balance of debug vs info has been hard to get right; I'm wondering if we should keep the last N messages of a DEBUG level in a ring log with some mechanism to dump them to stdout even if the instances are logging at INFO.

I'm also in the midst of working on an integration test rig so that I can do end-to-end testing on this and other autopilotpattern applications. So hopefully with better testing we'll be able to suss out these issues.

from mysql.

@muemich I've opened #29 and #30 to cover issues that we may have seen here (but aren't sure we saw).

from mysql.