Comments (10)
You've got this error in the logs above:
mysql_1 | InnoDB: Error number 28 means 'No space left on device'.
Can you check the disk before we start digging into this further? If that looks good, can you share whether you're using the Compose file and image as-is or whether you've made changes?
from mysql.
I saw that as well but
df -h
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk1 465Gi 245Gi 220Gi 53% 64195080 57650454 53% /
devfs 328Ki 328Ki 0Bi 100% 1136 0 100% /dev
map -hosts 0Bi 0Bi 0Bi 100% 0 0 100% /net
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
I'm using the images as is and the compose file. I added a link to the compose file for mysql to find consul
from mysql.
Interesting... what's the deployment environment? This looks like it's on Docker for Mac? Can you run docker run -it autopilotpattern/mysql df -h
for me against wherever you're deploying?
from mysql.
It is docker4mac
Filesystem Size Used Avail Use% Mounted on
none 60G 2.4G 54G 5% /
tmpfs 1002M 0 1002M 0% /dev
tmpfs 1002M 0 1002M 0% /sys/fs/cgroup
/dev/vda2 60G 2.4G 54G 5% /etc/hosts
shm 64M 0 64M 0% /dev/shm
Funny I just deleted the whole sparse image (18G) and rebooted the machine. The max. should be at 64G.
The above mentioned prob seems to be gone, but now I get this:
mysql_1 | Version: '5.6.29-76.2-log' socket: '/var/run/mysqld/mysqld.sock' port: 3306 Percona Server (GPL), Release 76.2, Revision ddf26fe
mysql_1 | 2016-05-09 13:22:22,817 ERROR manage.py mysql_tzinfo_to_sql returned error.
mysql_1 | 2016-05-09 13:22:22,818 INFO manage.py Generated root password: LrUbNKaQFpUrIHnhsAxT
mysql_1 | 2016-05-09 13:22:22 69 [Warning] 'proxies_priv' entry '@ root@e84c8306e2ee' ignored in --skip-name-resolve mode.
mysql_1 | 2016-05-09 13:22:22 69 [Warning] 'proxies_priv' entry '@ root@e84c8306e2ee' ignored in --skip-name-resolve mode.
mysql_1 | 2016-05-09 13:22:22,824 ERROR manage.py (1227, u'Access denied; you need (at least one of) the SUPER, REPLICATION CLIENT privilege(s) for this operation')
from mysql.
Can you drop that whole log in a gist?
from mysql.
https://gist.github.com/muemich/51f5eaa92581bf60bdd034ed34813644
After changing the user/pwd the master came up. But the slaves still aren't working.
from mysql.
Just so that I'm clear, the logs you have here are from a totally separate run, right?
You've got this line in your log for mysql_2
:
[32mmysql_2 |�[0m 2016-05-09 13:49:27,768 INFO manage.py Initializing database...
That log entry is in the initialize_db
function, which gets call in the pre_start
function only when there's no snapshot to fetch from the object store. Because we're initializing the DB in the replicas instead of getting the DB from the snapshot, the replicas don't have the user we need to set up replication.
So the thing we need to figure out is what happened to the snapshot. What jumps out at me in the log are the lines that we're missing. Somewhere in this section it looks like we died.
Here's where we should look next:
- Verify that the snapshot exists on Manta.
- Verify that the LAST_BINLOG_KEY and/or LAST_BACKUP_KEY have been written to Consul.
- If you create a replica with
LOG_LEVEL=DEBUG
, you'll see traces and exit values for most of the functions in this path.
from mysql.
Yes they are!
- there is a snapshot on Manta
- the values in consul are called
mysql-last-backup
andmysql-last-binlog
- Will try that
But there is a difference in the KEYS you mentioned and the KEYS in consul. So I deleted all backups and tried a new run. And it's working.
All of this is a bit strange. And I can't reproduce all the issues faced
from mysql.
But there is a difference in the KEYS you mentioned and the KEYS in consul.
Sorry, the ALL CAPS keys I mentioned are the consts in the code; they can be overridden in your _env
file which is why I called them by the const name rather than the specific bit of text.
And I can't reproduce all the issues faced
Ugh... that's the worst. Not your fault, of course, just the trouble with replicating these kinds of failures. Turns out on second look that the log entires I was looking for in your logs were log.debug
anyways, so as far as I can tell everything worked as expected from the primary and the failure was at the replicas. Maybe the replicas came up before the snapshot was done being written but once we check Manta/Consul everything is in place? That might be an area to improve (ref similar problem here: #22 (comment)).
It looks like there's a couple spots we could hook in more logging too. The balance of debug vs info has been hard to get right; I'm wondering if we should keep the last N messages of a DEBUG level in a ring log with some mechanism to dump them to stdout even if the instances are logging at INFO.
I'm also in the midst of working on an integration test rig so that I can do end-to-end testing on this and other autopilotpattern applications. So hopefully with better testing we'll be able to suss out these issues.
from mysql.
@muemich I've opened #29 and #30 to cover issues that we may have seen here (but aren't sure we saw).
from mysql.
Related Issues (20)
- Document SNAPSHOT_BACKEND
- Health check should verify replication
- Simplify CNS search domain for consumers HOT 4
- MySQL not starting HOT 1
- Not starting cluster, can't find mysql-primary HOT 5
- Re-configure after a consul system failure. HOT 1
- This container has its own installation of consul? HOT 2
- Replication failure HOT 41
- Create snapshot when mysql-primary is restarted/recreated HOT 2
- consul rpc error: invalid session HOT 2
- Consul 2 private interfaces HOT 2
- Failover not reporting to consul HOT 4
- Switch to Percona 5.7 HOT 1
- Support for semisynchronous replication HOT 3
- Consider adding Percona Monitoring and Management agent
- Failover lost master HOT 1
- Snapshots not backing up HOT 2
- Minio restarting causes unhealthy mysql-primary
- InnoDB restore in the future
- Not able to run cluster. HOT 6
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from mysql.