Comments (7)
You are using Pseudo-GTID to move replicas around. The logic behind this is to align binlog coordinates by inspecting binlog contents. orchestrator
looks for matching Pseudo-GTID entries in binlog coordinates of both involved servers, and then proceeds to iterate the binary logs, event by event.
It expects the binary logs to be identical in content. However in your case it found a mismatch. The binary logs on one server had an event not found in the other server (or, more likely, not in same order).
This might happen if you directly write onto your replicas. Or if you have active-active master-master. Do you have such a case?
from orchestrator.
@theTibi the pseudo-GTID insertion must only run on the master. If it runs on multiple nodes, then nothing makes sense to orchestrator
, because each node generates its own unique entries.
This agrees with your second observation. When you fail over, something needs to stop writing to the old master and start writing to the new master.
I've done this in various different ways:
- an
event_scheduler
, which is always implicitly disabled on replicas ; need to enable on failover - a
pt-heartbeat
running on the master: need toservice stop
on old master,service start
on new master - a
pt-hearteat
or similar, running from completely elsewhere, using the master's VIP; when the VIP changes, the heartbeat writes implicitly go to the new master, nothing to change in the heartbeat mechanism.
from orchestrator.
Thank you for the useful information. Yes, I am using Pseudo-GTID to move replicas around. Oracle GTID is not enabled. Previously, we did master-master and the application could feasibly write to both masters. However, we have stopped doing that as part of using Orchestrator and now only have master-slave but I still get this error even though all the slaves are read-only. We have many other MySQL severs that work great with PGTID and Orchestrator. I don't have this problem with any of the other stacks so it appears to be isolated to this particular application. In order to work around this error, I've had to write a script to try and simulate what Orchestrator is doing by capturing the binlog coordinates and executing a CHANGE MASTER command. However, I'd really like to figure what is causing this to happen because Orchestrator does a better job of relocating slaves.
from orchestrator.
Is there perhaps a local cronjob, or a event_scheduler
that has SUPER
privileges to override the read_only
configuration?
Are you using ROW
based replication everywhere? Perhaps you have ROW
on one server and MIXED
on another?
from orchestrator.
Hi,
I had the same issue, and I am a bit confused now. When we are using Pseudi-GTID , the event which inserts the unique entries to the binary log has to run on all the servers or just only the master. If it has to run only on the master and when it goes down some job has to start this event on the new master again?
Thanks.
from orchestrator.
Thank you @shlomi-noach to made this clear for me.
from orchestrator.
Thank you @shlomi-noach for the information. I didn't find any cronjobs running locally. The event_scheduler is on for the replicas but the only event I have is for P-GTID. There are a number of non-application users that have SUPER so it is possible that something is writing to the slave. I will continue to investigate. We are using ROW based replication everywhere so that doesn't seem to be a problem either.
from orchestrator.
Related Issues (20)
- Project dead? HOT 3
- DeadIntermediateMaster auto Recovery HOT 1
- Failure from hooks are not reflected on UI HOT 1
- Automated recovery invalid HOT 5
- Should set old master's read_only to false under DeadCoMaster failure
- Report IntermediateMaster errors under CoMaster deployment
- Set the old primary host read_only=0 if failover/switch over failed.
- Whether ORC provides APIs that enable/disable recovery for specific clusters? HOT 1
- Using Azure AD Authentication
- How to set autocommit for orchestrtor detection thread
- How do you perform system tests? HOT 1
- Improve logging of problems during db connection
- Orchestrator able to display topology but still reports as failure
- After the Mysql domain name connection is configured。If the orchestrator topology displays both ip addresses and domain names of the same backup database, how can I fix this problem
- Discovery doesn't discover in extremely simple setup HOT 4
- Does orchestrator's website :3000 has login account?
- Does anyone know why updates stopped? HOT 2
- add windows version
- Ptrhon API Token
- How to login with username and password through API ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from orchestrator.