Comments (3)
shlomi-noach
commented on May 14, 2024
is this is the expected behavior?
Of course: the assumption is the only the master has read_only=0.
I'm not sure I understand your setup: you have a topology where an intermediate master has read_only=0, and this is also the only server where read_only=0, and ProxySQL sends write traffic in that direction?
I assume this is related to #46
The solution should basically be "do it in hooks", unless I see a convincing reason to change such behavior, in a way that is generic enough to not cause damage to others.
from orchestrator.
theTibi
commented on May 14, 2024
@shlomi-noach I am just running tests and try to understand Orchestrator behaviours more.
So in my test case we have a replicaset where there are intermediate-masters these masters also get writes (they have extra schemas), so read_only=OFF.
On the salves which connect to this intermediate master the read_only=ON.
If intermediate master goes down the read_only stays ON on the promoted slave.
And yes I can handle this with hooks I was just wondering if Orchestrator promotes a server and the failed host had read_only=OFF should be read_only=OFF also on the promoted server?
from orchestrator.
theTibi
commented on May 14, 2024
It looks like this is much more complex because if I am running a sysbench on the intermediate master simulating writes on a schema which is not exist on the master, Orchestrator won't do the failover because it finds mismatching entires:
2017-01-12 14:00:28 DEBUG topology_recovery: - RecoverDeadIntermediateMaster: will next attempt to relocate up from rep2:3306
2017-01-12 14:00:28 INFO Will move 1 slaves below rep1:3306 via GTID
2017-01-12 14:00:28 DEBUG auditType:begin-maintenance instance:rep3:3306 cluster:rep1:3306 message:maintenanceToken: 305, owner: orc, reason: rep3:3306 match below rep1:3306 as part of MultiMatchBelow
2017-01-12 14:00:28 DEBUG MultiMatchBelow: stopping 1 slaves nicely
2017-01-12 14:00:28 DEBUG Stopping 1 slaves nicely
2017-01-12 14:00:28 INFO Stopped slave nicely on rep3:3306, Self:mysql-bin.000001:18925723, Exec:mysql-bin.000011:6036419
2017-01-12 14:00:28 INFO Stopped slave on rep3:3306, Self:mysql-bin.000001:18925723, Exec:mysql-bin.000011:6036419
2017-01-12 14:00:28 DEBUG MultiMatchBelow: 1 slaves merged into 1 buckets
2017-01-12 14:00:28 DEBUG +- bucket: mysql-bin.000011:6036419, 1 slaves
2017-01-12 14:00:28 DEBUG MultiMatchBelow: attempting slave rep3:3306 in bucket mysql-bin.000011:6036419
2017-01-12 14:00:28 INFO Will match rep3:3306 below rep1:3306
2017-01-12 14:00:28 DEBUG Stopping slave on rep3:3306
2017-01-12 14:00:28 INFO Stopped slave on rep3:3306, Self:mysql-bin.000001:18925723, Exec:mysql-bin.000011:6036419
2017-01-12 14:00:28 DEBUG PseudoGTIDPatternIsFixedSubstring: false
2017-01-12 14:00:28 DEBUG Compiling PseudoGTIDPattern
2017-01-12 14:00:28 DEBUG Searching for latest pseudo gtid entry in binlog mysql-bin.000001 of rep3:3306
2017-01-12 14:00:28 DEBUG getLastPseudoGTIDEntryInBinlog: starting with mysql-bin.000001:14067339
2017-01-12 14:00:29 DEBUG Found pseudo gtid entry in rep3:3306, mysql-bin.000001:18255502
2017-01-12 14:00:29 DEBUG PseudoGTIDPatternIsFixedSubstring: false
2017-01-12 14:00:29 DEBUG Compiling PseudoGTIDPattern
2017-01-12 14:00:29 DEBUG Searching for given pseudo gtid entry in rep1:3306. monotonicPseudoGTIDEntries=true
2017-01-12 14:00:29 DEBUG Searching for given pseudo gtid entry in binlog mysql-bin.000001 of rep1:3306
2017-01-12 14:00:29 DEBUG SearchEntryInBinlog: starting with mysql-bin.000001:292403
2017-01-12 14:00:29 DEBUG Matched ascending Pseudo-GTID entry in mysql-bin.000001
2017-01-12 14:00:29 DEBUG Matched entry in rep1:3306: mysql-bin.000001:303989
2017-01-12 14:00:29 DEBUG Returning 4185 events at mysql-bin.000001:18255502
2017-01-12 14:00:29 DEBUG Returning 30 events at mysql-bin.000001:303989
2017-01-12 14:00:29 DEBUG > mysql-bin.000001:18255502 Query; use `meta`; drop view if exists `meta`.`_pseudo_gtid_hint__asc:58777DD2:0000000000000232:6785F461`
2017-01-12 14:00:29 DEBUG < mysql-bin.000001:303989 Query; use `meta`; drop view if exists `meta`.`_pseudo_gtid_hint__asc:58777DD2:0000000000000232:6785F461`
2017-01-12 14:00:29 DEBUG > mysql-bin.000001:18255655 Query; BEGIN
2017-01-12 14:00:29 DEBUG < mysql-bin.000001:304150 Query; BEGIN
2017-01-12 14:00:29 DEBUG > mysql-bin.000001:18255718 Table_map; table_id: ### (test2.sbtest1)
2017-01-12 14:00:29 DEBUG < mysql-bin.000001:304230 Table_map; table_id: ### (meta.pseudo_gtid_status)
2017-01-12 14:00:29 ERROR Mismatching entries, aborting: table_id: ### (test2.sbtest1) <-> table_id: ### (meta.pseudo_gtid_status)
2017-01-12 14:00:29 INFO Started slave on rep3:3306
2017-01-12 14:00:29 DEBUG outdated keys: [rep2:3306]
2017-01-12 14:00:29 ERROR dial tcp 192.168.56.106:3306: getsockopt: connection refused
2017-01-12 14:00:29 ERROR ReadTopologyInstance(rep2:3306) show variables like 'maxscale%': dial tcp 192.168.56.106:3306: getsockopt: connection refused
2017-01-12 14:00:29 ERROR ReadTopologyInstance(rep2:3306) Cleanup: dial tcp 192.168.56.106:3306: getsockopt: connection refused
2017-01-12 14:00:29 WARNING instance is nil in discoverInstance. key=rep2:3306, error=Failed ReadTopologyInstance
2017-01-12 14:00:30 ERROR Unexpected: 0 events processed while iterating logs. Something went wrong; aborting. nextBinlogCoordinatesToMatch: <nil>
2017-01-12 14:00:30 DEBUG MultiMatchBelow: match result: <nil>, Unexpected: 0 events processed while iterating logs. Something went wrong; aborting. nextBinlogCoordinatesToMatch: <nil>
2017-01-12 14:00:30 ERROR Unexpected: 0 events processed while iterating logs. Something went wrong; aborting. nextBinlogCoordinatesToMatch: <nil>
2017-01-12 14:00:30 ERROR MultiMatchBelow: Cannot match up 1 slaves since their bucket mysql-bin.000011:6036419 is failed
2017-01-12 14:00:30 DEBUG auditType:end-maintenance instance:rep3:3306 cluster:rep1:3306 message:maintenanceToken: 305
2017-01-12 14:00:30 DEBUG auditType:relocate-slaves instance:rep2:3306 cluster:rep1:3306 message:relocated 0 slaves of rep2:3306 below rep1:3306
2017-01-12 14:00:30 ERROR topology_recovery: RecoverDeadIntermediateMaster failed to match up any slave from rep2:3306
from orchestrator.
Related Issues (20)
- Project dead? HOT 3
- DeadIntermediateMaster auto Recovery HOT 1
- Failure from hooks are not reflected on UI HOT 1
- Automated recovery invalid HOT 5
- Should set old master's read_only to false under DeadCoMaster failure
- Report IntermediateMaster errors under CoMaster deployment
- Set the old primary host read_only=0 if failover/switch over failed.
- Whether ORC provides APIs that enable/disable recovery for specific clusters? HOT 1
- Using Azure AD Authentication
- How to set autocommit for orchestrtor detection thread
- How do you perform system tests？ HOT 1
- Improve logging of problems during db connection
- Orchestrator able to display topology but still reports as failure
- After the Mysql domain name connection is configured。If the orchestrator topology displays both ip addresses and domain names of the same backup database, how can I fix this problem
- Discovery doesn't discover in extremely simple setup HOT 4
- Does orchestrator's website :3000 has login account?
- Does anyone know why updates stopped? HOT 2
- add windows version
- Ptrhon API Token
- How to login with username and password through API ?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from orchestrator.