Comments (2)
Example of degraded localtargets
from within etcd:
/ # etcdctl get '' --from-key
/skydns/com/example/cloud/failover/16fccd7b
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/24f6061f
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/6c871674
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/13ee1210
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/2485c7ba
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/315f5958
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
Notice the unexpected duplication of 172.17.0.5
At the same time related DNSEndpoint is fully ok:
k -n test-gslb get dnsendpoints -o yaml
....
- dnsName: failover.cloud.example.com
recordTTL: 30
recordType: A
targets:
- 172.17.0.2
- 172.17.0.3
- 172.17.0.5
So the problem seems to be fully isolated to upstream components of external-dns+etcd ... coredns reads whatever it has in etcd backend so it is behaving correctly.
Digging into crd_source -> external-dns -> etcd propagation code ...
from k8gb.
Made $ etcdctl del '' --from-key
effectively wiping out local etcd database
Then it was gslb reconcile
External-dns logs during the even :
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/4088e0a8 to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/461535c9 to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/57c491cc to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/5afed140 to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:32:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/68ec81be to Host=172.17.0.5, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.2, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.3, Text=, TTL=30"
time="2020-03-19T21:33:52Z" level=info msg="Add/set key /skydns/com/example/cloud/failover/localtargets/6a0a107c to Host=172.17.0.5, Text=, TTL=30"
So the first chunk of the log looks totally correct - each entry at different path and different hash at the end. Consequently we observe weird output of different values ending up at the same hash.
Surprisingly after the wipe out and reconcile it did fix itself:
# etcdctl get '' --from-key
/skydns/com/example/cloud/failover/4088e0a8
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/461535c9
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/68ec81be
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/57c491cc
{"host":"172.17.0.2","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/5afed140
{"host":"172.17.0.3","ttl":30,"targetstrip":1}
/skydns/com/example/cloud/failover/localtargets/6a0a107c
{"host":"172.17.0.5","ttl":30,"targetstrip":1}
No dups.
Something really weird happens here.
from k8gb.
Related Issues (20)
- K8GB for the service of Type ExternalName/Loadbalancer HOT 1
- Track the impact of HashiCorp license MPL -> BSL HOT 1
- HelmChart values validations
- Reverse proxy support? HOT 4
- 🟢 Revisit github workflow pipelines HOT 1
- 🟢 Revisit HelmChart
- 🟢 Revisit documentation
- 🟢 Release K8gb v1.0.0
- 🟢 Replace the old k8gb code with the new k8gb-lite code
- Push k8gb to OCP operator hub
- Security slam HOT 1
- Cluster can't return own entries
- document split brain scenario and the system behaviour HOT 2
- Create security self-assessment.md of k8gb and contribute it to CNCF Tag security
- Expand renovate to more codebase containing versions
- Please add yourself to `ADOPTERS.md`!
- "Additional property cloudflare is not allowed" appears when deploying Cloudflare integration HOT 2
- `TestFailoverPlayground ` flaky terratest
- Document the case of when a load balancing configuration is not deployed in every cluster
- Release pipeline fails with cosign validation
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from k8gb.