Originally ohmyglb operator was generated with operator-sdk version v0.12.0

There is no urgent need but eventually we might want to upgrade to latest v0.16.0 ( at the time of writing)

Upgrade path doc upstream: https://github.com/operator-framework/operator-sdk/blob/master/doc/migration/version-upgrade-guide.md

I would suggest we make it after implementation of strong e2e test suite to avoid wasting time on manual e2e regression testing

OhMyGLB was a silly name chosen for no particular reason when this project started.

Should we rename it now that it's maturing?

As per the supported load balancing strategies in the initial design a weighted round robin strategy should be implemented to ensure the guarantees stated:

Weighted round robin - Specialisation of the above (default round robin #45) strategy but where a percentage weighting is applied to determine which cluster's Ingress node IPs to resolve. E.g. 80% cluster X and 20% cluster Y

Scenario 1:

• Given 2 separate Kubernetes clusters, X, and Y
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on cluster X as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: roundRobin 
    weight: 80%

and a Gslb resource on cluster Y as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: roundRobin 
    weight: 20%

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 6 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 4
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 5
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 6
*   Trying 10.1.1.11...
...

The resolved node IP's that ingress traffic will be sent should be spread approximately according to the weighting configured on the Glsb resources. In this scenario that would be 80% (4 out of 6) resolved to cluster X and 20% (2 out of 6) resolved to cluster Y.

NOTE:

• The design of the specification around how to indicate the weighting as described in this issue is solely for the purpose of describing the scenario. It should not be considered a design.
• The scenario where there are more than 2 clusters is currently undefined. I.e. how do the weightings get distributed in the event of missing weightings or uneven weightings? E.g. Given 3 clusters but only 2 Gslb resources in 2 clusters have a weight specified (that might or might not add up to 100%). How does that affect the distribution over 3 clusters?
• Following on from the above, in the scenario where Deployments become unhealthy on a cluster, then the weighting should be adjusted to honour the weighting across the remaining clusters with healthy Deployments

This issue is a spin off from #47

Implement relevant metrics that provide insight into:

• Edge DNS integrations
• Split brain scenarios / heartbeat
• Failover scenarios
• Reconciliation
• Load balancing strategies used

#97 can be used as reference implementation

We currently didn't face any problems with the current setup but helm3 provides better crd installation support than just putting them under under templates

Refactor controller tests

• to AAA form
• split tests into functions

Follow standard template as shown below:

func TestReflectGeoTagInTheStatus(t *testing.T) {
 defer cleanup()
 init()
  //arrange
  //act
  //assert
}
func TestSomethingElse(t *testing.T) {
 defer cleanup()
 init()
  //arrange
  //act
  //assert
}
//this will be executed for each test that requires
func cleanup(){
   //cleaning env vars, resources etc
}
//this will be executed for each test that requires
func init(){
  // init reconcilers, clients etc.. whatever
}
//TestMain is reserved GO func. It says that: this will be executed only once for all tests. Don't use that if we don't need
func TestMain(m testing.M){
   //cleaning reasources for all tests i.e. remove cluster, or cloud resources etc... 
   defer TearDown()
   //prepare shared resources for all tests. i.e. create test cluster,  queue etc..
   InitForAllTests()
   t.Main()
}

What we have seen in Infoblox that we have a zone ohmyglb.test.good.org

if we have a delegated zone mynewglb "mynewglb.ohmyglb.test.good.org" that gets created with the Solution see that it creates duplicate zones

pne in the parent zone inside "test.good.org" as a full record inside the test.good.org as mynewglb.ohmyglb inside test.good.org and not below the ohmyglb.test.goo.org as new delegated Zone.

During on-prem tests we encountered a situation during EtcdCluster creation with etcd-operator is not picking up the overridden private registry image reference and still tries to pull quay.io/coreos/etcd during cluster creation.

In one cluster it 'fixed' itself automatically by redeployment in another cluster the issue persisted so we had to manually patch EtcdCluster resource.

Definitely worth investigation as it can complicate on-prem installations.

Also we need to figure out if the problem affects only 1.14 cluster deployment workaround ( see https://github.com/AbsaOSS/ohmyglb/blob/master/Makefile#L114 ) or it is also a case for standard stable deployment target ( https://github.com/AbsaOSS/ohmyglb/blob/master/Makefile#L104 )

Currently k8gb fully relies on underlying Ingress that it controls. It has limitations of exclusive http/https support so services on another ports and another protocols are not covered by Gslb operations.
We can change that by enabling k8gb to control underlying Service of type LoadBalancer. It will create mechanism to expose any tcp/udp service on any port and protocol. We definitely should keep current Ingress operations for L7 http/https .

Service enabled spec might look like:

apiVersion: k8gb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: test-gslb
  namespace: test-gslb
spec:
  loadBalancer:
    host: app1.cloud.example.com
    serviceName: app # Service with type: LoadBalancer
  strategy:
    type: roundRobin

We encountered an issue with Gslb.Status updated after scaling down sample workload and bringing it back. It took unusual amount of time for ohmyglb to pick up recent service health status correctly, also we observed json unmarshall error in the logs

Cannot unmarshall to check empty value '', err: 'unexpected end of JSON input'

It worth to mention that actual ohmyglb operations were not affected - DNSEndpoints were updated and associated DNS responses were constructed properly as expected

Most of the current use cases, automated tests and functional test scenarios were performed in context of 2 clusters only as it is most common and easier to test setup.

Meanwhile we need to be ready and test for >2 scenario cluster scenario.

This series of testing can reveal potential edge cases in current LB strategies implementation and can lead to LB logic extension and handling of >2 number of clusters

I've been trying to install this chart following the instructions in the helm 3 section of the readme, but I'm stuck waiting on etcd pods that won't get past the "pending" status.
Any ideas?

Here are the steps I took:

1. Follow instructions to add the ohmyglb repo
2. Create ohmyglb namespace. Installing the chart doesn't create it and will fail saying it can't find a "ohmyglb" namespace so I manually created it. Seems like this is expected behavior in Helm 3 (Correct me if I'm wrong).
3. Run: helm install --debug ohmyglb ohmyglb/ohmyglb

At this point most of the pods are deployed, but the following etcd pods will get stuck in "pending".

Kubectl get pods output:

NAMESPACE       NAME                                                           READY   STATUS             RESTARTS   AGE
default         ohmyglb-coredns-6b7c98d545-hh8kh                               1/1     Running            0          13m
default         ohmyglb-etcd-operator-etcd-backup-operator-7b7765f455-28m5j    1/1     Running            0          13m
default         ohmyglb-etcd-operator-etcd-operator-5d69b5449d-vktx9           0/1     Pending            0          13m
default         ohmyglb-etcd-operator-etcd-restore-operator-5ccc8fccf5-8fqgg   0/1     Pending            0          13m
ohmyglb         external-dns-5844f58797-pdshk                                  1/1     Running            0          13m
ohmyglb         ohmyglb-7c5d899559-zbwlp                                       1/1     Running            0          13m

ohmyglb-7c5d899559-zbwlp logs:

{"level":"info","ts":1588015395.9359705,"logger":"cmd","msg":"Operator Version: 0.5.6"}
{"level":"info","ts":1588015395.9365296,"logger":"cmd","msg":"Go Version: go1.14"}
{"level":"info","ts":1588015395.9368103,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"}
{"level":"info","ts":1588015395.937081,"logger":"cmd","msg":"Version of operator-sdk: v0.16.0"}
{"level":"info","ts":1588015395.9375331,"logger":"leader","msg":"Trying to become the leader."}
{"level":"info","ts":1588015396.6210973,"logger":"leader","msg":"No pre-existing lock was found."}
{"level":"info","ts":1588015396.6304686,"logger":"leader","msg":"Became the leader."}
{"level":"info","ts":1588015397.2903323,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":"0.0.0.0:8383"}
{"level":"info","ts":1588015397.2911112,"logger":"cmd","msg":"Registering Components."}
{"level":"info","ts":1588015399.4246273,"logger":"metrics","msg":"Metrics Service object created","Service.Name":"ohmyglb-metrics","Service.Namespace":"ohmyglb"}
{"level":"info","ts":1588015400.085832,"logger":"cmd","msg":"Could not create ServiceMonitor object","error":"an empty namespace may not be set during creation"}
{"level":"info","ts":1588015400.0861387,"logger":"cmd","msg":"Starting the Cmd."}
{"level":"info","ts":1588015400.0868874,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"gslb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1588015400.0877016,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"gslb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1588015400.088064,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"gslb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1588015400.0884078,"logger":"controller-runtime.controller","msg":"Starting EventSource","controller":"gslb-controller","source":"kind source: /, Kind="}
{"level":"info","ts":1588015400.0889385,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"}
{"level":"info","ts":1588015400.1968875,"logger":"controller-runtime.controller","msg":"Starting Controller","controller":"gslb-controller"}
{"level":"info","ts":1588015400.199356,"logger":"controller-runtime.controller","msg":"Starting workers","controller":"gslb-controller","worker count":1}

Some info that might be useful:
Running in a MacBook Pro
helm v3.2.0
docker desktop v2.2.0.5 (engine v19.03.8 and k8s v1.15.5)

As followup to
#97 (comment),
let's create sample Graphana dashboard showing off exposed prometheus metrics.
Dashboard can be uploaded with some sample data here:
https://grafana.com/grafana/dashboards?orderBy=name&direction=asc

In addition to the default Infoblox support, Route 53 should be added as an additional edge DNS option.

This should include examples and documentation on the usage and configuration of this option.

We already have all mechanics and automation to deploy ohmyglb locally(and also remotely) with some globally loadbalanced application on top. For local testing scenario we are using https://github.com/stefanprodan/podinfo as a sample application.

In context of this issue we want to extend README with end-to-end Howto which would include:

• Local 2-cluster ohmyglb setup
• Deployment of sample application workload on top ( podinfo )
• Demonstration of currently implemented RoundRobin and Failover strategies

We also might want to configure/modify podinfo to expose ohmyglb GeoTag right in the user interface so it will be highly illustrative in context of loadbalancing strategies demonstration in action.

We should implement the necessary tooling and infrastructure to be able to publish the Helm chart to various chart repositories when a release is tagged.

As a start we should support the following repositories:

• via GitHub Pages (see references below)
• via JFrog Artifactory (see references below)

References:

GitHub Pages

• https://helm.sh/docs/topics/chart_repository/#github-pages-example
• https://github.com/helm/hub/blob/master/docs/github.md#github-pages-as-a-repository

JFrog Artifactory

• https://helm.sh/docs/topics/chart_repository/#jfrog-artifactory
• https://www.jfrog.com/confluence/display/JFROG/Helm+Chart+Repositories

In preparation for the 1.0 release, complete the following items:

• API stabilisation
• Add full documentation
• Publish to OperatorHub

All documentation and publishing items above should have the corresponding infrastructure/pipelines implemented for automated publishing etc.

There is number of internal timeouts that currently provided as default hardcoded values within Gslb:

• DNS TTL of 30s
• Reschedule of Reconcile loop to pickup external Gslb targets and Zone Delegation api kick (30s)
• External-dns sync interval to update etcd backend of coredns (20s)

We need to make them configurable via CR spec (DNS TTL would probably go there) and operator deployment configuration ( Reconcile loop and ext-dns values)

Main use case is TLS enabled ingress and cert-manager integration

Makefile targets are dependent on full.sh and environment variables declared at the same file.

• refactor Makefile to GNU call Functions to get rid of dependencies on full.sh.
• refactor all places where full.sh is used (i.e. terratest build pipe)

Given the five phases of the Operator Maturity Model:

1. Basic Install
2. Seamless Upgrades
3. Full Lifecycle
4. Deep Insights
5. Auto Pilot

Consider all phases and implement the necessary tasks to achieve the desired maturity phase.

There is a possible issue that we need to have Separate DNS Zones for different Datacenters to cater for people who can only run Active Passive and then also hvae the GLB

Example is DC1-us.k8gb.gb and DC2-us.k8gb.gb then also DC.k8gb.gb

The request is to add support to host multiple DNS Zones in K8GB and not just one zone

• We are relying on operator namespace to be named 'k8gb' in the helm chart templates and in the codebase
• It is not really flexible and goes against https://github.com/operator-framework/community-operators/blob/master/docs/best-practices.md
• To fix that we might want to use the approach similar to https://github.com/helm/charts/pull/22378/files

The problem statement is as follows:

1. Can ohmyglb be built in such a way it would peer with a routable network using either bgp (Border Gateway Protocols)
2. Could you use the AS Number Private Ranges or an ospf Area config
3. Can you use authenticationon either BGP or Ospf to make sure it can be secured for only peer to peer session establishment
4. Can you use an ACL which can be used to present network ip's via the above two routing protocols
5. I am asking if the project feature request can be possibly looked at for this as this would mean that you could be location agnostic

We already made some experiments with k3d which looked very good in terms of speed and performance, the only visible blocker was inability to have the test on the same network as in
k3d-io/k3d#111

Looks like the issue is solved in recent k3d version and we might give it another shot as we definitely want faster test cycle both locally and within ci/cd pipelines

gosec started to fail with

[/github/workspace/pkg/apis/ohmyglb/v1beta1/zz_generated.deepcopy.go:113] - G601 (CWE-): Implicit memory aliasing in for loop. (Confidence: MEDIUM, Severity: MEDIUM)
  > &val


Summary:
   Files: 20
   Lines: 1969
   Nosec: 0
  Issues: 1

on generated code.

Does not look super critical but worth tracking.

I've unblocked pipeline by sticking to stable v2.2.0 release at #114

As per the supported load balancing strategies in the initial design a failover strategy should be implemented to ensure the guarantees stated:

Failover - Pinned to a specified primary cluster until that cluster has no available Pods, upon which the next available cluster's Ingress node IPs will be resolved. When Pods are again available on the primary cluster, the primary cluster will once again be the only eligible cluster for which cluster Ingress node IPs will be resolved

Scenario 1:

• Given 2 separate Kubernetes clusters, X, and Y
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on all 2 clusters as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: failover 
    primary: cluster-x

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 3 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.0.1.10...
...

The resolved node IP's that ingress traffic will be sent should be "pinned" to the primary cluster named explicitly in the Gslb resource above, even though there was a healthy Deployment in cluster Y, the Ingress node IPs for cluster Y would not be resolved.

Scenario 2:

• Same configuration as Scenario 1 except that the Deployment only has healthy Pods on one cluster, cluster Y. I.e. The Deployment on cluster X has no healthy Pods.

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 3 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.1.1.11...
...

In this scenario, only Ingress node IPs for cluster Y are resolved given that there is not a healthy Deployment for the Gslb host on the primary cluster, cluster X. Therefore, the "failover" cluster(s) are resolved instead (cluster Y in this scenario).

Now, given that the Deployment on cluster X (the primary cluster) now becomes healthy once again, I would expect the IP's resolved to reflect as follows (if this command was executed 2 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.0.1.10...
...

The primary cluster's Ingress node IPs are now resolved exclusively once again.

NOTE:

• The design of the specification around how to indicate the primary cluster as described in this issue is solely for the purpose of describing the scenario. It should not be considered a design.
• The existence of multiple "secondary" failover clusters should also be considered. For example, if there were 3 clusters (X, Y and Z) in the scenario 2 above, could the Ingress node IPs for both clusters (X and Z) be resolved and if so, how (in terms of "load balancing") would the Ingress node IPs across both those secondary/failover clusters be resolved? Would they use the default round robin strategy, if any strategy at all?

• Currently in case when service behind Gslb is down we hit downtime which is roughly equal to DNS TTL ( defaults to 30s) - that what it takes to propagate associated fqdn with new IP values
• During this downtime nginx ingress controller returns 504 http error code
• Instead of responding 504 we might send redirects with 302 to healthy cluster on Layer 7
• Looks like we can use https://itnext.io/custom-error-page-for-nginx-ingress-controller-cca4cfa82bb9 for that
• It will mitigate the DNS update based down time for most of the cases except of whole k8s cluster down which is supposed to be a rare thing to happen

Currently we have Gslb Status content as shown in example below:

  status:
    healthyRecords:
      app3.cloud.example.com:
      - 172.17.0.4
      - 172.17.0.5
      - 172.17.0.6
    managedHosts:
    - app1.cloud.example.com
    - app2.cloud.example.com
    - app3.cloud.example.com
    serviceHealth:
      app1.cloud.example.com: NotFound
      app2.cloud.example.com: Unhealthy
      app3.cloud.example.com: Healthy

It is easily visible that we have duplication between the managedHosts and serviceHealth
It happened historically during development of the project. Initially serviceHealth looked like

    serviceHealth:
      unhealthyServiceName: NotFound
      backend: Unhealthy
      frontend: Healthy

so basically referenced serviceName that is referenced in associated gslb ingress.
Later on it was changed to ingressHost:Status as it became apparent that it is practical to have them in a single data structure.

So some questions to discuss:

1. Do we need serviceName exposed somewhere in Status ?
2. If yes - should we extend serviceHealth or place it somewhere else ?
3. Easiest scenario - we just remove managedHosts

Great work guys... looking very promising. Give the team a hug and a donut from me.

This issue is a spin off from #47

Implement relevant tracing that provide insight into:

• Edge DNS integrations
• Split brain scenarios
• Failover
• Reconciliation
• Load balancing strategies used

Current consideration is to use OpenTracing Go API https://opentracing.io/guides/golang/.
This implies manual code instrumentation unless more non-intrusive solution is found.

This would allow in the use case where you may have more than 1 DNS server to query (for redundancy)

edgeDNSServer: &edgeDNSServer:
  - 1.1.1.1
  - 2.2.2.2 

High level feature to add support for all major public clouds:

• AWS ✅
• GCP ( #641 )
• Azure ( #642 )
• Digital Ocean

Every public cloud above represents tasks related to:

• Edge DNS solutions for each cloud DNS provider
• Examples and documentation for the specified cloud
• Additional integration tests including the specified cloud in the pipelines

As per the supported load balancing strategies in the initial design a manual strategy should be implemented to ensure the guarantees stated:

Manual - Eligibility is manually specified as to which cluster(s) are eligible. If there are no available Pods in the specified clusters, then no cluster Ingress node IPs will be resolved and the client will get a NXDOMAIN response

Scenario 1:

• Given 2 separate Kubernetes clusters, X, and Y
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on all 2 clusters as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: manual 
    clusters:
      - cluster-x

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 3 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.0.1.10...
...

The resolved node IP's will always be those of cluster X. Even if cluster X has no healthy Deployments and cluster Y does, the NXDOMAIN response will be returned regardless. This strategy allows full manual control over which cluster's Ingress node IPs can be resolved.

NOTE:

• The design of the specification around how to indicate the manual cluster preference as described in this issue is solely for the purpose of describing the scenario. It should not be considered a design.
• The scenario where there are more than 1 clusters added to the list of manually included clusters introduces the question around how (in terms of "load balancing") the Ingress node IPs of those clusters get resolved. Would they use the default round robin strategy, if any strategy at all?

We need to create contribution guide which would capture development topics like

• Useful development make targets
• unit testing approach
• end-to-end testing approach

This chunk of documentation will be useful both for us and external contributors.

#91 implements end-to-end testing of most of ohmyglb functionality with exception of EdgeDNS part which includes

• Zone Delegation
• Associated split brain handling.

As it is very special part of the codebase and very environment specific (Infoblox, Route53, ...) we need to address it separately from the rest of terratest e2e pipeline.

Steps to reproduce

• Deploy two cross-communicating ohmyglb setup locally
  $ make deploy-full-local-setup
• Check generated localtargets.* dnsendpoint conf

$ kubectl -n test-gslb get dnsendpoints test-gslb -o yaml
...
spec:
  endpoints:
  - dnsName: localtargets.app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.4
...

• Check if coredns returns matching A records

dig +short @localhost localtargets.app3.cloud.example.com
172.17.0.2
172.17.0.4
172.17.0.3

This is expected result. After some time localtargets.* can 'lose' one of the records in the following way:

• localtargets.* dnsendpoint conf is always consistent

$ kubectl -n test-gslb get dnsendpoints test-gslb -o yaml
...
spec:
  endpoints:
  - dnsName: localtargets.app3.cloud.example.com
    recordTTL: 30
    recordType: A
    targets:
    - 172.17.0.2
    - 172.17.0.3
    - 172.17.0.4
...

• Meanwhile actual DNS response might lose one of the A records

dig +short @localhost localtargets.app3.cloud.example.com
172.17.0.2
172.17.0.4

Issue is not really deterministic in its behaviour . Meanwhile we faced it several times over multiple deployments
In case of 2 cluster setup only single cluster is affected effectively making exposed through coredns only 5 out of 6 k8s worker.

DNSEndpoint CR generation looks always correct so the problem is somewhere in etcd coredns backend area.

make debug-test-etcd can help in debugging this issue runtime.

When we have mismatch between https://github.com/AbsaOSS/k8gb/blob/master/chart/k8gb/values.yaml#L11 and Gslb Ingress host like https://github.com/AbsaOSS/k8gb/blob/master/deploy/crds/k8gb.absa.oss_v1beta1_gslb_cr_failover.yaml#L9 we are going to send non-valid configuration to Infoblox/EdgeDNS

We probably want to:

• Detect the mismatch by k8gb operator
• Circuit break Infoblox API calls
• Clearly reflect the fact of misconfiguration in the main k8gb log stream

Existing build pipelines should be enhanced with full end to end integration tests running in managed Kubernetes clusters.

The goal of these tests would be to simulate:

• real world scenarios deploying to multiple managed Kubernete clusters (potentially across different clouds)
• exercise tests of the various load balancing strategies
• simulate the various failure scenarios including split brain examples
• verify the edge DNS integrations across the supported providers

These tests should be triggered at strategic and meaningful points in the development flow, considering that this test suite could take considerable time to run to completion.

We have non-deterministic test case within terratest pipeline which is failing from time to time

2020-04-27T22:57:56.9670279Z TestOhmyglbBasicAppExample 2020-04-27T22:57:56Z command.go:158: 'map[app1.cloud.example.com:NotFound app2.cloud.example.com:Unhealthy app3.cloud.example.com:Unhealthy]'
2020-04-27T22:57:56.9670560Z     TestOhmyglbBasicAppExample: ohmyglb_basic_app_test.go:100: 
2020-04-27T22:57:56.9670722Z         	Error Trace:	ohmyglb_basic_app_test.go:100
2020-04-27T22:57:56.9670841Z         	Error:      	Not equal: 
2020-04-27T22:57:56.9671273Z         	            	expected: "'map[app1.cloud.example.com:NotFound app2.cloud.example.com:Unhealthy app3.cloud.example.com:Unhealthy]'"
2020-04-27T22:57:56.9671659Z         	            	actual  : "'map[app1.cloud.example.com:NotFound app2.cloud.example.com:Unhealthy app3.cloud.example.com:Healthy]'"
2020-04-27T22:57:56.9671778Z         	            	
2020-04-27T22:57:56.9671897Z         	            	Diff:
2020-04-27T22:57:56.9672149Z         	            	--- Expected
2020-04-27T22:57:56.9672263Z         	            	+++ Actual
2020-04-27T22:57:56.9672512Z         	            	@@ -1 +1 @@
2020-04-27T22:57:56.9672864Z         	            	-'map[app1.cloud.example.com:NotFound app2.cloud.example.com:Unhealthy app3.cloud.example.com:Unhealthy]'
2020-04-27T22:57:56.9673213Z         	            	+'map[app1.cloud.example.com:NotFound app2.cloud.example.com:Unhealthy app3.cloud.example.com:Healthy]'
2020-04-27T22:57:56.9673353Z         	Test:       	TestOhmyglbBasicAppExample
2020-04-27T22:57:56.9673806Z TestOhmyglbBasicAppExample 2020-04-27T22:57:56Z command.go:87: Running command kubectl with args [--namespace ohmyglb-test-n68arr delete -f 

We already have great design architecture docs in place, now we need to amend them with specific implementation details depicting open source components we are using to solve specific tasks, e.g.

• CoreDns to respond to DNS queries
• Dedicated ETCD to act as CoreDNS backend
• external DNS as a glue between DNSEndpoint CRDs and CoreDNS configuration
• Set of CRDs and k8s primitives that involved in k8gb operations

Fancy diagrams are welcomed.

Currently we are solving potential splitbrain situation of zone delegation configuration by putting timestamp as TXT to heartbeat dns record in edgeDNS zone. Implementation details are in #44

During e2e tests we realized that TTL for this TXT record is implicitly inherited by dns zone configuration of Infoblox.
When TTL is longer then we have an obvious problem with liveness check incorrectness.
We need to find a reliable way to control TTL of edgeDNS splitbrain TXT records.

Current hypo is to use special API call of TTL update

It will also require another upstream modification of https://github.com/infobloxopen/infoblox-go-client as it does not support TTL handling.

Issue is still workaroundable by setting low TTL in edgeDNSzone configuration in Infoblox

Given someone that is looking for a cloud native, Kubernetes native GSLB solution and who finds this project. The README should concisely and simply answer the following questions:

• What problem does it solve?
• How does it solve the problem? (Briefly, simply and with links to detailed docs)
• How do I run it locally to see how it works (Quick start)
• If I wanted to contribute, what should I know and how could I setup my local development environment (see #100)

The goal of the README should leave no doubt as to what this project does, how it does it and an easy, quick way to try it out.

It's somehow hard to reproduce but we periodically facing

time="2020-01-29T17:08:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.4;172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:08:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:08:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:09:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.2;172.17.0.4;172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:09:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:09:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6904e7b4 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:10:08Z" level=debug msg="Skipping endpoint localtargets.app3.cloud.example.com 30 IN A  172.17.0.5 [] because owner id does not match, found: \"\", required: \"\"ohmyglb\"\""

Recreation of gslb CR fixes the issue

time="2020-01-29T17:10:08Z" level=info msg="Delete key /skydns/com/example/cloud/app3/6904e7b4"
time="2020-01-29T17:10:08Z" level=info msg="Delete key /skydns/com/example/cloud/app3"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/6d89b98f to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/483cc44e to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/732d34f7 to Host=172.17.0.5, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/03acc020 to Host=172.17.0.2, Text=\"heritage=external-dns,external-dns/owner=\"ohmyglb\",external-dns/resource=crd/test-gslb/test-gslb\", TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/4b25072e to Host=172.17.0.4, Text=, TTL=30"
time="2020-01-29T17:13:08Z" level=info msg="Add/set key /skydns/com/example/cloud/app3/localtargets/4fe76c50 to Host=172.17.0.5, Text=, TTL=30"

Looks like a race condition somewhere between external-dns / etcd backend population of local coredns

Spin off from #47
OhMyGLB supports integration with Prometheus Operator thanks to Operator SDK native support.
However, actual procedure of Prometheus Operator setup for proper scraping is not clear, so it makes sense to reflect it in OhMyGLB documentation.

We lint project from /pkg/controller folder due to invalid /pkg/apis/ohmyglb/v1beta1/zz_generated.openapi.go. Generated file is obsolete and functions within are not used

• remove obsolete zz_generated.openapi.go
• set linter to project root

The default round robin load balancing strategy should be implemented in a way that gives consistent results when resolving a Gslb host.

ℹ️ Load balancing in the context of this feature corresponds to the distribution of resolved IP's and does not refer in any way to the actual balancing of network traffic

Scenario:

• Given 3 separate Kubernetes clusters, X, Y and Z.
• Each cluster has a healthy Deployment with a backend Service called app and that backend service exposed with a Gslb resource on all 3 clusters as:

apiVersion: ohmyglb.absa.oss/v1beta1
kind: Gslb
metadata:
  name: app-gslb
  namespace: test-gslb
spec:
  ingress:
    rules:
      - host: app.cloud.example.com
        http:
          paths:
            - backend:
                serviceName: app
                servicePort: http
              path: /
  strategy: roundRobin 

• Each cluster has one worker node that accepts Ingress traffic. The worker node in each cluster has the following name and IP:

cluster-x-worker-1: 10.0.1.10
cluster-y-worker-1: 10.1.1.11
cluster-z-worker-1: 10.2.1.12

When issuing the following command, curl -v http://app.cloud.example.com, I would expect the IP's resolved to reflect as follows (if this command was executed 6 times consecutively):

$ curl -v http://app.cloud.example.com # execution 1
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 2
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 3
*   Trying 10.2.1.12...
...

$ curl -v http://app.cloud.example.com # execution 4
*   Trying 10.0.1.10...
...

$ curl -v http://app.cloud.example.com # execution 5
*   Trying 10.1.1.11...
...

$ curl -v http://app.cloud.example.com # execution 6
*   Trying 10.2.1.12...
...

As above, the resolved node IP's that ingress traffic will be sent should be evenly "load balanced" between the clusters.

NOTE:

• Whether or not the node IP's for each cluster should be "load balanced" is dependant on the additional complexity. The main goal of this feature is to "load balance" across clusters in a consistent way, hence the additional balancing of the actual node IP's within each cluster is optional.
• This feature should ignore any notion of a weighted balancing initially. However, provision for this feature should be taken into account with the design.

Implement relevant metrics that provide insight into:

• Edge DNS integrations
• Split brain scenarios / heartbeat
• Reconciliation
• Load balancing strategies
• etc.

Metrics should be expressed as Prometheus text exposition format.

Consider the following CoreDNS plugins as reference:

• https://coredns.io/plugins/metrics/
• https://coredns.io/plugins/trace/

Related to this note on #46:

The existence of multiple "secondary" failover clusters should also be considered. For example, if there were 3 clusters (X, Y and Z) in the scenario 2 above, could the Ingress node IPs for both clusters (X and Z) be resolved and if so, how (in terms of "load balancing") would the Ingress node IPs across both those secondary/failover clusters be resolved? Would they use the default round robin strategy, if any strategy at all?

we need to validate that the IPs resolved from the secondary/failover clusters are handled correctlty and consistently in line with a load balancing strategy of their own.

there are several places in the code where we directly read input configuration (i.e. strategy.type,...)
Exclude GeoTag from refatoring please...

Move all inputs into depresolver and

• validate
• cover by tests
• provide predefined value if necessary