eumel8 / otc-rds-operator Goto Github PK

8.0 2.0 0.0 368 KB

RDS Operator for Open Telekom Cloud

License: MIT License

Dockerfile 0.44% Makefile 1.45% Go 96.29% Smarty 1.41% Shell 0.42%

go golang kubernetes kubernetes-operator opentelekomcloud operator otc rds rds-database

otc-rds-operator's Issues

Extend ServiceMonitor metrics

within github.com/gotway/gotway/pkg/metrics the Operator can serve metrics for Prometheus, which works for Kubernetes standard metrics.

New:
Number of managed RDS (sum, state ACTIVE/FAULTY/other state)

Ref:
https://prometheus.io/docs/guides/go-application/
https://github.com/prometheus-operator/prometheus-operator/blob/main/pkg/operator/operator.go#L152
https://github.com/eumel8/otc-rds-operator/blob/master/cmd/main.go#L85
https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/internal/controller/metrics/metrics.go

Handling of database user and permission is not part of OTC API. only root user will be created on installation.
If we assume the Kubernetes Cluster has access to the RDS instance we can create user/databases from Operator.
There are enough standard workflows for that.
CRD needs to be advanced:

databases:
  - abc
  - cde
users:
  - name: uvw
     password: xxxx
  - name: xyz
     password: xxxx
privileges:
      - "GRANT SELECT ON abc.* TO 'xyz'@'10.0.0.0/24"

hint: Mysql? Postgresql?
password secret handling?

Gophertelekomcloud Alarmrule changed

opentelekomcloud/gophertelekomcloud#381

go: finding module for package github.com/opentelekomcloud/gophertelekomcloud/openstack/cloudeyeservice/alarmrule
github.com/eumel8/otc-rds-operator/pkg/controller imports
	github.com/opentelekomcloud/gophertelekomcloud/openstack/cloudeyeservice/alarmrule: module github.com/opentelekomcloud/gophertelekomcloud@latest found (v0.5.23), but does not contain package github.com/opentelekomcloud/gophertelekomcloud/openstack/cloudeyeservice/alarmrule

Fix Helm Chart SAST Report

SAST reported some minor security issues. This needs to fix.

Upgrade from 0.3.1 to 0.5.0 failed.

after updating from 0.3.1 to 0.5.0 the pods of the deployment are in a CrashLoopBackoff:

{"level":"info","msg":"Sending events to apis","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:29Z"}
{"level":"info","msg":"starting HA controller","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:29Z","type":"runner"}
{"level":"info","msg":"metrics server listening in /metrics:2112","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:29Z","type":"metrics"}
I0624 11:31:29.150164       1 leaderelection.go:248] attempting to acquire leader lease rdsoperator/rdsoperator-otc-rds-operator...
{"level":"info","msg":"leader elected: 'rdsoperator-otc-rds-operator-d6447964f-v2htt'","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:29Z","type":"runner"}
I0624 11:31:46.580398       1 leaderelection.go:258] successfully acquired lease rdsoperator/rdsoperator-otc-rds-operator
{"level":"info","msg":"obtained leadership","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"runner"}
{"level":"info","msg":"start leading","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"runner"}
{"level":"info","msg":"starting controller","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
{"level":"info","msg":"starting informers","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
{"level":"info","msg":"waiting for informer caches to sync","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
{"level":"info","msg":"starting 4 workers","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
{"level":"info","msg":"controller ready","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
{"level":"info","msg":"starting smn listener","node":"rdsoperator-otc-rds-operator-d6447964f-skzdr","service":"otc-rds-operator","time":"2022-06-24T11:31:46Z","type":"controller"}
I0624 11:31:47.907513       1 event.go:294] "Event occurred" object="rdsoperator/my-rds" kind="Rds" apiVersion="otc.mcsps.de/v1alpha1" type="Normal" reason="Create" message="This instance is creating."
E0624 11:31:56.944103       1 runtime.go:78] Observed a panic: "invalid memory address or nil pointer dereference" (runtime error: invalid memory address or nil pointer dereference)
goroutine 75 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x1500960?, 0x238c830})
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:74 +0x86
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0005130b0?})
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:48 +0x75
panic({0x1500960, 0x238c830})
	/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).rdsUpdateStatus(0x16e2f2b?, {0x18f8280, 0xc0000b6b00}, 0x16e2f2b?, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/resource.go:670 +0x1ba
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).UpdateStatus(0xc0004b0300, {0x18f8280, 0xc0000b6b00}, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/resource.go:994 +0x1f6
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processUpdateRds(0xc0004b0300, {0x18f8280, 0xc0000b6b00}, 0xc0002d4000, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/worker.go:78 +0x59
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processEvent(0x0?, {0x18f8280?, 0xc0000b6b00?}, {0x1598480?, 0xc0005130b0?})
	/go/src/otc-rds-operator/pkg/controller/worker.go:55 +0x1ae
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processNextItem(0xc0004b0300, {0x18f8280, 0xc0000b6b00})
	/go/src/otc-rds-operator/pkg/controller/worker.go:27 +0xfb
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).runWorker(...)
	/go/src/otc-rds-operator/pkg/controller/worker.go:16
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).Run.func1()
	/go/src/otc-rds-operator/pkg/controller/controller.go:56 +0x46
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x18e5060, 0xc00040fbc0}, 0x1, 0xc000115620)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).Run
	/go/src/otc-rds-operator/pkg/controller/controller.go:55 +0x4d2
panic: runtime error: invalid memory address or nil pointer dereference [recovered]
	panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x30 pc=0x135e89a]

goroutine 75 [running]:
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc0005130b0?})
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/runtime/runtime.go:55 +0xd8
panic({0x1500960, 0x238c830})
	/usr/local/go/src/runtime/panic.go:838 +0x207
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).rdsUpdateStatus(0x16e2f2b?, {0x18f8280, 0xc0000b6b00}, 0x16e2f2b?, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/resource.go:670 +0x1ba
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).UpdateStatus(0xc0004b0300, {0x18f8280, 0xc0000b6b00}, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/resource.go:994 +0x1f6
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processUpdateRds(0xc0004b0300, {0x18f8280, 0xc0000b6b00}, 0xc0002d4000, 0xc0002d42c0)
	/go/src/otc-rds-operator/pkg/controller/worker.go:78 +0x59
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processEvent(0x0?, {0x18f8280?, 0xc0000b6b00?}, {0x1598480?, 0xc0005130b0?})
	/go/src/otc-rds-operator/pkg/controller/worker.go:55 +0x1ae
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).processNextItem(0xc0004b0300, {0x18f8280, 0xc0000b6b00})
	/go/src/otc-rds-operator/pkg/controller/worker.go:27 +0xfb
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).runWorker(...)
	/go/src/otc-rds-operator/pkg/controller/worker.go:16
github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).Run.func1()
	/go/src/otc-rds-operator/pkg/controller/controller.go:56 +0x46
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1(0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:155 +0x3e
k8s.io/apimachinery/pkg/util/wait.BackoffUntil(0x0?, {0x18e5060, 0xc00040fbc0}, 0x1, 0xc000115620)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:156 +0xb6
k8s.io/apimachinery/pkg/util/wait.JitterUntil(0x0?, 0x3b9aca00, 0x0, 0x0?, 0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:133 +0x89
k8s.io/apimachinery/pkg/util/wait.Until(0x0?, 0x0?, 0x0?)
	/go/src/otc-rds-operator/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:90 +0x25
created by github.com/eumel8/otc-rds-operator/pkg/controller.(*Controller).Run
	/go/src/otc-rds-operator/pkg/controller/controller.go:55 +0x4d2

Level 5 approach: React on RDS usage and scale

With OTC Cloud Eye and SMN are services available to skip the Operator Level 4 approach and go directly to auto pilot.

generate alert rules in Cloud Eye for storage and cpu usage
generate SMN and fire webhook to an https service
scale storage or flavor on event. Storage has his limits (steps to 10GB up to 4000), Flavor

$ openstack rds flavor list mysql 8.0 | grep ha
+----------------------------+---------------+-------+-----+
| name                       | instance_mode | vcpus | ram |
+----------------------------+---------------+-------+-----+
| rds.mysql.s1.medium.ha     | ha            | 1     |   4 |
| rds.mysql.c2.medium.ha     | ha            | 1     |   2 |
| rds.mysql.m1.large.ha      | ha            | 2     |  16 |
| rds.mysql.m1.xlarge.ha     | ha            | 4     |  32 |
| rds.mysql.m1.2xlarge.ha    | ha            | 8     |  64 |
| rds.mysql.m1.4xlarge.ha    | ha            | 16    | 128 |
| rds.mysql.m1.8xlarge.ha    | ha            | 32    | 256 |
| rds.mysql.m3.15xlarge.8.ha | ha            | 60    | 512 |
| rds.mysql.s1.large.ha      | ha            | 2     |   8 |
| rds.mysql.s1.xlarge.ha     | ha            | 4     |  16 |
| rds.mysql.s1.2xlarge.ha    | ha            | 8     |  32 |
| rds.mysql.c2.large.ha      | ha            | 2     |   4 |
| rds.mysql.c2.xlarge.ha     | ha            | 4     |   8 |
| rds.mysql.c3.15xlarge.4.ha | ha            | 60    | 256 |
| rds.mysql.c2.2xlarge.ha    | ha            | 8     |  16 |
| rds.mysql.c2.4xlarge.ha    | ha            | 16    |  32 |
| rds.mysql.s1.4xlarge.ha    | ha            | 16    |  64 |
| rds.mysql.c2.8xlarge.ha    | ha            | 32    |  64 |
| rds.mysql.s1.8xlarge.ha    | ha            | 32    | 128 |
| rds.mysql.c3.15xlarge.2.ha | ha            | 60    | 128 |

extra hint not all flavors are on all AZ available:

                "az_status": {
                    "eu-de-01": "normal",
                    "eu-de-02": "sellout",
                    "eu-de-03": "normal"
                },

Implement Unittests

Code coverage at the moment 0%.

Missing Helm Chart Testing

The last release contains a broken Helm chart. Time to setup a Github Action workflow to test the chart
e.g.
helm template
helm lint

ref to include a test: https://helm.sh/docs/topics/chart_tests/

Log handling from OTC RDS

As a user I want to be informed about RDS log events (error log, slow query log). This can be many information in a structured way, but can not handle in the RDS K8s resource itself. Client-go Event Log could be an option.

Operator can't create sql user due restricted NetworkPolicy

If a default restricted Egress NetworkPolicy is deployed, no access to the database (port tcp/3306) is possible. Creating on sql schema and user will fail.

Fix Go Security Issues

Semgrep reported some minor security issues. This needs to fix

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.