oracle / mysql-operator Goto Github PK

Create, operate and scale self-healing MySQL clusters in Kubernetes

Makefile 0.63% Go 96.86% Shell 0.74% Python 1.59% Dockerfile 0.19%

mysql-operator's Introduction

MySQL Operator has moved

This version of MySQL Operator has been discontinued. This repository has been archived and will not receive further updates.

Please find the new repository here: http://github.com/mysql/mysql-operator

mysql-operator's People

Contributors

Stargazers

Watchers

Forkers

daniel-garcia alika liuqian1990 dikang123 neumayer gites aland-zhang eroteme anywhy kejau alejandroesc kjm0001 pintu0789 bajrang kiranmudemela satriajidam gianlucaborello louishust cimomo rrblanchett huangzhiyong capttofu simon-li fossabot johnnyrun m4oc hashstone huanwei srcnblgc binxiong ysakashita xiaojingchen owainlewis jjswain2018 maduhu cloud-ark appvia allingeek marcinc bajrang0789 kiseru-io paltaa tlyng quickserver mauritsvdvijgh xuechaos henryxie1 hoatle alantang888 kevindwei ym adriannemo perrynzhou desmondh0 jijeesh tom2jack soglad lalamove akshaychitneni pzghost blademainer syz521 wanghe-fit2cloud yuong2015 buhe drinkstore shawn0915 linqingping farice4 crazyacking umairs123 michaelfig mrmimo vuvandang1995 sinotopia y2ket haj deanwei yukirii topochan nitinpadalia geoffcraig iamwel cotocn maoqide yoshiakiyamasaki guangbochen jadeyliu939 issotina vivek-tamilarasan nnao45 zikkinbun mr-robot-in kingsqv jacklli gjj2016 vineetpatel2611 taron-ai kalebwalton xchliu

mysql-operator's Issues

Remove BMCS Go SDK

Remove the following dependency as it's not cleared for release

[[constraint]]
name = "github.com/oracle/bmcs-go-sdk"

Simplify deployment. Do we really need Helm?

Most operators can be installed without Helm. Should we revert to simple YAML deployments?

MySQL Group Replication Secure Socket Layer Support (SSL)

The MySQL Operator does not currently support SSL by default for Group Repliation.

We should document how to configure SSL for Group Replication and (potentially?) implement it by default using self signed certs.

Reference

https://dev.mysql.com/doc/refman/8.0/en/group-replication-secure-socket-layer-support-ssl.html

Add prometheus support

This is pretty easy to do as something k8s users opt into by running prometheus: tag the resulting mysql pods with the typical prometheus labels and run a sidecar that collects and exposes the stats.

(This sidecar will need some read-only creds to get to the database - presumably an account can be created for this that only accepts localhost connections.)

MysqlBackupSchedule missing attributes - no user-visible report

Had a malformed (strictly, missing attributes) MysqlBackupSchedule object. Eventually found an error about it - it was hiddin in the mysql-operator logs.

Is there a way to surface that through an event on the MBS object itself?

E0115 15:22:13.403773       1 controller.go:231] Backup schedule validation failed, err: [backupTemplate.executor: Required value: missing executor, backupTemplate.storage: Required value: missing storage]

Currently the examples used for volumes use a hostpath volume. This has, from experience, led to some confusion since it generally won't work without modification of the example. We would be better off using a different example for volumes.

Backup documentation is invalid

The backup documentation examples appear to be incorrect

examples/backup.yaml
docs/user/backup.md
We should use a from-literal when creating S3 credentials (currently documentation talks about creating these from a file which is overkill).

kubectl create secret generic s3-credentials --from-literal=accessKey=XXX --from-literal=secretAccessKey=XXX

Use controller expectations

We should make use of controller expectations in controllers where we create/delete resources.

See ReplicaSetController in core.

The code hasn't yet made its way into client-go though (kubernetes/client-go#332) so we can either vendor core or copy the files into the project.

mysql-agent can't recreate MySQL Cluster after full k8s cluster restart

BUG REPORT

Versions

MySQL Operator Version: 0.1.0

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: minikube
OS (e.g. from /etc/os-release): Buildroot 2017.11
Kernel (e.g. uname -a): Linux minikube 4.9.64 #1 SMP Fri Mar 30 21:27:22 UTC 2018 x86_64 GNU/Linux
Others:

What happened?

Mysql-agent can't recreate MySQL Cluster after full k8s restart (the VM runing one node k8s was restarted)

What you expected to happen?

MySQL Cluster should have been recreated by mysql-agent

How to reproduce it (as minimally and precisely as possible)?

Create one node k8s cluster
Deploy MySQL Cluster to it
Restart k8s VM
Check cluster state

Anything else we need to know?

kubectl get pod -n ops                                 
NAME                              READY     STATUS    RESTARTS   AGE
mysql-cluster1-0                  1/2       Running   2          22m
mysql-cluster1-1                  1/2       Running   2          22m
mysql-cluster1-2                  1/2       Running   2          21m
mysql-operator-854d4684dc-t462l   1/1       Running   3          23m

kubectl -n ops log mysql-cluster1-0 mysql-agent
Starting mysql-agent version 0.1.0                  
ERROR: logging before flag.Parse: I0418 13:23:29.251728       1 flags.go:52] FLAG: --address="0.0.0.0"   
ERROR: logging before flag.Parse: I0418 13:23:29.253051       1 flags.go:52] FLAG: --alsologtostderr="false"                                                                                                       
ERROR: logging before flag.Parse: I0418 13:23:29.253064       1 flags.go:52] FLAG: --healthcheck-port="10512"                                                                                                      
ERROR: logging before flag.Parse: I0418 13:23:29.253068       1 flags.go:52] FLAG: --hostname="mysql-cluster1-0"                                                                                                   
ERROR: logging before flag.Parse: I0418 13:23:29.253072       1 flags.go:52] FLAG: --log-backtrace-at=":0"                                                                                                         
ERROR: logging before flag.Parse: I0418 13:23:29.253077       1 flags.go:52] FLAG: --log-dir=""          
ERROR: logging before flag.Parse: I0418 13:23:29.253080       1 flags.go:52] FLAG: --log-flush-frequency="5s"                                                                                                      
ERROR: logging before flag.Parse: I0418 13:23:29.253084       1 flags.go:52] FLAG: --logtostderr="true"  
ERROR: logging before flag.Parse: I0418 13:23:29.253087       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"                                                                                                   
ERROR: logging before flag.Parse: I0418 13:23:29.253090       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster1"                                                                                           
ERROR: logging before flag.Parse: I0418 13:23:29.253093       1 flags.go:52] FLAG: --namespace="ops"     
ERROR: logging before flag.Parse: I0418 13:23:29.253203       1 flags.go:52] FLAG: --stderrthreshold="2" 
ERROR: logging before flag.Parse: I0418 13:23:29.253207       1 flags.go:52] FLAG: --v="4"               
ERROR: logging before flag.Parse: I0418 13:23:29.253209       1 flags.go:52] FLAG: --vmodule=""          
ERROR: logging before flag.Parse: I0418 13:23:29.480862       1 cluster_manager.go:105] Database not yet running. Waiting...                                                                                       
ERROR: logging before flag.Parse: I0418 13:24:39.530507       1 innodb_cluster.go:121] Clearing the MySQL binary logs                                                                                              
ERROR: logging before flag.Parse: I0418 13:24:39.641214       1 cluster_manager.go:280] Creating the cluster on the primary instance                                                                               
ERROR: logging before flag.Parse: E0418 13:24:39.718935       1 cluster_manager.go:121] Error bootstrapping cluster: creating cluster: mysqlsh --uri root:fwUSRT7OBXzV7N3I@mysql-cluster1-0:3306 --py -e dba.create_cluster('MySQLCluster'): err=exit status 1         
stdout:                                             
mysqlx: [Warning] Using a password on the command line interface can be insecure.                        

stderr:                                             
Traceback (most recent call last):                  
  File "<string>", line 1, in <module>              
SystemError: ArgumentError: Dba.create_cluster: A Cluster with the name 'MySQLCluster' already exists.   

: exit status 1                                     
failed to create new cluster                        
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).bootstrap             
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:283  
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync                  
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:119  
github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run                                                 
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:108                  
main.main                                           
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/main.go:40                              
runtime.main                                        
        /usr/local/go/src/runtime/proc.go:195       
runtime.goexit                                      
        /usr/local/go/src/runtime/asm_amd64.s:2337

kubectl -n ops log mysql-cluster1-1 mysql-agent
Starting mysql-agent version 0.1.0
ERROR: logging before flag.Parse: I0418 13:23:28.683839       1 flags.go:52] FLAG: --address="0.0.0.0"
ERROR: logging before flag.Parse: I0418 13:23:28.683863       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0418 13:23:28.683867       1 flags.go:52] FLAG: --healthcheck-port="10512"
ERROR: logging before flag.Parse: I0418 13:23:28.683871       1 flags.go:52] FLAG: --hostname="mysql-cluster1-1"
ERROR: logging before flag.Parse: I0418 13:23:28.683873       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0418 13:23:28.683877       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0418 13:23:28.683880       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0418 13:23:28.683884       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0418 13:23:28.683887       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0418 13:23:28.683890       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster1"
ERROR: logging before flag.Parse: I0418 13:23:28.683892       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0418 13:23:28.683895       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0418 13:23:28.683897       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0418 13:23:28.683900       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: I0418 13:24:29.479744       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:24:50.750149       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:00.885368       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:11.117466       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0418 13:25:21.515948       1 cluster_manager.go:127] Cluster not yet present. Waiting...

kubectl  -n ops  logs mysql-operator-854d4684dc-t462l 
Starting mysql-operator version '0.1.0'
ERROR: logging before flag.Parse: I0418 13:25:06.941234       1 options.go:90] No '/etc/mysql-operator/mysql-operator-config.yaml' was present.
ERROR: logging before flag.Parse: I0418 13:25:06.943858       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0418 13:25:06.943882       1 flags.go:52] FLAG: --kubeconfig=""
ERROR: logging before flag.Parse: I0418 13:25:06.943887       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0418 13:25:06.943893       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0418 13:25:06.943897       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0418 13:25:06.943901       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0418 13:25:06.943904       1 flags.go:52] FLAG: --master=""
ERROR: logging before flag.Parse: I0418 13:25:06.943907       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0418 13:25:06.943912       1 flags.go:52] FLAG: --mysql-agent-image="wcr.io/oracle/mysql-agent"
ERROR: logging before flag.Parse: I0418 13:25:06.943915       1 flags.go:52] FLAG: --mysql-server-image="mysql/mysql-server"
ERROR: logging before flag.Parse: I0418 13:25:06.944003       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0418 13:25:06.944008       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0418 13:25:06.944011       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0418 13:25:06.944014       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: W0418 13:25:06.944020       1 client_config.go:529] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
ERROR: logging before flag.Parse: I0418 13:25:07.007388       1 controller.go:172] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007552       1 operator_controller.go:91] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007673       1 operator_controller.go:99] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.007802       1 controller.go:77] Creating event broadcaster
ERROR: logging before flag.Parse: I0418 13:25:07.008288       1 reflector.go:202] Starting reflector *v1.MySQLBackupSchedule (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.008382       1 reflector.go:240] Listing and watching *v1.MySQLBackupSchedule from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.008766       1 operator_controller.go:169] Starting OperatorController
ERROR: logging before flag.Parse: I0418 13:25:07.008792       1 operator_controller.go:172] Waiting for caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.008799       1 cache.go:30] Waiting for caches to sync for operator-restore-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.009099       1 controller.go:263] Starting MySQLCluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.009111       1 controller.go:266] Waiting for MySQLCluster controller informer caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.009116       1 cache.go:30] Waiting for caches to sync for mysql cluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.009400       1 operator_controller.go:159] Starting OperatorController
ERROR: logging before flag.Parse: I0418 13:25:07.009434       1 operator_controller.go:162] Waiting for caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.009440       1 cache.go:30] Waiting for caches to sync for operator-backup-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.009791       1 reflector.go:202] Starting reflector *v1.MySQLCluster (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.009803       1 reflector.go:240] Listing and watching *v1.MySQLCluster from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.009916       1 controller.go:140] Starting backup schedule controller
ERROR: logging before flag.Parse: I0418 13:25:07.010008       1 controller.go:143] Waiting for backup schedule controller caches to sync
ERROR: logging before flag.Parse: I0418 13:25:07.010153       1 reflector.go:202] Starting reflector *v1.Pod (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.010236       1 reflector.go:240] Listing and watching *v1.Pod from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.017895       1 reflector.go:202] Starting reflector *v1.ConfigMap (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.017933       1 reflector.go:240] Listing and watching *v1.ConfigMap from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018157       1 reflector.go:202] Starting reflector *v1.Service (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018197       1 reflector.go:240] Listing and watching *v1.Service from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018403       1 reflector.go:202] Starting reflector *v1beta1.StatefulSet (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018197       1 reflector.go:240] Listing and watching *v1.Service from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018403       1 reflector.go:202] Starting reflector *v1beta1.StatefulSet (23h17m9.992603544s) from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018422       1 reflector.go:240] Listing and watching *v1beta1.StatefulSet from github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86
ERROR: logging before flag.Parse: I0418 13:25:07.018572       1 reflector.go:202] Starting reflector *v1.MySQLBackup (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.019838       1 reflector.go:240] Listing and watching *v1.MySQLBackup from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.019767       1 reflector.go:202] Starting reflector *v1.MySQLRestore (19h15m21.324440719s) from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.020075       1 reflector.go:240] Listing and watching *v1.MySQLRestore from github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72
ERROR: logging before flag.Parse: I0418 13:25:07.156798       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.156937       1 controller.go:147] Backup schedule controller caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.203952       1 controller.go:636] Processing object: mysql-cluster1
ERROR: logging before flag.Parse: I0418 13:25:07.210427       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210466       1 cache.go:37] Caches are synced for operator-restore-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.210475       1 operator_controller.go:180] Caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.210546       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210551       1 cache.go:37] Caches are synced for mysql cluster controller
ERROR: logging before flag.Parse: I0418 13:25:07.210556       1 controller.go:276] Starting MySQLCluster controller workers
ERROR: logging before flag.Parse: I0418 13:25:07.210563       1 controller.go:282] Started MySQLCluster controller workers
ERROR: logging before flag.Parse: I0418 13:25:07.210718       1 shared_informer.go:122] caches populated
ERROR: logging before flag.Parse: I0418 13:25:07.210818       1 cache.go:37] Caches are synced for operator-backup-controller controller
ERROR: logging before flag.Parse: I0418 13:25:07.210824       1 operator_controller.go:169] Caches are synced
ERROR: logging before flag.Parse: I0418 13:25:07.220728       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.244966       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.246377       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com/v1", ResourceVersion:"775", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:25:07.253329       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.288727       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.289259       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com/v1", ResourceVersion:"775", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:25:07.291028       1 controller.go:505] ops/mysql-cluster1: ss.Spec.Replicas=3, ss.Status.ReadyReplicas=0, ss.Status.Replicas=3
ERROR: logging before flag.Parse: I0418 13:25:07.291079       1 controller.go:315] Successfully synced 'ops/mysql-cluster1'
ERROR: logging before flag.Parse: I0418 13:25:07.291096       1 event.go:218] Event(v1.ObjectReference{Kind:"MySQLCluster", Namespace:"ops", Name:"mysql-cluster1", UID:"87e256bb-430a-11e8-9df3-080027ab5287", APIVersion:"mysql.oracle.com", ResourceVersion:"1467", FieldPath:""}): type: 'Normal' reason: 'Synced' MySQLCluster synced successfully
ERROR: logging before flag.Parse: I0418 13:30:26.174949       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.Service total 0 items received
ERROR: logging before flag.Parse: I0418 13:30:36.206328       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1beta1.StatefulSet total 0 items received
ERROR: logging before flag.Parse: I0418 13:30:53.172602       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLCluster total 1 items received
ERROR: logging before flag.Parse: I0418 13:31:37.206167       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.Pod total 1 items received
ERROR: logging before flag.Parse: I0418 13:32:14.058020       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLBackup total 0 items received
ERROR: logging before flag.Parse: I0418 13:32:18.056845       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1.ConfigMap total 0 items received
ERROR: logging before flag.Parse: I0418 13:33:26.057179       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLRestore total 0 items received
ERROR: logging before flag.Parse: I0418 13:33:33.079811       1 reflector.go:428] github.com/oracle/mysql-operator/pkg/generated/informers/externalversions/factory.go:72: Watch close - *v1.MySQLBackupSchedule total 0 items received
ERROR: logging before flag.Parse: I0418 13:36:40.210043       1 reflector.go:428] github.com/oracle/mysql-operator/vendor/k8s.io/client-go/informers/factory.go:86: Watch close - *v1beta1.StatefulSet total 0 items received

Make registry public and remove references to Docker secret

Once we have legal approval to Open Source, make the Docker repo public and remove all references to the odx-docker-pull-secret from documentation.

Rethink cluster args (do we need all of them)

The MySQL Operator initialises with the cluster args defined here. https://github.com/oracle/mysql-operator/blob/master/pkg/resources/statefulsets/statefulset.go#L163

These were based on some recommendation and examples but in retrospect some of these appear invalid and we should aim to remove as many as possible since the args here will take precedence over anything defined by a user in configuration.

Support GCP

Running the operator on Google Cloud requires an additional auth library be imported. I had a fix but never merged it in.

_ "k8s.io/client-go/plugin/pkg/client/auth/gcp"

Add changelog

Add a change log to the project.

Support multi-primary mode

As a consumer of the MySQL Operator, I would like it to support multi-primary mode.

https://dev.mysql.com/doc/refman/5.7/en/group-replication-multi-primary-mode.html

TODO:

Extend MySQLClusterSpec type to include multiPrimary bool field
Update the cluster labeller to support multi-primary mode by either guarding its execution with leader election or by disabling it in multi-primary mode and always applying the primary label.

always pending

[root@node1 mysql-operator]# kubectl describe mysqlcluster alpaca-mysql -n mysql-operator
Name:         alpaca-mysql
Namespace:    mysql-operator
Labels:       v1.mysql.oracle.com/cluster=alpaca-mysql
              v1.mysql.oracle.com/version=0.1.0
Annotations:  <none>
API Version:  mysql.oracle.com/v1
Kind:         MySQLCluster
Metadata:
  Cluster Name:        
  Creation Timestamp:  2018-04-12T06:14:27Z
  Generation:          0
  Resource Version:    229389
  Self Link:           /apis/mysql.oracle.com/v1/namespaces/mysql-operator/mysqlclusters/alpaca-mysql
  UID:                 c10f0f64-3e18-11e8-a488-002197403677
Spec:
  Multi Master:  true
  Replicas:      2
  Version:       5.7.20-1.1.2
Status:
  Errors:  <nil>
  Metadata:
    Creation Timestamp:  <nil>
  Phase:                 Pending
Events:
  Type    Reason  Age              From            Message
  ----    ------  ----             ----            -------
  Normal  Synced  1m (x4 over 1m)  mysql-operator  MySQLCluster synced successfully

Migrate release process to use Wercker pipeline.

Migrate the release process to use an automated wercker pipeline. See oci-volume-provisioner for reference implementation.

Publish new v0.1.0 release and update changelog

When open sourcing we will need to publish a new release and update the change log to reflect reality.

:q

Prometheus Metrics for MySQL Agent

We have Prometheus metrics for the MySQL Operator but do not have any for the MySQL Agent.

We should add metrics to the MySQL Agent that shows the number of backups and restores as well as lifecycle events such as errors in the cluster. See https://github.com/oracle/mysql-operator/blob/master/pkg/controllers/cluster/manager/cluster_manager.go

In particular metrics around cluster state are likely useful for operations teams observing cluster errors

switch instanceStatus {
	case innodb.InstanceStatusOnline: ...
	case innodb.InstanceStatusRecovering: ...
	case innodb.InstanceStatusMissing: ...
	case innodb.InstanceStatusNotFound: ...
	default:
		glog.Errorf("Received unrecognised cluster membership status: %q", instanceStatus)
	}

Migrate E2E test infrastructure to Ginkgo

In order to maintain our tests longer term we should migrate towards Ginko. This will be more consistent with other projects we maintain and result in simplified tests that are easier to read and maintain.

Record events in MysqlBackupSchedule controller.

See #21

We record events for validation failures in the backup and restore controllers and could use the same mechanism in the backup schedule controller.

Backup documentation and examples are incorrect

I believe the backup documentation (scheduled backups) and examples are incorrect when comparing to the go structs.

(Spike) Explore Aggregated API Server and contrast with CRD's

Group files in the examples folder

Group files in the examples folder to make things more organised.

For example examples/cluster/cluster.yml

Use a folder per feature

examples
- backup
- cluster
- restore

Global configuration file for MySQL Operator

Currently we hardcode some potentially dynamic values as constants in the operator codebase. For example

const (
	// BaseImageName is the base Docker image for the operator (mysql-ee-server).
	BaseImageName = "mysql/mysql-server"
	// AgentImageName is the base Docker image for the MySQL backup/restore agent
	AgentImageName = "wcr.io/oracle/mysql-agent"
)

As a user of the MySQL Operator, I want a global configuration file. It is assumed that this would be a KubernetesConfig map and that this config map would be created as part of the Helm chart install.

A user should be able to change these values via configuration for obvious reasons. Other values to consider might be things such as context timeouts when querying via mysqlsh.

An example configuration file for the MySQL Operator might look something like this

operator:
  baseImageName: mysql/mysql-server
  agentImageName: wcr.io/oracle/mysql-agent

Limit MySQLCluster replicas to 9

A group can consist of maximum 9 servers. Attempting to add another server to a group with 9 members causes the request to join to be refused.

https://dev.mysql.com/doc/refman/5.7/en/group-replication-frequently-asked-questions.html

Creating clusters in other namespaces than default?

When I try to create a cluster outside of the namespace default, I get the error "service account namespace/mysql-agent was not found", however the helm install only adds it to the default namespace. How do I create another serviceaccount like this one?

Create initial DB and user

At the moment I'm using a helm chart to make a job that initialises the cluster once it's up. This feels like a bit of a hack:

apiVersion: batch/v1
kind: Job
metadata:
  name: example-mysql-init
spec:
  template:
    metadata:
      name: example-mysql-init
      labels:
        role: mysql-init
    spec:
      containers:
      - name: init
        image: mysql
        command: 
        - "bash"
        - "-c"
        - |
          set -ex
          mysqladmin -h {{ .Release.Name }}-mysql --user=root --password=$DB_ROOT_PASSWD ping
          mysql -h {{ .Release.Name }}-mysql --user=root --password=$DB_ROOT_PASSWD --batch --skip-column-names <<-EOF
          	create database if not exists $DB_DATABASE;
          	grant all privileges on $DB_DATABASE.* to '$DB_USER'@'%' identified by '$DB_PASSWD';
          	flush privileges;
          	EOF
        env:
          - name: DB_PASSWD
            valueFrom:
              secretKeyRef:
                name: cluster-mysql
                key: mysql-password
          - name: DB_ROOT_PASSWD
            valueFrom:
              secretKeyRef:
                name: cluster-mysql-root-password
                key: password
          - name: DB_HOST
            value: cluster-mysql
          - name: DB_DATABASE
            value: my_db_name
          - name: DB_USER
            value: my_db_user
      restartPolicy: Never
  # the default of 6 might not be enough
  backoffLimit: 600

More complex restore example

If I'm backing up from a main (say, 3-node) cluster, I might want to test a restore to a different, sacrificial 1-node cluster in the same k8s. (It's not a backup until you know you can restore it!)

It'd be handy to see a worked example of this.

E2E tests continue to run after they've failed

Running E2E tests takes a fairly long time (~10 minutes). When a test fails the others continue even though the pipeline will never recover.

It would be more desirable to fail fast in the event of a test failure.

Init scripts

Is this a BUG REPORT or FEATURE REQUEST?

FEATURE REQUEST

What?

When I create a database cluster specify a volume which holds init scripts which are then run against the server.

A bit like: https://github.com/docker-library/mysql/blob/2e4f2224f31e101d07c1e6a7c0da8245fde725c9/8.0/docker-entrypoint.sh#L178

Custom boilerplate checker

We borrow some code to do boilerplate checking from Kubernetes which is written in Python. Would be nicer to have a single go file to perform this as it's a simple enough task.

Scheduled backup documentation missing endpoint config

It's there for ad hoc backups, not for the schedule.

How to configure readiness probes?

Hi!

Could I have some hints on how to change the readiness probes? Right now they seem to be a bout 5 minutes long, and since the first failes, it takes about 5 minutes until the next probe starts going up.

Allow to set custom server_id base for clusters

FEATURE REQUEST?

Description:
Please add an option to MySQLCluster spec definition that will allow changing default "base" server id. Currently, all cluster will have the same sets of servers ids (1000,1001,1002...) as the base value is hardcoded here: https://github.com/oracle/mysql-operator/blob/master/pkg/resources/statefulsets/statefulset.go#L164
This option would allow setting different server ids for different clusters (ex.: 100, 101, 102 for the first cluster, 200,201, 202 for the second one, etc...).

Motivation:
I want to run asynchronous replication between two group replication clusters running on k8s under this operator. In order for asynchronous replication to work, all servers should have different server_id value.

Example manifest:

# cluster: cluster1
---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: mysql-cluster1
spec:
  baseServerId: 100
  replicas: 3

# cluster: cluster2
---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: mysql-cluster2
spec:
  baseServerId: 200
  replicas: 3

server_id flag should be set based on value from field baseServerId. As a result of applying this manifest operator should create two clusters, first with server ids 100, 101, 102 and second with server id 200, 201, 202.

Versions

MySQL Operator Version: 0.1.0

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Bare metal
OS (e.g. from /etc/os-release): Ubuntu 14.04 LTS
Kernel (e.g. uname -a): 4.13.0-36-generic #40-Ubuntu SMP Fri Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Others:

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

If you are open to contributions I would be more then happy to implement this feature.

Implement support for Pod Security Policies (PSPs)

https://kubernetes.io/docs/concepts/policy/pod-security-policy/

Example: https://github.com/coreos/etcd-operator/blob/master/doc/user/spec_examples.md#custom-pod-security-context

How to change server_id?

Hi,

is there any way to change the (base) server id that is specified here https://github.com/oracle/mysql-operator/blob/master/pkg/resources/statefulsets/statefulset.go#L164

Depending on the cluster I would like to have different server ids (ex.: 100, 101, 102 for the first cluster, 200,201, 202 for the second one).

Allow user to create a my.cnf configuration file

As a user of the MySQL Operator, I want to specify a custom config map containing a my.cnf file so that I can customise the configuration of my clusters.

How to label pods?

FEATURE REQUEST (?)

How can I label the mysql-pods? Is it possible to achieve from the mysqlcluster-resource?

YAML files are untested

Possibly find a way to test the examples folder since mistakes have been creeping in.

Error creating cluster using examples/cluster/cluster.yaml

Hi,

Following instructions from the tutorial to create a simple MySQL cluster,

kubectl apply -f cluster.yaml

I get the following error message

error: unable to recognize "cluster.yaml": no matches for mysql.oracle.com/, Kind=MySQLCluster

Tried this on both minikube (v0.25.2) and Google GKE.

Thanks

Migrate to different logging library

We currently use Glog for logging. This is deprecated and no longer actively supported. We should move to a more suitable logging library.

Move documentation into Github Pages

resolv.conf fiddling fails for custom cluster dns names

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

MySQL Operator Version: 0.1.0

Environment:

Kubernetes version (use kubectl version):
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.5", GitCommit:"f01a2bf98249a4db383560443a59bed0c13575df", GitTreeState:"clean", BuildDate:"2018-03-19T15:50:45Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Cloud provider or hardware configuration: bare metal
OS (e.g. from /etc/os-release): coreos 1688.5.3
Kernel (e.g. uname -a): Linux 4.14.32-coreos

What happened?

cat <<'YAML' | kubectl apply -f -
apiVersion: mysql.oracle.com/v1
kind: MySQLCluster
metadata:
  name: clusterdb
spec:
  multiMaster: true
  replicas: 2
YAML

clusterdb-0 starts up nicely, but then clusterdb-1 stuck during startup (clusterdb-1 mysql is actually up and running, but mysql-agent not)

clusterdb-1 mysql-agent] ERROR: logging before flag.Parse: I0418 15:27:39.335504       1 cluster_manager.go:127] Cluster not yet present. Waiting...

What you expected to happen?

Database starts up

How to reproduce it (as minimally and precisely as possible)?

Setup kubernetes cluster with a DNS name different than "cluster.local", e.g. k8s-test.local

Anything else we need to know?

custer.local is currently hard-coded in

mysql-operator/pkg/resources/statefulsets/statefulset.go

Line 220 in f3b89a4

echo "$search %s.${POD_NAMESPACE}.svc.cluster.local" >> /etc/resolv.conf

Either the script should properly read out resolve.conf and produce the correct search entry or the cluster dns name should be configurable in the operator

BackupSpec has mislabeled JSON attribute

https://github.com/oracle/mysql-operator/blob/master/pkg/apis/mysql/v1/backup.go#L57

"nodeSelector" should be spelt "config" (!) :-)

Leader election

FEATURE REQUEST

Implement (config map) leader election so that only one instance of the MySQL operator is running at once. Will need a little bit of thinking re support for global vs namespace mode.

Fail to scale cluster after few days

BUG REPORT

Versions

MySQL Operator Version: 0.1.0

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Bare metal
OS (e.g. from /etc/os-release): Ubuntu 14.04 LTS
Kernel (e.g. uname -a): 4.13.0-36-generic #40-Ubuntu SMP Fri Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Others:

What happened?

Can't scale cluster from 3 to 4 nodes after 3d from the cluster creation

What you expected to happen?

Scale cluster to 4 nodes

How to reproduce it (as minimally and precisely as possible)?

Create 3 node cluster
Wait 3 days
Try to scale the cluster to 4 nodes

Anything else we need to know?

Manifest:

# cluster: cluster2
---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: mysql-cluster2
  namespace: ops
  lables:
    app: mysql-cluster2
spec:
  multiMaster: false
  replicas: 4

Pod list:

 kubectl -n ops get po -l v1.mysql.oracle.com/cluster=mysql-cluster2
NAME               READY     STATUS    RESTARTS   AGE
mysql-cluster2-0   2/2       Running   0          3d
mysql-cluster2-1   2/2       Running   0          3d
mysql-cluster2-2   2/2       Running   0          3d
mysql-cluster2-3   1/2       Running   0          1h

Logs from mysql-cluster2-3 mysql:

 kubectl -n ops logs mysql-cluster2-3 mysql 
++ grep '^search' /etc/resolv.conf
+ search='search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod'
+ echo 'search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod mysql-cluster2.ops.svc.cluster.local'
++ cat /etc/hostname
++ grep -o '[^-]*$'
+ index=3
++ expr 1000 + 3
+ /entrypoint.sh --server_id=1003 --user=mysql --datadir=/var/lib/mysql --log-error=/var/lib/mysql/mysqld.err --default-storage-engine=innodb --default-tmp-storage-engine=innodb --internal-tmp-disk-storage-engine=innodb --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_520_ci --core-file --default-password-lifetime=0 --default-time-zone=SYSTEM --explicit-defaults-for-timestamp=ON --performance-schema-consumer-events-transactions-current=ON --performance-schema-consumer-events-transactions-history=ON --innodb-buffer-pool-size=128M --innodb-buffer-pool-instances=4 --innodb-autoinc-lock-mode=2 --innodb-flush-method=O_DIRECT_NO_FSYNC --innodb-open-files=128 --innodb-log-buffer-size=4M --innodb-monitor-enable=% --innodb-print-all-deadlocks=ON --innodb-undo-log-truncate=ON --innodb-undo-tablespaces=2 --innodb-undo-logs=2 --binlog_checksum=NONE --gtid_mode=ON --enforce_gtid_consistency=ON --log_bin --binlog-format=ROW --log-slave-updates=ON --master-info-repository=TABLE --relay-log-info-repository=TABLE --slave-preserve-commit-order=ON --disabled_storage_engines=MyISAM,BLACKHOLE,FEDERATED,ARCHIVE --transaction-isolation=READ-COMMITTED --transaction-write-set-extraction=XXHASH64 --loose-group-replication-ip-whitelist=0.0.0.0/0
[Entrypoint] MySQL Docker Image 5.7.20-1.1.2
[Entrypoint] Initializing database
[Entrypoint] Database initialized
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.

[Entrypoint] ignoring /docker-entrypoint-initdb.d/*

[Entrypoint] Server shut down

[Entrypoint] MySQL init process done. Ready for start up.

[Entrypoint] Starting MySQL 5.7.20-1.1.2

Logs from mysql-cluster2-3 mysql-agent:

Starting mysql-agent version 0.1.0
ERROR: logging before flag.Parse: I0410 06:26:41.837826       1 flags.go:52] FLAG: --address="0.0.0.0"
ERROR: logging before flag.Parse: I0410 06:26:41.837915       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0410 06:26:41.837935       1 flags.go:52] FLAG: --healthcheck-port="10512"
ERROR: logging before flag.Parse: I0410 06:26:41.837953       1 flags.go:52] FLAG: --hostname="mysql-cluster2-3"
ERROR: logging before flag.Parse: I0410 06:26:41.837969       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0410 06:26:41.837988       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0410 06:26:41.838004       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0410 06:26:41.838025       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0410 06:26:41.838071       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0410 06:26:41.838100       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster2"
ERROR: logging before flag.Parse: I0410 06:26:41.838116       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0410 06:26:41.838132       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0410 06:26:41.838147       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0410 06:26:41.838163       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: I0410 06:26:41.854248       1 cluster_manager.go:105] Database not yet running. Waiting...
ERROR: logging before flag.Parse: I0410 06:26:52.208175       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:02.564109       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:12.848716       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:23.107479       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:33.385198       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:43.756710       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:27:54.010975       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:28:04.516377       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:28:14.808158       1 cluster_manager.go:127] Cluster not yet present. Waiting...
ERROR: logging before flag.Parse: I0410 06:28:25.080715       1 cluster_manager.go:127] Cluster not yet present. Waiting...
[..cut...]

Posible cause:
It seems that mysql-agent set different MySQL root password for the fourth POD, than for the first three.

# kubectl -n ops exec  -ti mysql-cluster2-0 -c mysql env | grep PASS
MYSQL_ROOT_PASSWORD=43UwOBsxQ6NgQWRS
## kubectl -n ops exec  -ti mysql-cluster2-1 -c mysql env | grep PASS
MYSQL_ROOT_PASSWORD=43UwOBsxQ6NgQWRS
# kubectl -n ops exec  -ti mysql-cluster2-2 -c mysql env | grep PASS
MYSQL_ROOT_PASSWORD=43UwOBsxQ6NgQWRS
# kubectl -n ops exec  -ti mysql-cluster2-3 -c mysql env | grep PASS
MYSQL_ROOT_PASSWORD=lDq8zR07XvYl16Z2

Can't set MySQL root password via secret

BUG REPORT

Versions

MySQL Operator Version: 0.1.0

Environment:

Kubernetes version (use kubectl version):

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:55:54Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.0", GitCommit:"fc32d2f3698e36b93322a3465f63a14e9f0eaead", GitTreeState:"clean", BuildDate:"2018-03-26T16:44:10Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}

Cloud provider or hardware configuration: Bare metal
OS (e.g. from /etc/os-release): Ubuntu 14.04 LTS
Kernel (e.g. uname -a): 4.13.0-36-generic #40-Ubuntu SMP Fri Feb 16 20:07:48 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Others:

What happened?

mysql-operator fails to create cluster when MySQL root password is set via secret

What you expected to happen?

Have MySQL cluster with root password set from the secret.

How to reproduce it (as minimally and precisely as possible)?

Create cluster with root password set from the secret.

Anything else we need to know?

Manifest:

# cluster: cluster1
---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: mysql-cluster1
  namespace: ops
  lables:
    app: mysql-cluster1
spec:
  multiMaster: false
  replicas: 3
  secretRef:
    name: mysql-cluster1-root-pass
---
apiVersion: v1
kind: Secret
metadata:
  name: mysql-cluster1-root-pass
  namespace: ops
type: Opaque
data:
  password: NDNVd09Cc3hRNk5nUVdSUwo=

mysql-agent log:

Starting mysql-agent version 0.1.0
ERROR: logging before flag.Parse: I0409 13:44:16.103594       1 flags.go:52] FLAG: --address="0.0.0.0"
ERROR: logging before flag.Parse: I0409 13:44:16.103706       1 flags.go:52] FLAG: --alsologtostderr="false"
ERROR: logging before flag.Parse: I0409 13:44:16.103731       1 flags.go:52] FLAG: --healthcheck-port="10512"
ERROR: logging before flag.Parse: I0409 13:44:16.103762       1 flags.go:52] FLAG: --hostname="mysql-cluster1-0"
ERROR: logging before flag.Parse: I0409 13:44:16.103791       1 flags.go:52] FLAG: --log-backtrace-at=":0"
ERROR: logging before flag.Parse: I0409 13:44:16.103828       1 flags.go:52] FLAG: --log-dir=""
ERROR: logging before flag.Parse: I0409 13:44:16.103850       1 flags.go:52] FLAG: --log-flush-frequency="5s"
ERROR: logging before flag.Parse: I0409 13:44:16.103882       1 flags.go:52] FLAG: --logtostderr="true"
ERROR: logging before flag.Parse: I0409 13:44:16.103913       1 flags.go:52] FLAG: --min-resync-period="12h0m0s"
ERROR: logging before flag.Parse: I0409 13:44:16.103939       1 flags.go:52] FLAG: --mysql-cluster-name="mysql-cluster1"
ERROR: logging before flag.Parse: I0409 13:44:16.103960       1 flags.go:52] FLAG: --namespace="ops"
ERROR: logging before flag.Parse: I0409 13:44:16.104031       1 flags.go:52] FLAG: --stderrthreshold="2"
ERROR: logging before flag.Parse: I0409 13:44:16.104065       1 flags.go:52] FLAG: --v="4"
ERROR: logging before flag.Parse: I0409 13:44:16.104089       1 flags.go:52] FLAG: --vmodule=""
ERROR: logging before flag.Parse: I0409 13:44:16.119717       1 cluster_manager.go:105] Database not yet running. Waiting...
ERROR: logging before flag.Parse: I0409 13:44:26.134670       1 cluster_manager.go:105] Database not yet running. Waiting...
ERROR: logging before flag.Parse: I0409 13:44:36.332867       1 innodb_cluster.go:121] Clearing the MySQL binary logs
ERROR: logging before flag.Parse: I0409 13:44:36.400822       1 cluster_manager.go:280] Creating the cluster on the primary instance
ERROR: logging before flag.Parse: E0409 13:44:36.408250       1 cluster_manager.go:121] Error bootstrapping cluster: creating cluster: mysqlsh --uri root:43UwOBsxQ6NgQWRS
@mysql-cluster1-0:3306 --py -e dba.create_cluster('MySQLCluster'): err=exit status 1
stdout:

stderr:
Logger: Tried to log to an uninitialized logger.
Invalid value specified in --uri parameter.
: exit status 1
failed to create new cluster
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).bootstrap
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:283
github.com/oracle/mysql-operator/pkg/controllers/cluster/manager.(*ClusterManager).Sync
        /go/src/github.com/oracle/mysql-operator/pkg/controllers/cluster/manager/cluster_manager.go:119
github.com/oracle/mysql-operator/cmd/mysql-agent/app.Run
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/app/mysql_agent.go:108
main.main
        /go/src/github.com/oracle/mysql-operator/cmd/mysql-agent/main.go:40
runtime.main
        /usr/local/go/src/runtime/proc.go:195
runtime.goexit
        /usr/local/go/src/runtime/asm_amd64.s:2337
ERROR: logging before flag.Parse: I0409 13:44:46.595388       1 innodb_cluster.go:121] Clearing the MySQL binary logs
ERROR: logging before flag.Parse: I0409 13:44:46.662407       1 cluster_manager.go:280] Creating the cluster on the primary instance
ERROR: logging before flag.Parse: E0409 13:44:46.670595       1 cluster_manager.go:121] Error bootstrapping cluster: creating cluster: mysqlsh --uri root:43UwOBsxQ6NgQWRS
@mysql-cluster1-0:3306 --py -e dba.create_cluster('MySQLCluster'): err=exit status 1
stdout:

stderr:
Logger: Tried to log to an uninitialized logger.
Invalid value specified in --uri parameter.
: exit status 1
failed to create new cluster

mysql server log:

++ grep '^search' /etc/resolv.conf
+ search='search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod
search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod mysql-cluster1.ops.svc.cluster.local'
+ echo 'search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod
search ops.svc.cluster.local svc.cluster.local cluster.local wikia-prod mysql-cluster1.ops.svc.cluster.local mysql-cluster1.ops.svc.cluster.local'
++ cat /etc/hostname
++ grep -o '[^-]*$'
+ index=0
++ expr 1000 + 0
+ /entrypoint.sh --server_id=1000 --user=mysql --datadir=/var/lib/mysql --log-error=/var/lib/mysql/mysqld.err --default-storage-engine=innodb --default-tmp-storage-engine=innodb --internal-tmp-disk-storage-engine=innodb --character-set-server=utf8mb4 --collation-server=utf8mb4_unicode_520_ci --core-file --default-password-lifetime=0 --default-time-zone=SYSTEM --explicit-defaults-for-timestamp=ON --performance-schema-consumer-events-transactions-current=ON --performance-schema-consumer-events-transactions-history=ON --innodb-buffer-pool-size=128M --innodb-buffer-pool-instances=4 --innodb-autoinc-lock-mode=2 --innodb-flush-method=O_DIRECT_NO_FSYNC --innodb-open-files=128 --innodb-log-buffer-size=4M --innodb-monitor-enable=% --innodb-print-all-deadlocks=ON --innodb-undo-log-truncate=ON --innodb-undo-tablespaces=2 --innodb-undo-logs=2 --binlog_checksum=NONE --gtid_mode=ON --enforce_gtid_consistency=ON --log_bin --binlog-format=ROW --log-slave-updates=ON --master-info-repository=TABLE --relay-log-info-repository=TABLE --slave-preserve-commit-order=ON --disabled_storage_engines=MyISAM,BLACKHOLE,FEDERATED,ARCHIVE --transaction-isolation=READ-COMMITTED --transaction-write-set-extraction=XXHASH64 --loose-group-replication-ip-whitelist=0.0.0.0/0
[Entrypoint] MySQL Docker Image 5.7.20-1.1.2
[Entrypoint] Starting MySQL 5.7.20-1.1.2

Create a cluster with a persistent volume failed

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

MySQL Operator Version:git commit ID:f3b89a4a8fb01a2c8aa13a67a4c7735861fb7dab

Environment:

Kubernetes version (use kubectl version):
server：v1.9.5
client：v1.8.9
Cloud provider or hardware configuration:
Bare metal ,k8s run in docker container(Deployed via rancher2.0)
OS (e.g. from /etc/os-release):CentOS Linux release 7.2.1511 (Core)
Kernel (e.g. uname -a):3.10.0-327.el7.x86_64
Others:Docker version:1.12.6

An error has occurred

When I refer to the tutorial（https://github.com/oracle/mysql-operator/blob/master/docs/user/clusters.md#clusters） to Create a cluster with a persistent volume，An error has occurred：
Error: lstat /tmp/data1: no such file or directory

But,I had create the directory " /tmp/data1" on each k8s worker node.

Details:

yaml file：

mysql-cluster-with-volume.yaml

---
kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: mysql-sc
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    type: local
  name: mysql-local-pv
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 10Gi
  hostPath:
    path: /tmp/data1
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: mysql-sc

---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: example-mysql-cluster-with-volume
spec:
  replicas: 1
  volumeClaimTemplate:
    metadata:
      name: data
    spec:
      storageClassName: mysql-sc
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi

kubectl create -f mysql-cluster-with-volume.yaml

kubectl get pods

kubectl describe pod example-mysql-cluster-with-volume-0

docker logs kubelet

Unable to mount data volume to multi-master replica based on example

BUG REPORT

Versions

MySQL Operator Version:
0.1.0

Environment:

Kubernetes version (use kubectl version):
Client: 1.9.2
Server: 1.7.5
Cloud provider or hardware configuration:
Azure
OS (e.g. from /etc/os-release):
Ubuntu 16.04.2 LTS
Linux
Kernel (e.g. uname -a):
4.4.0-119-generic
Others:
https://github.com/oracle/mysql-operator

What happened?

Following the example attaching a volume, to a mulimaster cluster of 3 errors about directory
'/tmp/data' does not exist and never progresses with provisioning. Sticking to the default and creating
an EmptyDir works fine.

Pod Status: lstat /tmp/data: no such file or directory

What you expected to happen?

Expected for the persistent volume to be created and attached to each pod of the cluster.

How to reproduce it (as minimally and precisely as possible)?

Using the following manifest:

---
apiVersion: v1
kind: PersistentVolume
metadata:
  labels:
    type: local
    app: mysql-multimaster-cluster
  name: mysql-local-volume
spec:
  accessModes:
  - ReadWriteMany
  capacity:
    storage: 50Gi
  hostPath:
    path: /tmp/data
  persistentVolumeReclaimPolicy: Recycle
  storageClassName: manual
---
apiVersion: "mysql.oracle.com/v1"
kind: MySQLCluster
metadata:
  name: mysql-multimaster-cluster
  labels:
    app: mysql-multimaster-cluster
spec:
  multiMaster: true
  replicas: 3
  volumeClaimTemplate:
    metadata:
      name: data
    spec:
      storageClassName: manual
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi

Anything else we need to know?

kublet log entry:

pr 09 16:42:02 k8s-default-10099566-2 docker[3490]: I0409 16:42:02.505704    3580 kuberuntime_manager.go:457] Container {Name:mysql Image:mysql/mysql-server:5.7.20-1.1.2 Command:[/bin/bash -ecx # Note: We fiddle with the resolv.conf fil
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          # can refer to each other using just thier hostnames (e.g. mysql-N), thus do not need
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          # to qualify their names with the name of the (headless) service (e.g. mysql-N.mysql)
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          search=$(grep ^search /etc/resolv.conf)
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          echo "$search mysql-multimaster-cluster.${POD_NAMESPACE}.svc.cluster.local" >> /etc/resolv.conf
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          # Finds the replica index from the hostname, and uses this to define
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          # a unique server id for this instance.
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          index=$(cat /etc/hostname | grep -o '[^-]*$')
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]:          /entrypoint.sh --server_id=$(expr 1000 + $index) --user=mysql --datadir=/var/lib/mysql --log-error=/var/lib/mysql/mysqld.err --default-storage-engine=innodb --default-tmp-stor
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: f:nil,SecretKeyRef:&SecretKeySelector{LocalObjectReference:LocalObjectReference{Name:mysql-multimaster-cluster-root-password,},Key:password,Optional:nil,},}} {Name:MYSQL_ROOT_HOST Valu
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: I0409 16:42:02.505861    3580 kuberuntime_manager.go:457] Container {Name:mysql-agent Image:wcr.io/oracle/mysql-agent:0.1.0 Command:[] Args:[--v=4] WorkingDir: Ports:[] EnvFrom:[] Env:
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: E0409 16:42:02.510681    3580 kuberuntime_manager.go:718] container start failed: lstat /tmp/data: no such file or directory: Generate Container Config Failed
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: E0409 16:42:02.513152    3580 kuberuntime_manager.go:718] container start failed: lstat /tmp/data: no such file or directory: Generate Container Config Failed
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: E0409 16:42:02.513286    3580 pod_workers.go:182] Error syncing pod 518c03de-3c14-11e8-9f11-000d3a71c393 ("mysql-multimaster-cluster-0_default(518c03de-3c14-11e8-9f11-000d3a71c393)"),
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: , failed to "StartContainer" for "mysql-agent" with lstat /tmp/data: no such file or directory: "Generate Container Config Failed"
Apr 09 16:42:02 k8s-default-10099566-2 docker[3490]: ]

oracle / mysql-operator Goto Github PK

mysql-operator's Introduction

MySQL Operator has moved

mysql-operator's People

Contributors

Stargazers

Watchers

Forkers

mysql-operator's Issues

BUG REPORT

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Is this a BUG REPORT or FEATURE REQUEST?

What?

FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Is this a BUG REPORT or FEATURE REQUEST?

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

FEATURE REQUEST

BUG REPORT

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

BUG REPORT

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Is this a BUG REPORT or FEATURE REQUEST?

Versions

An error has occurred

Details:

mysql-cluster-with-volume.yaml

kubectl create -f mysql-cluster-with-volume.yaml

kubectl get pods

kubectl describe pod example-mysql-cluster-with-volume-0

docker logs kubelet

Versions

What happened?

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

Recommend Projects

Recommend Topics

Recommend Org