bitpoke / mysql-operator Goto Github PK

make image has dependency on cmd/gen-crds-yaml, and expects its directory and dockerfile under build directory.

CGO_ENABLED=0 \ GOOS=linux \ GOARCH=amd64 \ go build -ldflags "-X github.com/presslabs/mysql-operator/pkg/util.AppGitState=dirty -X github.com/presslabs/mysql-operator/pkg/util.AppGitCommit=09e611034b4a166498414615f798a985ed3eb7d8 -X github.com/presslabs/mysql-operator/pkg/util.AppVersion=v0.1.4-dirty" \ -v -o bin/mysql-operator_linux_amd64 cmd/mysql-operator/main.go CGO_ENABLED=0 \ GOOS=linux \ GOARCH=amd64 \ go build -ldflags "-X github.com/presslabs/mysql-operator/pkg/util.AppGitState=dirty -X github.com/presslabs/mysql-operator/pkg/util.AppGitCommit=09e611034b4a166498414615f798a985ed3eb7d8 -X github.com/presslabs/mysql-operator/pkg/util.AppVersion=v0.1.4-dirty" \ -v -o bin/mysql-helper_linux_amd64 cmd/mysql-helper/main.go CGO_ENABLED=0 \ GOOS=linux \ GOARCH=amd64 \ go build -ldflags "-X github.com/presslabs/mysql-operator/pkg/util.AppGitState=dirty -X github.com/presslabs/mysql-operator/pkg/util.AppGitCommit=09e611034b4a166498414615f798a985ed3eb7d8 -X github.com/presslabs/mysql-operator/pkg/util.AppVersion=v0.1.4-dirty" \ -v -o bin/gen-crds-yaml_linux_amd64 cmd/gen-crds-yaml/main.go set -e; for cmd in mysql-operator mysql-helper gen-crds-yaml; do \ install -m 755 bin/${cmd}_linux_amd64 hack/docker/${cmd}/${cmd} ; \ done install: hack/docker/gen-crds-yaml/gen-crds-yaml: No such file or directory make: *** [install-docker] Error 71

Currently if orchestrator is down during pod creation, it won't get registered into orchestrator. A better approach would be to continuously check the registration.

Currently, passing around xtrabackup snaps is done trough netcat. This works as a prototype, but leveraging HTTP can improve security (eg. basic auth) and interoperability (eg. expose endpoint to ingress).

https://www.percona.com/doc/percona-toolkit/LATEST/pt-kill.html

We should expose some parameters in spec. Maybe couple it with #33.

spec:
  targetSLO:
    queryLimits:
      maxIdleTime: ...                # pt-kill --idle-time
      maxQueryTime: ...               # pt-kill --busy-time
      kill: oldest|all|all-but-oldest # pt-kill --victims
      killMode: query|connection      # pt-kill --kill-query or pt-kill --kill
      ignoreDb: []                    # pt-kill --ignore-db ...
      ignoreCommand: []               # pt-kill --ignore-command ...
      ignoreUser: []                  # pt-kill --ignore-user

1. Project description, status (beta but used in production at Presslabs)
2. Usage
   a. helm install ... (install the operator)
   b. kubectl apply -f examples/mysql-cluster.yaml (deploy a mysql cluster)
   c. scale the cluster
   d. how to access orchestrator
   e. make a backup
3. Development
4. License

Initially only the master is exposed.

We should sync the cluster status from orchestrator:

status:
  nodes:
    - name: mysql-0
      conditions:
        - type: lagged
          status: true/false/unknown
          lastTransitionStatus: 2018-...
        - type: replicating
          status: true/false/unknown
          lastTransitionStatus: 2018-...
        - type: master
          status: true/false/unknown
          lastTransitionStatus: 2018-...
      ...
  conditions:
    - type: pendingFailoverAck
      status: true/false/unknown
      lastTransitionStatus: 2018-...

kubernetes/kube-openapi#13

https://github.com/appscode/kutil/tree/master/apiextensions/v1beta1

Also the helm chart should have a .Values.installCRDs which adds the proper permissions and starts the controller with --install-crds.

We should handle credential rotations, eg. updates of:

1. utility user credentials
2. root password
3. user password
4. replication password

Add a minAvailable filed in spec, which creates a PodDisruptionBudget. The default should 50% of replicas if replicas is greater than one.

If minAvailable is 0, delete the PDB.

The StatefullSet should cread read-only replicas when scaled up.

• https://www.percona.com/blog/2017/10/23/mysql-point-in-time-recovery-right-way/
• http://lefred.be/content/howto-make-mysql-point-in-time-recovery-faster/

After making a physical backup using xtrabackup, make sure we can restore from that backup by starting a local mysql server using the backup data.

We can also check the backup using https://www.percona.com/doc/percona-xtrabackup/LATEST/howtos/backup_verification.html

Hi,

This Operator seems interesting to us as it focuses on handling Percona. You mention in README that the operator is in alpha stage and that you are using it internally on non-critical workloads. I was wondering if you can elaborate on some of the essential missing pieces that would make it production ready.

Thanks!
Devdatta

The initialization should be considered completed when a marker file exists. Otherwise the init process should cleanup everything and run again.

Project name is too coupled with google. We should allow specifying a bucket prefix, which we can default to release name.

• create a new deployment
• scale the deployment (automatic slave creation)
• manual backup
• restore from backup

• create cluster
• cluster scale-up
• cluster scale-down
• node removed from service when latency > spec.maxSlaveLatency
• node removed from service when not replicating
• spec.readOnly flag with failover
• update nodes when mysql config changes
• change in cpu, memory (storage size) config of mysql object

The controller should write backup final URI into status field of the MysqlBackup object, and that should be the source of truth regarding the backup location.

kind: MysqlBackup
spec:
  cluster: ...
status:
  backupURI: gs://...
  backupTime: 2018-05-02T16:04:00Z
  complete: true

https://www.percona.com/doc/percona-server/LATEST/management/utility_user.html

When a cluster is recovered, we should automatically acknowledge that the cluster is healed.

• https://github.com/prometheus/mysqld_exporter
• https://www.percona.com/doc/percona-toolkit/LATEST/pt-heartbeat.html for measuring lag

• generate a ConfigMap with values from .spec.mysqlConfig (see #1)
• at container init copy my.cnf from config map into /etc/mysql/my.cnf
• at container init generate a /etc/mysql/conf.d/10-dynamic.cnf - with dyanmic settings (server-id, innodb-buffer-size)
• at container init generate a /etc/mysql/conf.d/10-utility-user.cnf - with utility user credentials

The statefulset should be annotated with the configmap hash so that the server is redeployed when they change.

Do a rm -rf /var/lib/mysql/lost+found on init.

https://dev.mysql.com/doc/refman/5.7/en/replication-gtids-howto.html

https://github.com/prometheus/mysqld_exporter

Objects required for creating a cluster:

DB credentials

apiVersion: v1
kind: Secret
metadata:
  name: my-cluster-db-credentials
type: Opaque
data:
  ROOT_PASSWORD: ~RANDOM
  REPLICATION_USER: ~replication
  REPLICATION_PASSWORD: ~RANDOM
  USER: ~DATABASE
  PASSWORD: ~RANDOM
  DATABASE: ~if exists and not empty, fill in USER and PASSWORD if not specified

Backup credentials (the init credentials use the same format)

apiVersion: v1
kind: Secret
metadata:
  name: backup-credentials
type: Opaque
data:
  AWS_ACCESS_KEY_ID:
  AWS_SECRET_ACCESS_KEY:
  S3_STORAGE_CLASS: (if not specified STANDARD_IA)
  S3_REGION: (if not specified us-east-1)
  S3_ENDPOINT_URL:
  GOOGLE_SERVICE_ACCOUNT_JSON_KEY:
  GOOGLE_PROJECT_ID:
  GCS_STORAGE_CLASS: (if not specified COLDLINE)

Mysql cluster CRD

apiVersion: titanium.presslabs.net/v1alpha1
kind: MysqlCluster
metadata:
  name: my-cluster
spec:
  readReplicas: 0
  secretName: ^^MysqlCluster.name-db-credentials
  mysqlVersion: 5.7.10

  initBucketURI: gs://..., s3://
  initBucketSecretName: ~

  mysqlConf:
    skip-character-set-client-handshake: true
    ...

  backupBucketURI: gs://name/prefix/a/b, s3://name/prefix/a/b
  backupBucketSecretName: ~
  backupSchedule: ""
  podSpec:
    labels:
      app: mysql
      titanium-cluster: ^^MysqlCluster.name
    annotations: {}
    resources: {}
    afinity: {}
    nodeSelector: {}
  volumeClaimTemplate:
    # https://kubernetes.io/docs/concepts/storage/persistent-volumes/#persistentvolumeclaims

This should create/update the following resources (in this order):

1. DB Credentials secret
2. A secret with utility user credentials
3. Headless service
4. Statefull set

We should continuously sync the StatefullSet status with our CRD.

https://www.percona.com/blog/2017/12/04/internal-temporary-tables-mysql-5-7/

For example, the cluster spec should specify that any slave lag is at most X seconds. We can use this it take the node out of rotation or automatically acknowledge node recovery in orchestrator.

spec:
  targetSLO:
    maxSlaveLatency: 10s

• https://tools.percona.com/wizard
• innodb
• utf8mb4 everywhere
• proper binnary logging (log slave updates, same binlog name)
  • http://mysqlhighavailability.com/mysql-8-0-3-defaults-story-binary-logging-log-bin-and-replication-chains-log-slave-updates-are-on/
• row based replication (safer replication)
  • https://dev.mysql.com/doc/refman/5.7/en/replication-features-functions.html
  • https://dev.mysql.com/doc/refman/5.7/en/replication-sbr-rbr.html
  • https://dev.mysql.com/doc/refman/5.7/en/replication-features-limit.html
  • https://dev.mysql.com/doc/refman/5.7/en/replication-options-binary-log.html#sysvar_binlog_rows_query_log_events
• gtid enabled
• start read-only

We should add a spec.readOnly which would set the entire cluster read-only (even the master node).

The controller should make sure that once the readOnly condition in status is set to true, there is no way the master becomes writable. It is ok to have a spec.readOnly=false cluster with all it's pod read-only, but it's a bug to have a spec.readOnly=true and status.conditions[readOnly]=true with writable pods.

For this we need to do the following:

1. all cluster pods start read-only
2. the controller is responsible of making the master pod writable
3. make sure that orchestrator is started with ApplyMySQLPromotionAfterMasterFailover set to false

https://kubernetes.io/docs/concepts/containers/container-lifecycle-hooks/

When a pod is shut down gracefully it should:

1. set to maintenance in orchestrator
2. if it's master, then the operator should do a graceful failover if there are other slaves