Helm chart - use external etcd about spilo HOT 39 CLOSED

If the master fails, one of replicas will promote to master.
You can use them as read replicas, just create a Service with labelSelector = spilo-role=replica for that.

from spilo.

tried to pin to etcd-operator created cluster svc Etcd.Host=etcd-cluster-client, that did not work still patroni etcd was created.

from spilo.

as I tried resilience of of patroni etcd, which is not good, if etcd pod gets restarted/moved to another node it does not come up anymore:

kubectl logs patroni1-etcd-2
cat: can't open '/var/run/etcd/member_id': No such file or directory
Re-joining etcd member

from spilo.

I thing you need to set value of Etcd.Host to the first Pod of etcd cluster created by etcd-operator

https://github.com/coreos/etcd-operator#create-and-destroy-an-etcd-cluster

$ kubectl get pods
NAME                            READY     STATUS    RESTARTS   AGE
example-etcd-cluster-0000       1/1       Running   0          1m
example-etcd-cluster-0001       1/1       Running   0          1m
example-etcd-cluster-0002       1/1       Running   0          1m

In this example it would be example-etcd-cluster-0000. Patroni will use it and discover all other nodes of etcd-cluster.

from spilo.

but if that first etcd pod gets destroyed, then etcd-operator creates new pod with the new name.
not really HA setup
svc is better to be used there

from spilo.

but if that first etcd pod gets destroyed, then etcd-operator creates new pod with the new name.

Will it? I've thought it will preserve original name and mimic so to say StatefulSet behaviour.

svc is better to be used there

It also could work. You can create kubernetes Service with labelSelector finding all Pods of etcd-cluster and specify such service in the Etcd.Host.

At the end Patroni will anyway use such Service only once, to get a topology of etcd-cluster and later it will connect to event node individually.

from spilo.

no, it does not mimic StatefulSet behaviour
I already tried to use to etcd-operator created cluster svc Etcd.Host=etcd-cluster-client, that did not work still patroni etcd was created.

from spilo.

that's not good approach to be used with etcd-operator:

At the end Patroni will anyway use such Service only once, to get a topology of etcd-cluster and later it will connect to event node individually.

as etcd-operator always recreates a new pod with the new name

from spilo.

as etcd-operator always recreates a new pod with the new name

Patroni is much smarter than you think. If the "Pod" it connected to has failed, it will switch to another "Pod" and rediscover topology of etcd cluster. If nothing is failing, it will refresh topology every 5 minutes. If all Pods failed at the same time, Patroni will go back to the original ETCD_HOST specified in the configuration. If it points to the Service - everything will be fine. Basically you can rotate all etcd Pods and Patroni will survive.

from spilo.

ok, cool then
but why it did not connect to SVC of the etcd-operator created cluster?

from spilo.

Does etcd operator creates service?

from spilo.

yup, as you can see below it is etcd-cluster-client.

$ k get service
NAME                   CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
etcd-cluster           None         <none>        2379/TCP,2380/TCP   2h
etcd-cluster-client    10.3.0.149   <none>        2379/TCP            2h

$ k describe svc etcd-cluster-client
Name:			etcd-cluster-client
Namespace:		spcqm-system
Labels:			app=etcd
			etcd_cluster=etcd-cluster
Annotations:		service.alpha.kubernetes.io/tolerate-unready-endpoints=true
Selector:		app=etcd,etcd_cluster=etcd-cluster
Type:			ClusterIP
IP:			10.3.0.149
Port:			client	2379/TCP
Endpoints:		10.2.2.11:2379,10.2.3.16:2379,10.2.4.13:2379
Session Affinity:	None
Events:			<none>

$ k get pods -l app=etcd -o wide
NAME                READY     STATUS    RESTARTS   AGE       IP          NODE
etcd-cluster-0000   1/1       Running   0          2h        10.2.4.13   xxx
etcd-cluster-0001   1/1       Running   0          2h        10.2.3.16   xxx
etcd-cluster-0002   1/1       Running   0          2h        10.2.2.11   xxx

from spilo.

And what curl http://etcd-cluster-client:2379/v2/machines shows?

You need to execute it from one of the pods (for example one of patroni pods)

from spilo.

root@patroni3-patroni-0:/home/postgres# curl http://etcd-cluster-client:2379/v2/machines

http://etcd-cluster-0000.etcd-cluster.spcqm-system.svc:2379, http://etcd-cluster-0001.etcd-cluster.spcqm-system.svc:2379, http://etcd-cluster-0002.etcd-cluster.spcqm-system.svc:2379

looks good there

from spilo.

Looks good.
Is http://etcd-cluster-0000.etcd-cluster.spcqm-system.svc:2379 accessible from patroni pod?
And what echo $ETCD_HOST shows?

from spilo.

root@patroni3-patroni-0:/home/postgres# env | grep ETCD_HOST
ETCD_HOST=etcd-cluster-client

from spilo.

root@patroni3-patroni-0:/home/postgres# curl http://etcd-cluster-0000.etcd-cluster.spcqm-system.svc:2379
404 page not found

from spilo.

etcd-operator is installed to the same namespace as patroni

from spilo.

DNS check of the POD is fine:

 kubectl exec busybox -- nslookup etcd-cluster-0000.etcd-cluster.spcqm-system.svc
Server:    10.3.0.10
Address 1: 10.3.0.10 kube-dns.kube-system.svc.cluster.local

Name:      etcd-cluster-0000.etcd-cluster.spcqm-system.svc
Address 1: 10.2.4.13 etcd-cluster-0000.etcd-cluster.spcqm-system.svc.cluster.local

from spilo.

Everything looks good. Patroni is configured to use etcd cluster deployed by etcd operator.

Now I am completely lost and don't understand what you problem is.

from spilo.

it is more patroni related issue, etcd-operator is functioning fine, I do not have RBAC enabled there

from spilo.

it is more patroni related issue

Not really Patroni issue, but patroni helm chart.
I am not really familiar with helm chart internals, but it seem Patroni chart has etcd as dependency: https://github.com/kubernetes/charts/blob/master/incubator/patroni/requirements.yaml

from spilo.

chart’s readme says that etcd_host is not used

from spilo.

i will play remove that dependency tomorrow, but if that env var is not used by patroni patroni should fail

from spilo.

chart’s readme says that etcd_host is not used

Looking on chart internals
(https://github.com/kubernetes/charts/blob/master/incubator/patroni/templates/statefulset-patroni.yaml#L49) I can tell that it is definitely used and propagated to the StatfulSet and underlying Pods.
Readme is just wrong, sorry about that, I am not maintainer of Patroni helm chart.
You can create a pull request updating helm chart documentation.

P.S. I am working on Patroni kubernetes native deployment: zalando/patroni#500
It makes it possible to deploy Patroni on kubernetes without etcd. If you have time please try it.

from spilo.

from spilo.

@CyberDem0n is the Patroni kubernetes native deployment and https://github.com/zalando-incubator/postgres-operator the same thing?

from spilo.

No, postgres-operator is a tool similar to the etcd-operator.

from spilo.

interesting, you guys have two new projects to run postgres in kube

from spilo.

now I'm not sure which one to stick to

from spilo.

Actually not two, but three.

Patroni - does all heavy lifting, like automatic failover and so one. Can work on bare metal and inside docker.
Spilo - this is a docker package of Patroni+PostgreSQL+wal-e+some other useful stuff.
postgres-operator - deploys Spilo on kubernetes using third party resources

from spilo.

heh, I know three, but I meant Patroni kubernetes native deployment and https://github.com/zalando-incubator/postgres-operator ones are new projects.
ok, for now will use the spilo one via helm chart, but in future which one of two new will becoame the main kubernetes solution?

from spilo.

postgres-operator is deploying Spilo cluster on kubernetes the same way as etcd-operator deploys etcd cluster

from spilo.

btw all the time Etcd.Host=etcd-cluster-client was fine for spilo (stupid me), deleted the etcd requirement and patroni cluster is working fine with etcd-operator.
sorry for the noise, at least learned more about patroni heh

from spilo.

@CyberDem0n one question regarding patroni/spilo, there is one master, are the rest of nodes used as read only replicas?

from spilo.

can I put some proxy in front of master/read replicas? so only one service can be used to access master/read replicas?

from spilo.

You mean Layer 7 proxy which will parse a SQL statement and figure out where to send them?
It's not possible, there is no way to distinguish between reads and writes, think about sproc call: SELECT * FROM insert_log(1,2,3); which will insert row into a table.

from spilo.

ok, not to worries then, so far with etcd-operator it is working fine, will use such setup for now.
But in future which project will be the main one for kubernetes?

from spilo.

same issue You had first @rimusz :
helm/charts#685
Says it will be fixed with this PR: helm/charts#2864

from spilo.