Hi,

I am trying to run your Kubernetes example.

I was able to achieve to run all pods/services, but when I tried to connect to proxy service (command psql --host my_ip --port 5432 postgres -U stolon -W ), it failed. So I "connect" to running stolon-keeper Docker container and tried to connect to Postgres instance by command: psql --host localhost --port 5432 postgres -U stolon -W . After password is written the error occurs:

psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?

So, I listed keeper's log:

And there is written that postgresql was stoped (time 14:27:13090). Any suggestions what is wrong?

Note: I followed kubernetes example step by step. Only for topic: Setup Superuser password, I did not know what should I do exactly. Thanks.

These are needed to let only authorized components to read and/or write to the cluster entries.

I've looked at stolon a little on off for a day or two now - sadly only in my spare time - specifically for use in a kubernetes cluster (just tested it on minikube) but haven't been able to get through to postgres following the tutorial (I am on Ubuntu 16.04 lts)

I spun up ectd from https://github.com/coreos/etcd/tree/master/hack/kubernetes-deploy and pointed all stolon endpoints to the etcd client.

After creating the keeper, sentinel, proxy, proxy service and secret it turns out that the endpoint is inside the cluster only, so I exposed the proxy service as a nodeport to allow testing.

When running:
psql --host minikubeip --port describenodeport -U stolon -w (using password1) I get a:
psql: FATAL: database "stolon" does not exist

I tried it on another type of cluster that was not minikube (https://coreos.com/kubernetes/docs/latest/kubernetes-on-vagrant.html) with the same result.

Reading the stolon-keeper, it says PG_SU_USERNAME is set to "stolon" and I have run the example completely as is, and haven't changed anything except the etcd endpoint. What am I doing wrong here?

This was yesterday - when I tried today I kept getting "ErrImagePull: "net/http: request canceled" on all the pods (trying with master, latest and 0.3.0 that I saw was comitted earlier).

Despite the trouble I am having, I think you made a really cool tech here. I love the idea and the implementation as well, it's quite clever.

Since stolon already proxies the database it can be useful to provide a multiplexing connection pool to avoid too many clients connecting to the database at once.
stolon should read the maximum connections allowed to the elected master instance using SHOW max_connections and set that as a limit to the proxy. Above that, connections will be multiplexed.
When a new master is elected, the maximum connections setting should be read again and update the maximum connections the connection pool will hold.
The connection pool will hold at least N connections to the database as it is configured from etcd.

What do you think?

When running behind a nat or another routable address (like docker with default bridge config) the listening addresses and ports will be different than the one the stolon processes are binded.
To handle this case some new --advertise options are needed.

Related to #101

For multi datacenter replication (for disaster recovery) a good enhancement will be to be able to create a "standby" stolon cluster where the "master" pg instance (but this definition should be changed) node will be a postgres standby instance following an external pg instance. The other cluster instances will just be standbys of this instance like usual. The external instance can be another stolon cluster with the replication connection string pointing to one stolon proxies or multiple load balanced (or a k8s service) proxies. So we can achieve multi datacenter replicated stolon clusters. Obviously only one of these will be the "master" cluster and the user should manually promote a standby cluster to a master cluster.

Another interesting use case for this feature will be (near) zero downtime migration to a new stolon cluster from another postgres architecture (standalone, another kind of cluster etc...).

stolonctl will add a command to promote the "standby" cluster to a "master" cluster.

Stolon version: 0.3.0 (same issue on master branch) - built from source
Consul version: 0.7.0 (default Arch Linux package)
PostgreSQL version: 9.5.4 (default Arch Linux package)
Go version: 1.7.1 (default Arch Linux package)

I've created 3 virtual machines using Virtual Box. VMs are in 10.0.3.0/24 network - 10.0.3.7 (archlinux1), 10.0.3.8 (archlinux2) and 10.0.3.9 (the last one is not used below).

Steps to reproduce.

On archlinux1:

./bin/stolon-sentinel --cluster-name mycluster --store-backend consul --listen-address 10.0.3.7

echo -n 'supasswd123' > ~/.pgsupasswd
echo -n 'replpasswd123' > ~/.pgreplpasswd

./bin/stolon-keeper --data-dir ./postgres1 --id postgres1 --cluster-name mycluster --pg-su-passwordfile ~/.pgsupasswd --pg-repl-username repluser --pg-repl-passwordfile ~/.pgreplpasswd --store-backend consul --pg-listen-address 10.0.3.7

./bin/stolon-proxy --cluster-name mycluster --listen-address 10.0.3.7 --port 25432 --store-backend consul

psql --host localhost --port 5432 -d postgres

CREATE DATABASE test;
CREATE USER test WITH PASSWORD 'qwerty';
GRANT ALL PRIVILEGES ON DATABASE test TO test;

psql --host 10.0.3.7 --port 25432 --user test

Single node configuration works as expected.

On archlinux2:

./bin/stolon-keeper --data-dir ./postgres2 --id postgres2 --cluster-name mycluster --pg-su-passwordfile ~/.pgsupasswd --pg-repl-username repluser --pg-repl-passwordfile ~/.pgreplpasswd --store-backend consul --pg-listen-address 10.0.3.8

./bin/stolon-sentinel --cluster-name mycluster --store-backend consul --listen-address 10.0.3.8

sentinel output:

2016-10-20 18:06:44.186737 [pkg_logger.go:134] I | sentinel: id: 96674841
2016-10-20 18:06:44.188273 [pkg_logger.go:134] I | sentinel: Trying to acquire sentinels leadership

(and nothing more)

keeper output:

2016-10-20 18:06:39.308606 [pkg_logger.go:134] I | keeper: id: postgres2
2016-10-20 18:06:39.313567 [pkg_logger.go:134] I | postgresql: Stopping database
2016-10-20 18:06:39.324793 [pkg_logger.go:134] I | keeper: current pg state: master
2016-10-20 18:06:39.325030 [pkg_logger.go:134] I | keeper: our keeper requested role is not available
2016-10-20 18:06:44.330639 [pkg_logger.go:134] I | keeper: current pg state: master
2016-10-20 18:06:44.330683 [pkg_logger.go:134] I | keeper: our keeper requested role is not available
2016-10-20 18:06:49.336491 [pkg_logger.go:134] I | keeper: current pg state: master

(etc, etc, etc)

keeper --debug output: http://afiskon.ru/s/21/a5c6ef5ab5_keeper.txt
sentinel --debug output: http://afiskon.ru/s/96/5affa784dc_sentinel.txt

Any advice?

I am having a hard time getting things to work on Kubernetes (on Google Container Engine), and I'll try to post as specific of issues as I can, but there may be a few - let me know if there's a better way to figure all of this out!

I couldn't find a reasonably durable way to launch etcd on Container Engine, so I went with Consul (with discovery through Atlast tokens), but Stolon appears to ignore my configuration. In my modified stolon-keeper.yaml file:

          - name: STKEEPER_CLUSTER_NAME
          # TODO(sgotti) Get cluster name from "stoloncluster" label using a downward volume api instead of duplicating the name here
            value: "kube-stolon"
          - name: STKEEPER_STORE_BACKEND
            value: "consul" # Or consul
          - name: STKEEPER_STORE_ENDPOINTS
            value: "10.127.255.151:8500"
            # Enable debugging

I tried the service name (svc-consul), the IP address of the service, and and the above IP and port. When executing this, the keeper crash-loops, with this output:

2016-05-23 17:16:36.587970 [keeper.go:772] I | keeper: id: 54974a8f
2016-05-23 17:16:36.588783 [keeper.go:782] C | keeper: cannot create keeper: error retrieving cluster data: client: etcd cluster is unavailable or misconfigured

It probably is a typo (mentioning etcd instead of the STORE_BACKEND), because consul appears to be running normally based on its logs, though I don't know how to test it from a command line or whatever.

Right now only the leader sentinel does:

• Member discovery
• Keepers status check

If will be good if all the sentinel discovery and statuses are retrieved from the leader sentinel and "merged" to get the final membersState. This means that a quorum of sentinels should agree on the discovered members and status of every member.

I am interested in bringing up production postgres databases in OpenShift Origin. It looks like there has been some work to template stolon for OpenShift so it might be a good fit. Part of our requirements include Point In Time Recovery (PITR) but I don't see any backup/restore features in stolon. Is it on the roadmap?

I have done some work with pghoard (https://github.com/ohmu/pghoard) to have it work in a docker and plan to do more work to allow it to serve archived WALs for recovery out of a docker (with object store backing). Does this fit architecturally with Stolon?

Hello.
It is not clear from documentation how I can deploy cluster with existing database dump.

Before we uses dump of data directory. I look into stolon sources and it seems what stolon calls initdb during start so only option - connect to master after cluster start and push data from dump created by pg_dump. Thanks in advance.

Hello,
I started to testing stolon+consul and have this issue:

I have three datacenter (dc1,dc2,dc3) with consul masters in each DC. This masters form single consul cluster, but each master has its own kv storage.
stolon uses kv consul storage to exchange data with other stolon instances.
So, I need to set store-endpoints to one of the consul datacenter (as example dc1).
After that stolon cluster works correctly.

But in this case we don't have fault tolerance: in case of dc1 fault, stolon in dc2,dc3 can't exchange data thru dc1 and can't select leader.

stolon should not allow to connect to database that do not follow the minimum PostgreSQL requirement.

To avoid multiple keeper processes trying to access the same datadir an exclusive fs lock should be taken. This can happen for user mistakes or for kubernetes scheduling problems and is always a good idea to avoid this from happening.

It appears that when you attempt to start up a stolon cluster, and 2-n keepers are required, there is the distinct possibility that the cluster will not come up due to a deadlock wrt electing a leader/master.

I've been trying to start a stolon cluster with 1 proxy, 1 sentinel, and 2 or 3 keepers. The underlying env is docker related. All of the containers are started at one time. All configuration of the containers is done via environment variables - not stolonctl.

The following is a very typical snapshot of the env (logs):

KEEPER #1

2016-09-04` 19:17:29.672906 [keeper.go:997] W | keeper: superuser name and replication user name are the same. Different users are suggested.
2016-09-04 19:17:29.673004 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:29.673060 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:29.673122 [keeper.go:1042] I | keeper: generated id: a209cacc
2016-09-04 19:17:29.673177 [keeper.go:1049] I | keeper: id: a209cacc
2016-09-04 19:17:29.673191 [keeper.go:1052] I | keeper: running under kubernetes.
2016-09-04 19:17:39.679676 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:39.690762 [keeper.go:620] I | keeper: Initializing database
2016-09-04 19:17:42.928930 [postgresql.go:127] I | postgresql: Setting required accesses to pg_hba.conf
2016-09-04 19:17:42.929011 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:43.945958 [postgresql.go:138] I | postgresql: Setting roles
2016-09-04 19:17:43.946005 [postgresql.go:245] I | postgresql: Adding replication role to superuser
2016-09-04 19:17:43.950187 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:44.964989 [keeper.go:648] I | keeper: current pg state: master

2016-09-04 19:17:44.965029 [keeper.go:660] I | keeper: our keeper requested role is not available

2016-09-04 19:17:44.965037 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:50.999407 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:50.999506 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:56.012936 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:56.013023 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:18:01.027622 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:18:01.027805 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:18:06.044353 [keeper.go:648] I | keeper: current pg state: master

KEEPER #2

2016-09-04 19:17:12.683218 [keeper.go:997] W | keeper: superuser name and replication user name are the same. Different users are suggested.
2016-09-04 19:17:12.683297 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:12.683332 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:12.683591 [keeper.go:1042] I | keeper: generated id: 097ec11b
2016-09-04 19:17:12.683680 [keeper.go:1049] I | keeper: id: 097ec11b
2016-09-04 19:17:12.683695 [keeper.go:1052] I | keeper: running under kubernetes.
2016-09-04 19:17:22.708022 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:22.786089 [keeper.go:620] I | keeper: Initializing database
2016-09-04 19:17:32.169408 [postgresql.go:127] I | postgresql: Setting required accesses to pg_hba.conf
2016-09-04 19:17:32.169552 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:33.663659 [postgresql.go:138] I | postgresql: Setting roles
2016-09-04 19:17:33.663715 [postgresql.go:245] I | postgresql: Adding replication role to superuser
2016-09-04 19:17:33.677821 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:34.690691 [keeper.go:648] I | keeper: current pg state: master

2016-09-04 19:17:34.690865 [keeper.go:660] I | keeper: our keeper requested role is not available

2016-09-04 19:17:34.690887 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:41.135703 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:41.135803 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:46.311731 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:46.311814 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:51.419087 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:51.419115 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:56.432315 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:56.432512 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:18:01.453198 [keeper.go:648] I | keeper: current pg state: master

KEEPER #3

2016-09-04 19:17:12.619702 [keeper.go:997] W | keeper: superuser name and replication user name are the same. Different users are suggested.
2016-09-04 19:17:12.619889 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:12.619957 [keeper.go:125] W | keeper: password file /etc/secrets/stolon permissions 01000000777 are too open. This file should only be readable to the user executing stolon! Continuing...
2016-09-04 19:17:12.620002 [keeper.go:1042] I | keeper: generated id: 8e5b50f3
2016-09-04 19:17:12.620123 [keeper.go:1049] I | keeper: id: 8e5b50f3
2016-09-04 19:17:12.620137 [keeper.go:1052] I | keeper: running under kubernetes.
2016-09-04 19:17:22.636370 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:22.653779 [keeper.go:620] I | keeper: Initializing database
2016-09-04 19:17:32.168523 [postgresql.go:127] I | postgresql: Setting required accesses to pg_hba.conf
2016-09-04 19:17:32.168907 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:33.661895 [postgresql.go:138] I | postgresql: Setting roles
2016-09-04 19:17:33.662090 [postgresql.go:245] I | postgresql: Adding replication role to superuser
2016-09-04 19:17:33.678542 [postgresql.go:177] I | postgresql: Stopping database
2016-09-04 19:17:34.692113 [keeper.go:648] I | keeper: current pg state: master

2016-09-04 19:17:34.692304 [keeper.go:660] I | keeper: our keeper requested role is not available

2016-09-04 19:17:34.692342 [postgresql.go:158] I | postgresql: Starting database
2016-09-04 19:17:41.134165 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:41.134333 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:46.311174 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:46.311212 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:51.418279 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:51.418317 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:17:56.432622 [keeper.go:648] I | keeper: current pg state: master
2016-09-04 19:17:56.432664 [keeper.go:660] I | keeper: our keeper requested role is not available
2016-09-04 19:18:01.448981 [keeper.go:648] I | keeper: current pg state: master

SENTINEL

2016-09-04 19:16:59.149493 [sentinel.go:870] I | sentinel: id: fd5d7ce1
2016-09-04 19:16:59.166822 [sentinel.go:94] I | sentinel: Trying to acquire sentinels leadership
2016-09-04 19:17:09.188373 [sentinel.go:104] I | sentinel: sentinel leadership acquired
2016-09-04 19:17:09.192785 [sentinel.go:797] I | sentinel: Initializing cluster
2016-09-04 19:17:14.207201 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, no keepers registered
2016-09-04 19:17:19.217947 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, no keepers registered
2016-09-04 19:17:24.323122 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered
2016-09-04 19:17:29.417230 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered
2016-09-04 19:17:34.430276 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered
2016-09-04 19:17:39.443094 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered
2016-09-04 19:17:44.459136 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered
2016-09-04 19:17:49.482514 [sentinel.go:816] E | sentinel: failed to update clusterView: cannot choose initial master, more than 1 keeper registered

PROXY

2016-09-04 19:18:31.656789 [proxy.go:258] I | proxy: id: c66ab4ca
2016-09-04 19:18:31.684417 [proxy.go:107] I | proxy: Starting proxying
2016-09-04 19:18:31.684606 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master
2016-09-04 19:18:36.705424 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master
2016-09-04 19:18:41.738874 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master
2016-09-04 19:18:46.748185 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master
2016-09-04 19:18:51.756402 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master
2016-09-04 19:18:56.764637 [proxy.go:188] I | proxy: no proxyconf available, closing connections to previous master

As you can see Keeper#1 and Keeper#2 BOTH try to acquire the leader/master role for keepers at the very same HH:MM:SS. It appears that this causes some sort of deadlock.

Possible solutions?

1. Increase the determination of the keeper leader based on the millisecond.
2. Introduce a random delay of between, say 0 and 15, 20, 30 seconds into the election of the leader of each type of node (keeper, sentinel, proxy) or all node types via a new ENV VAR.
3. Introduce a random delay of between, say 0 and 15, 20, 30 seconds into the startup of each node or type of node (could be done via a new ENV VAR).
4. Another option?

I'd be willing to offer a PR for one of the above if we could come to an agreement on the preferred approach.

I just performed an upgrade of my stolon cluster, and the cluster appeared to come up ok - I could log into my app and use it as normal. However, when I used stolonctl to look at the cluster, I saw the following:

=== Active sentinels ===

ID      LISTENADDRESS       LEADER
feb16396    172.18.22.9:6431    true

=== Active proxies ===

ID      LISTENADDRESS       CV VERSION
41c35f58    172.18.28.15:5432   5

=== Keepers ===

ID      LISTENADDRESS       PG LISTENADDRESS    CV VERSION  HEALTHY
1e3248eb    172.18.22.15:5431   172.18.22.15:5432   4       false
606b5dc6    172.18.28.18:5431   172.18.28.18:5432   5       true
6ce6660e    172.18.28.12:5431   172.18.28.12:5432   5       true

=== Required Cluster View ===

Version: 5
Master: 606b5dc6

===== Keepers tree =====

606b5dc6 (master)
├─1e3248eb
└─6ce6660e

I've looked thru the code for the keeper and the sentinel, and it doesn't look like degraded/unhealthy keepers like the one above get automatically reprovisioned back into the cluster as a standby. I've been waiting for about 20 minutes now, and it doesn't appear to be going to reprovision.

I kill the pod in Kubernetes and the keeper comes back, and none of the above changed. It shows the following in the log:

2016-10-08T22:51:36.019566201Z 2016-10-08 22:51:36.019463 [keeper.go:1049] I | keeper: id: 1e3248eb
2016-10-08T22:51:36.019590261Z 2016-10-08 22:51:36.019479 [keeper.go:1052] I | keeper: running under kubernetes.
2016-10-08T22:51:36.038971598Z 2016-10-08 22:51:36.038852 [postgresql.go:177] I | postgresql: Stopping database
2016-10-08T22:52:36.111371808Z 2016-10-08 22:52:36.111192 [postgresql.go:177] I | postgresql: Stopping database

As explained in http://www.postgresql.org/docs/9.4/static/warm-standby.html#STREAMING-REPLICATION-SLOTS-MANIPULATION replication slot names must match some rules.

Right now the slot name is the keeper's ID. So it should be validated against the same rules (I'd like to avoid splitting the keeper's ID from the replication slot name).

Excluding the parameters controlled by the keeper, an user should be able to set the other server parameters.

We'll need to understand if and how to handle wrong parameters (unknown or with bad value) and rollback to the previous configuration since a reload will ignore them but a restart or new start will fail.

Static binaries are useful for using them independently from the build environment and for this reason in docker images/ACIs.

As seen in kubernetes, etcd and other project using go install or go build -i with adhoc install tagfor CGOENABLED=0 (usually cgo) will make repeated builds faster since the go std lib and all dependencies objects are saved but will need user have write access to the go install pkg path. If this isn't possible we can use go build making repeated builds slower (and write a warning on the reason).

Since multiple proxies can be load balanced, the lb needs to know which proxy is healthy.

Right now on error communicating with etcd the proxies closes connections to the previous master but continues accepting connection and closing them instantly.

Instead will be better to just stop listening. In this way a load balancer with a standard tcp check will detect the proxy as not healthy and not balance to him.

On kubernetes, the kube-proxy doesn't do checks but these are done by the kubelet using a liveness/readiness probe. So we can setup a tcp check readiness probe to detect when the proxy is listening.

If stolon already supports multiple masters, let's specify that in the README.
If it doesn't let's discuss if we want to add it.

Currently the synchronous standby mode set in synchronous_standby_names are all the defined standbys.
So it may happen that if both the master and the synced standby goes down another standby that cannot be fully in sync can be elected as the master (losing some transactions).

We can add an option to enable a stricter logic. In this logic the synchronous standby(s) are choosed by the sentinel so the keeper will set only them among all the available standbys. With potstgres <=9.5 it will be only one standby while with postgresql >= 9.6 more than one could be defined.

In this way the sentinel knows which are the standby(s) that are in real sync and when the master fails it'll choose one of them. Instead if the master and all the synced standbys goes down the sentinel won't choose another available standby (that cannot be in sync) but wait for one of them to come back.

Dead keeper are polluting the cluster data and the logs. I can see two different mode to remove them:

• Add a 'DeadKeeperRemovalInterval' to the cluster spec, the sentinels will remove dead keepers after they are failed for more than the specified interval. A sane default could be something like 48 hours.
• Add a stolonctl keeper remove command to manually remove a keeper. To be safer, stolonctl should check that the keeper is failed and not the current master before doing the removal (and perhaps add a --force option).

would not make more sense to use the official postgres image as a base image for stolon?

Related to #51

stolonctl will, if enabled, like other components, use etcd auth to access the cluster hierarchy on etcd and read the needed keys.

Since the client, to do some operations also needs to communicate with the leader sentinel, we'll use etcd authentication and authorization for protecting the sentinel api instead of inventing a new authz layer.

Practically the leader sentinel will write an authentication token to an etcd key. If the client can read this key then it can use the token to communicate with the sentinel.
Additionally the sentinel will change the token every n seconds.

At startup the keeper should wait for its updated ip to appear in the store's keeperState view.

This will fix a very uncommon and strange case where a standby keeper is reconfigured with the ip of the master keeper AND the sentinels are down (#154).

Hi everybody, Stolon seem as a nice tool for Postgres failover on Kubernetes cluster. But when I was reading about it, I have not found that it is possible to do hot failover. I mean Master (read-write) and Standbys (read only). Is it possible to do this with Stolon? If it is not, will you improve Stolon in this way? Thank you.

In order for this project to be deployed easily in a kubernates cluster it'd be good to have a docker container for each released version of stolon.

When I connect to the postgres instance and create a password for the stolon superuser:

[stolon@stolon-keeper-rc-hwqxd ~]$ psql -h localhost -p 5432 postgres

I get the following error:

[stolon@stolon-keeper-rc-hou7z ~]$ psql -h localhost -p 5432 postgres
psql: could not connect to server: Connection refused
Is the server running on host "localhost" (::1) and accepting
TCP/IP connections on port 5432?
could not connect to server: Connection refused
Is the server running on host "localhost" (127.0.0.1) and accepting
TCP/IP connections on port 5432?

Add a new cluster config field for custom entries in pg_hba.conf

Experiment if it's possibile to automate major version db upgrades using pg_upgrade.

To implement features like #144 #157 #159 some breaking changes in the current store model and also in keepers local db file handling are probably inevitable.

Use this breaking point to also move the store model to a full declarative model (similar to the k8s one).
There won't be a "real" api server like in k8s since it'll require implementing it inside the sentinels (to avoid adding additional processes) and making them a frontend between the clients and the store (this will require secure http calls, authentication, need to load balance calls to them, retry if a sentinel doesn't work and many other issues to handle). Instead the store writers (sentinels and stolonctl) will directly generate and validate the "api" data and write them to the store.

This will ease implementation of the above features, remove some overlapping and confusing logics and make future improvements easier to implement.

Stolon should be able to configure, if requested, postgres synchronous replication. The keeper should automatically reconfigure master's synchronous_standby_names to the one provided by the cluster view.

Additionally an option to choose if the cluster should disable sync replication (cleaning synchronous_standby_names) or to enforce it (blocking if no slaves are syncing) will be useful and will match different use cases.

We have a stolon cluster that gets deployed within our environment, which sits on top of Kubernetes. Quite frequently, when we deploy our cluster, two of the three keepers come up, join the cluster, and work fine. The third node:

2016-10-06 16:14:09.818702 [keeper.go:1049] I | keeper: id: f2530f38
2016-10-06 16:14:09.818729 [keeper.go:1052] I | keeper: running under kubernetes.
2016-10-06 16:14:09.845721 [postgresql.go:177] I | postgresql: Stopping database
2016-10-06 16:15:09.915870 [postgresql.go:177] I | postgresql: Stopping database

The keeper keeps rebooting over and over ...

Now the clusterconfig is fixed to its hardcoded default. To make it consistend above all the members of the cluster (keepers, sentinels, proxies) it should be defined in an etcd key.

If updated all the cluster members should reconfigure themself to use the new values.

The sentinels and proxies should be able to run inside nomad with docker/rkt drivers. Since with docker their external IP and port (with the default config which uses the docker bridge network) are different from the container's ones, a new --advertise-address and pg-advertise-address option will be needed.

For the keeper, since it needs persistent data, we should wait for hashicorp/nomad#150

After an upgrade of our cluster, we are seeing messages that the client cannot perform any operations against the database. It appears that our application is pointed at the wrong keeper (standby) instead of the master.

I checked status of the cluster:

=== Active sentinels ===

ID      LISTENADDRESS       LEADER
31497ca2      172.18.62.16:6431 false

=== Active proxies ===

ID      LISTENADDRESS       CV VERSION
40a35cc7      172.18.62.11:5432 5

=== Keepers ===

ID      LISTENADDRESS       PG LISTENADDRESS    CV VERSION  HEALTHY
0c9aa03a      172.18.48.12:5431 172.18.48.12:5432                  5        true
2b0269f8      172.18.62.13:5431 172.18.62.13:5432                  5        true
e738a36b      172.18.62.12:5431 172.18.62.12:5432                  5        true

=== Required Cluster View ===

Version: 5
Master: 0c9aa03a

===== Keepers tree =====

0c9aa03a (master)
├─2b0269f8
└─e738a36b

Then I jumped on each of the three keepers and did the following:

ubuntu@ip-10-0-1-138:~$ psql -U <username> -h 172.18.48.12 -p 5432 -d <db>
db=> SELECT pg_is_in_recovery();
 pg_is_in_recovery 
-------------------
 t
(1 row)

ubuntu@ip-10-0-1-138:~$ psql -U <username> -h 172.18.48.14 -p 5432 -d <db>
db=>  SELECT pg_is_in_recovery();
 pg_is_in_recovery 
-------------------
 t
(1 row)

ubuntu@ip-10-0-1-138:~$ psql -U <username> -h 172.18.62.13 -p 5432 -d <db>
db=> SELECT pg_is_in_recovery();
 pg_is_in_recovery 
-------------------
 f
(1 row)

1. Why doesn't stolon know the master has changed?
2. Why is the sentinel marked as the leader?
3. Is a single sentinel an invalid approach? Should I always have >1 sentinel?

Sometimes apps maintains a 2 pool of connections; one for master node for writing purposes only and another one for replicas for 'read' operations. It would be good if proxies allow somehow to do that

It'll be good to support additional stores like consul and zookeper.

If a keeper is dead or not syncing for a configuration defined time it should be removed from the cluster or from the cluster view. In this way the related replication slot on the sender can be freed and the unnecessary wal to be archived.

Instead of listening on localhost in addition to the user provided postgres listen address.

A stolon client (stolonctl) will be useful to:

• list clusters
• get cluster status
• set cluster configuration

Some operation will need a communication with the sentinel leader (set cluster configuration)

I'm in the midst of standing up stolon for a project. I have one keeper, one sentinel and one proxy container (based on stolon:master on dockerhub) running in a cluster, and am attempting to get them to talk to etcd. I have a staticly configured etc cluter, no proxy nodes (yet), three nodes. I can docker exec onto the three above stolon containers, and I find nothing related to an etcd executable on those containers.

This page implies that something etcd related is running locally;
https://github.com/sorintlab/stolon/blob/c0d4a4b2c9cd22cbed84279994be4c90ecb4f53f/doc/syncrepl.md
"Assuming that you cluster name is mycluster and etcd is listening on localhost:2379"

Is this the case with the default docker images?

1. Should my etcd cluster have an etcd proxy node in front of it to receive stolon traffic and forward on to etcd hosts? If so, it seems that I would then have a single point of failure on that proxy node. Alternatively, it looks like I can set env vars that will point to multiple etcd nodes (STPROXY_STORE_ENDPOINTS, STKEEPER_STORE_ENDPOINTS, STSENTINEL_STORE_ENDPOINTS) Is this correct?

2. Should one or more of my stolon nodes be able to talk directly to etcd? If so, how would they do that in a general way that doesn't force them to talk to a specific member of the etcd cluster in case of failure?

Any advice or a use case or two would be helpful! Thanks!

If a slave is elected as the new master, the other slaves will switch syncing
to the new master. If for any reasons the elected slave was behind other slaves
it will fork a new timeline before the other slaves. The other slaves cannot
sync with it but will need a full resync (pg_basebackup at the moment).