Follow this pv tutorial

When I deleted a pod in rc, chances are will k8s start a new pod on another node. Meanwhile, the ebs volume is still attached to the old node. And there'll be events like 'Failed to mount, ..., VolumeInUse'. But sometimes k8s just start a new pod on the same node, so a little hard to repeat.

Could you please help me with my questions:

1. Is it an expected behavior that k8s'll auto detach ebs volume when recover a pod on another node?
2. If not, I should use petset?

Currently I am using the setup provided in this repo for statefulset with a few exceptions -

• NOT using configure-node.sh
• I am using "terminationGracePeriodSeconds" to be 10 instead of 0
• In couchbase service, I have not added "clusterIP: None" option
• NOT using "couchbase-ui service".

The issue I am facing is that when or if Pod - "couchbase-0" restarts, its ip will change due to flannel and because of that I am not able to connect to the UI by port-forwarding and the error log says

[ns_server:error,2017-07-20T19:23:34.313Z,nonode@nohost:dist_manager<0.142.0>:dist_manager:init:178]Configured address '10.2.17.3' seems to be invalid. Will refuse to start for safety reasons.

Since we are using persistant volumes the ip must be stored in them and if I go in the pod and manually change the "/opt/couchbase/var/lib/couchbase/ip" file, it kind of resets the settings I had done to the couchbase server and I have to set it up all again.
Also it should detect the two servers i.e. "couchbase-0 and couchbase-1" by itself but it does not do that, I have to manually add the servers and rebalance them.

I have also opened all the ports required - 8091,8092,11207,11209-11211, 11214,11215,18091,18092,21100-21299 but still the problem persists.
Could you tell me what could I do to fix this issue?

We are trying to implemenent multi data center Couchbase clusters, we will have two or more Couchbase servers running in one data center and two or more running in another data center, providing HA across data center as well as failover capabilites within data center.

Question:
Can we setup a Couchbase Server to advertise a different hostname or IP address than is defined for the host running the Couchbase Server?

Details:
Each data center will have its own Kubernetes Cluster and Couchbase clusters will be deployed on Kubernetes Cluster. As per our current understanding and analysis, Couchbase servers need to reach each individual server in the cluster and across data center. First it reaches to Admin Port and from there it gets the Hostname/IP address to connect on intra node ports.

In a single kubernetes cluster we could have used Stateful Sets and each node could discover each other with internal pod ips, however these pods ips are internal to cluster only and are not routeable outside cluster. So this approach will not work with multiple kubernetes cluster.

To mitigate this issue, we decided to use Demonsets and pinned different Clouchbase Nodes to particular hosts. This way we can port forward the ports from known host to internal pods. This works well for admin port 8091, however when to reach to intra node ports, they use the Hostname/IP of the pod itself which is not routeable. We tried to set the hostname with the name which can be resolved by our dns server, but it seems that Couchbase should be able to listen to that address, only then it will accept it as Hostname.

To mitigate this issue we decided to use use the hostNetwork feature of Kubernetes, which essentially replicates the same network interfaces inside the pod. This worked perfectly as now advertised address is host address and pod is able to listen on that address. But this is not advisable from security standpoint as now pod can influence host network.

Is there a way in which we can listen/bind to internal pod ip (without host networking) and Advertise the host address and then forward the port from host to pod?

I'm deploying couchbase-server using the statefull sets. I'm unable to connect to the cluster after trying for several different connection strings. What is the correct connection string to use ?