umich-arc / keepalived Goto Github PK

doing some HA testing, i powered down the host running docker that has this keepalived docker container running.. when the host came back up, the keepalive daemon continually complains about it already running:

Tue Aug  7 19:09:56 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:09:56 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:09:56 2018: daemon is already running
Tue Aug  7 19:09:57 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:09:57 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:09:57 2018: daemon is already running
Tue Aug  7 19:09:57 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:09:57 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:09:57 2018: daemon is already running
Tue Aug  7 19:09:59 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:09:59 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:09:59 2018: daemon is already running
Tue Aug  7 19:10:00 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:10:00 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:10:00 2018: daemon is already running
Tue Aug  7 19:10:02 2018: Starting Keepalived v1.3.9 (11/11,2017)
Tue Aug  7 19:10:02 2018: Opening file '/etc/keepalived/keepalived.conf'.
Tue Aug  7 19:10:02 2018: daemon is already running

anyway to clear this condition? or modify the dockerfile/scripts to recover from this situation ?

Please clarify the usage of SERVICE_KEEPALIVED_CMD env variable at row 130 of the init.sh script.
Thanks

one of the reasons i love this particular container image for keepalived (there are quite a few out there) is the promise of integrating detection of available apiserver daemon running.

it was not something i was prepared to play with initially (as i was still getting familiar with kubernetes itself), but once i felt ready and started to look at this functionality, i couldn't get it to work properly. i might just be misunderstanding its design, so i'm opening this issue to discuss and hopefully figure out if it is designed in a way that would fit for my use-case, or if maybe it can/should be adjusted to work for me.

so as i'm trying to lab up a k8s cluster w/ HA, i have 3 master nodes that all have the api-server running. i bring up the first master node, and use this docker image with keepalived (along with a second nginx container to proxy across the 2 other systems also running api-server). this docker container starts and takes the VIP and i use the VIP address to apiserver as the advertise address, and the primary/main IP (so non-loopback) address as the listen-address.

the idea that keepalived should only take/serve the VIP if the apiserver is running, is brililant. unfortunately because of the way i have my apiserver NOT listening on localhost, this check doesn't ever succeed with the script on my host. and i continually see this from the container log:

Tue Aug  7 19:56:16 2018: /usr/lib/keepalived/scripts/chk_kube_apiserver.sh x.x.x.131 6443 exited with status 1
Tue Aug  7 19:56:19 2018: /usr/lib/keepalived/scripts/chk_kube_apiserver.sh x.x.x.131 6443 exited with status 1
Tue Aug  7 19:56:22 2018: /usr/lib/keepalived/scripts/chk_kube_apiserver.sh x.x.x.131 6443 exited with status 1

but I notice that on the next line, the script checks to see if the $vip is available on any interface, and the value of $vip passed into the script, in my case is the actual main IP of the node hosting the container (not the VIP), and this check fails (so it never tries the second curl)

So for example, the above log entries are from a host on x.x.x.131, and here is how i am starting the container when i have this issue:

docker run -d --net=host --cap-add NET_ADMIN   \
  --name k8s_api_keepalived --restart always   \
  -e KEEPALIVED_AUTOCONF=true                  \
  -e KEEPALIVED_STATE=MASTER                   \
  -e KEEPALIVED_INTERFACE=$INTERFACE           \
  -e KEEPALIVED_VIRTUAL_ROUTER_ID=2            \
  -e KEEPALIVED_UNICAST_PEER_1=x.x.x.132       \
  -e KEEPALIVED_UNICAST_PEER_2=x.x.x.133       \
  -e KEEPALIVED_KUBE_APISERVER_CHECK=true      \
  -e KUBE_APISERVER_ADDRESS=x.x.x.131          \
  -e KEEPALIVED_TRACK_INTERFACE_1=$INTERFACE   \
  -e KEEPALIVED_VIRTUAL_IPADDRESS_1="x.x.x.130/24 dev $INTERFACE" \
  arcts/keepalived

So could you please comment and help me figure out if/how the init.sh/chk_kube_apiserver.sh script can/should be modified to work in my use case, when the apiserver is NOT listening on localhost (and i can't have it simply bind to 0.0.0.0 as then it would also bind on the VIP), and the vip passed into the script is the main IP (and not detected in the container).

should i be passing in the .130 address as KUBE_APISERVER_ADDRESS? I would think that keepalived would not take this address until the script succeeds, which would never happen as the address isn't listed.

i am also confused why this check fails, since when i'm in the container and i run ip addr, it successfully sees the main IP:

/usr/lib/keepalived/scripts # ip addr | grep -c x.x.x.131
1

Hello,
thanks for sharing this awesome project. I noticed that the chk_kube_apiserver.sh script checks the / of the APIs server and having authorisation errors in the APIs server logs.

curl --silent --max-time 2 --insecure "https://$vip:$port/"
{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {
    
  },
  "status": "Failure",
  "message": "forbidden: User \"system:anonymous\" cannot get path \"/\"",
  "reason": "Forbidden",
  "details": {
    
  },
  "code": 403
}

I would suggest to use the health check instead /healthz

Here a rewite

#!/bin/sh

vip="$1"
port="$2"

 errorExit() {
     echo "*** $*" 1>&2
     exit 1
 }

 curl --silent --max-time 2 --insecure "https://localhost:$port/healthz" -o /dev/null || errorExit "Error GET https://localhost:$port/healthz"
 if ip addr | grep -q "$vip"; then
     curl --silent --max-time 2 --insecure "https://$vip:$port/healthz" -o /dev/null || errorExit "Error GET https://$vip:$port/healthz"
 fi

Thanks

The current image is quite old. I propose to create a GH actions workflow for a scheduled image build.