I just had a nasty crash of haproxy-consul because the data coming from consul led to a bad haproxy.cfg file. This caused haproxy to crash on restart, stopping the container.

I would be nice if haproxy-consul could check the config file (using haproxy -f /haproxy/haproxy.cfg -c) and if it's clean, only then restart haproxy. I would make the overall more robust and more trustworthy ;-)

An issue we have seen is where a service exists, but is not running any tasks, as when a Marathon app is suspended. This causes haproxy to have an invalid configuration.

ALERT] 253/161656 (8102) : Unable to use proxy 'xxx_backend' with wrong mode, required: http, has: tcp.
[ALERT] 253/161656 (8102) : You may want to use 'mode http'.
[ALERT] 253/161656 (8102) : Proxy 'www': unable to find required use_backend: 'xxx_backend'.
[ALERT] 253/161656 (8102) : Fatal errors found in configuration.

I have an application specified in Marathon that shouldn't have any ports assigned to it. In fact, I don't assign any ports to it, period; I omit the portMappings and network directives in my application definition.

Even, still, port information still makes it in to the haproxy configuration. The haproxy-consul container fails to work properly because it creates a configuration that tries to make haproxy listen on interface *:0, which is not valid.

The documentation implies that by setting HAPROXY_HTTP to 'false' will skip support for haproxy in the application, but this does not appear to be the case.

Hi,

I've several problems with the generated configuration of HAproxy.

Problem

Here is an extract of the current configuration :

listen kibana_30070
    mode tcp
    bind *:30070

    server <no value> dc1-docker-dev-mesos-master1.dev.dc1.kelkoo.net.node.consul:31306 # TASK_RUNNING

This <no value> should be the taskId, it also break make the configuration invalid.

In consulKV (marathon/kibana/tasks/<kibana.ID> ) I got :

{
  "timestamp": "2015-07-16T12:41:21.370Z",
  "slaveId": "20150716-074236-1046891530-15050-30616-S0",
  "id": "kibana.f2a929ba-2bb7-11e5-a9ad-ae227afcbacd",
  "taskStatus": "TASK_RUNNING",
  "appId": "/kibana",
  "host": "dc1-docker-dev-mesos-master1.dev.dc1.kelkoo.net.node.consul",
  "ports": [
    31306
  ],
  "version": "2015-07-16T12:41:12.875Z"
}

With this id key which used to be taskId.

It's a recent modification of marathon-consul :

Incriminated (no offence, You are doing a great jog @BrianHicks ) commit :
CiscoCloud/marathon-consul@c7b968b#diff-a3520fb65206c9e232f10baaff310c7a

Solutions:

Change the HAproxy template and use this id key instead of taskId. It's easy and simple but I'm against this id key which is a too generic name (we already got slaveId so taskId is not a long name in comparison ).

OR

Adapt the incriminated commit (poke @BrianHicks ) and stick with taskId .

OR

go talk with marathon people and make them use consistent name for every taskId. Slowest solution since it has to be discuss and approve on marathon side and to be release in marathon.

/launch.sh: line 68: openssl: command not found

When letting launch.sh generate pem file, the openssl binary is not found. Should be added.

Hi,

I love this project - it works perfectly together with my Consul environment! Thank you so much for your work!

One thing I'd like in this project is the ability to enable/disable HA statistics. I can add it myself in a PR but I'd like to know if there is any reason for it not already being implemented?

Hi,

I just tried your image. I was taking a look around and was trying out the /reload.sh which fails to run. If I call with debug enabled I get the following.

/ $ DEBUG=true sh reload.sh 
+ HAPROXY_MODE=consul
+ HAPROXY_DOMAIN=haproxy.service.consul
+ CONSUL_TEMPLATE=/usr/local/bin/consul-template
+ CONSUL_CONFIG=/consul-template/config.d
+ CONSUL_CONNECT=consul.service.consul:8500
+ CONSUL_MINWAIT=2s
+ CONSUL_MAXWAIT=10s
+ CONSUL_LOGLEVEL=info
+ function update_configuration {
reload.sh: line 1: function: not found

It seems as something is not working as expected in the script, but I haven't figured out what exactly.

After my first try I was very interested in the solution of almost zero package drop on reloading the configuration. So I took a look into the lines beginning at #48.
There you make use of nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --buffer which is also recommended by some other people dealing with haproxy and low package drop while reloading.

But the command nl-qdisc-add could not be executed because the Dockerfile is missing to install the right package (libnl3-cli). After installing it is still not possible to make use of it, because the shell is complaining about missing permissions. As you can see below.

Error: Unable to add qdisc: Operation not permitted
Adding qdisc plug dev lo id 40: parent 1:4
  refcnt 0no options
Error: Unknown qdisc "plug--release-indefinite"

Do you have an idea, recommendations, tips how I can get this to work? Any help would be very appreciated.

Cheers,
Volker

Using the default HAPROXY_MODE of 'consul', I believe I'm seeing what appears to be breakage when parsing for .Name to configure backend servers.

I'm using marathon-consul to get Marathon data in to consul. It works great and I can curl GET query consul for this Marathon data all day long.

An excerpt from haproxy.cfg:

backend api.184.integration.project.platform.domain_backend


backend database.184.integration.project.platform.domain_backend


backend frontend.184.integration.project.platform.domain_backend


backend project-api_backend

   server mesos-master01.infrastructure.domain 172.32.x.x:31132

backend project-database_backend

   server mesos-master01.infrastructure.domain 172.32.x.x:32861

backend project-frontend_backend

   server mesos-master01.infrastructure.domain 172.32.x.x:32072

I'm using SERVICE_NAME to name the service (http://gliderlabs.com/registrator/latest/user/services/) and had been wondering if this is what broke the configuration, until I did a test using SERVICE_NAME but using a more simple name, one which did not represent a FQDN.

If I don't use SERVICE_NAME, or if I don't name the service to contain periods, my backends are then assigned a server as expected.

I've been using haproxy-consul for some weeks now and I still don't get the requirement of specifying the HAPROXY_DOMAIN environment variable for consul use.

Wouldn't it be easier to just define the acl in the tmpl file like this:

acl host_{{ .Name }} hdr(host) -m beg {{ .Name }}.
use_backend {{ .Name }}_backend if host_{{ .Name }}

Notice the dot following the second occurrence of {{ .Name }}. It is here to ensure that services don't get confused (for example "mail" and "mailinglist").

I'm not sure though how this would work on the marathon counterpart but I don't believe HAPROXY_DOMAIN should be a requirement for at least consul.

I went even further and extended the tmpl file for multiple domains use, using consul's tags for storing the domains under the form <domain>_<tld>.

Here are the relevant parts:

# Generated automatically by consul-template

#tagged services, tag is use for domain information
{{ range $tag, $services := services | byTag }}
{{ range $services}}
    acl host_{{ .Name }}_{{ $tag}} hdr(host) -i {{ .Name }}.{{ $tag |replaceAll "_" "." }} 
    use_backend {{ .Name }}_{{$tag}}_backend if host_{{ .Name }}_{{ $tag }}
{{ end }}
{{ end }}

#not tagged services, the url starting part is used
{{range services}}{{if eq (.Tags |len) 0 }}
    acl host_{{ .Name }} hdr(host) -m beg {{ .Name }}.
    use_backend {{ .Name }}_backend if host_{{ .Name }}
{{ end }}{{ end }}

#backend definition for tagged services
{{ range $tag, $services := services | byTag }}{{range $services}}
backend {{ .Name }}_{{$tag}}_backend{{ range service (print $tag "." .Name) }}
   server {{ .Node }} {{ .Address }}:{{ .Port }}{{ end }}
{{ end }}{{ end }}

#backend definition for untagged services
{{ range services }}{{ if eq (.Tags |len) 0 }}
backend {{ .Name }}_backend{{ range service .Name }}
   server {{ .Node }} {{ .Address }}:{{ .Port }}{{ end }}
{{ end }}{{ end }}

There is a lot of room for improvement ;-)

I had random issues of my load balancer returning 503 errors on available services. The configuration was ok, the server seemed fine. I couldn't grasp the cause of the issue until I noticed that several instances of haproxy were running in the container.

Here is a ps from within a failing container:

PID   USER     TIME   COMMAND
    1 root       0:00 {launch.sh} /bin/bash /launch.sh
    7 root       0:06 /usr/local/bin/consul-template -config /consul-template/config.d -log-level info -wait 2s:10s -consul 127.0.0.1:8500
   17 root       0:22 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf
   21 root       0:00 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf 17
   25 root       0:00 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf 21
   26 root       0:00 sh
   31 root       0:00 ps

So the new instance, bearing the correct configuration, is living amoung the old instances. All of them are listening and answering to requests on 0.0.0.0:80. This gives random error because the correct instance could be the one to answer the call. But as time goes by, more zombie instances are living, making the correct answer more and more improbable.

I will try to fix this issue, so far I had to kill and start new containers for it to work...

Is this project still alive by the way ? I see PR waiting and the last commit is 4 months old...

Need a method add https support.

It looks like marathon 0.11 uses Id instead of taskId. This causes <no value> to be set in the haproxy.cfg file when marathon mode is enabled.

I am receiving the same messages all the time, but i am not sure what the problem is:

2016/02/11 07:27:11 [WARN] ("key(service/haproxy/timeouts/client)") Consul returned no data (does the path exist?)
2016/02/11 07:27:11 [WARN] ("key(service/haproxy/timeouts/connect)") Consul returned no data (does the path exist?)
2016/02/11 07:27:12 [WARN] ("key(service/haproxy/maxconn)") Consul returned no data (does the path exist?)
2016/02/11 07:27:14 [WARN] ("key(service/haproxy/timeouts/server)") Consul returned no data (does the path exist?)
2016/02/11 07:28:11 [WARN] ("key(service/haproxy/timeouts/client)") Consul returned no data (does the path exist?)
2016/02/11 07:28:13 [WARN] ("key(service/haproxy/timeouts/connect)") Consul returned no data (does the path exist?)
2016/02/11 07:28:15 [WARN] ("key(service/haproxy/timeouts/server)") Consul returned no data (does the path exist?)
2016/02/11 07:28:16 [WARN] ("key(service/haproxy/maxconn)") Consul returned no data (does the path exist?)

Any ideas?

Hi again! :)

I was reading up on existing PRs after creating my own and saw some issues being fixed that I'd very much like to use in my environment.

I could help out with reviewing these changes if that could speed up merges?

Cheers!

Runnung docker stop haproxy-consul followed by docker start haproxy-consul results in the following error in the logs:

ln: /consul-template/template.d/haproxy.tmpl: File exists

It looks like this line in launch.sh needs to remove the link before creating it or needs to test for the existence of the link:

ln -s /consul-template/template.d/${HAPROXY_MODE}.tmpl /consul-template/template.d/haproxy.tmpl

The same problem happens if the docker daemon or the VM it runs on are restarted.

Nice... hit too quickly. Following up shortly.

I'm running the container as a upstart service , when stopping container an starting it again the launch.sh skript tries to link the haproxy template again and the "ln" command fails which causes the container to stop.

https://github.com/CiscoCloud/haproxy-consul/blob/master/launch.sh#L54

easy fix for this is to add "-sf" instead of "-s" to the link command.

or labels. What would be the most appropriate thing to expect here?

@stevendborrelli @ChrisAubuchon

Ref: mantl/mantl#241

Haproxy suffers from the issue where reloading the configuration leads to TCP connections being dropped, and requests failing. There's a number of workarounds for the problem, for example: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html

It would be nice if this haproxy-consul Docker image could have inbuilt support for zero-downtime reloads!