ciscocloud / haproxy-consul Goto Github PK
View Code? Open in Web Editor NEWDynamic haproxy configuration using consul
License: Apache License 2.0
Dynamic haproxy configuration using consul
License: Apache License 2.0
I just had a nasty crash of haproxy-consul because the data coming from consul led to a bad haproxy.cfg file. This caused haproxy to crash on restart, stopping the container.
I would be nice if haproxy-consul could check the config file (using haproxy -f /haproxy/haproxy.cfg -c
) and if it's clean, only then restart haproxy. I would make the overall more robust and more trustworthy ;-)
An issue we have seen is where a service exists, but is not running any tasks, as when a Marathon app is suspended. This causes haproxy to have an invalid configuration.
ALERT] 253/161656 (8102) : Unable to use proxy 'xxx_backend' with wrong mode, required: http, has: tcp.
[ALERT] 253/161656 (8102) : You may want to use 'mode http'.
[ALERT] 253/161656 (8102) : Proxy 'www': unable to find required use_backend: 'xxx_backend'.
[ALERT] 253/161656 (8102) : Fatal errors found in configuration.
I have an application specified in Marathon that shouldn't have any ports assigned to it. In fact, I don't assign any ports to it, period; I omit the portMappings and network directives in my application definition.
Even, still, port information still makes it in to the haproxy configuration. The haproxy-consul container fails to work properly because it creates a configuration that tries to make haproxy listen on interface *:0, which is not valid.
The documentation implies that by setting HAPROXY_HTTP to 'false' will skip support for haproxy in the application, but this does not appear to be the case.
Hi,
I've several problems with the generated configuration of HAproxy.
Here is an extract of the current configuration :
listen kibana_30070
mode tcp
bind *:30070
server <no value> dc1-docker-dev-mesos-master1.dev.dc1.kelkoo.net.node.consul:31306 # TASK_RUNNING
This <no value>
should be the taskId
, it also break make the configuration invalid.
In consulKV (marathon/kibana/tasks/<kibana.ID> ) I got :
{
"timestamp": "2015-07-16T12:41:21.370Z",
"slaveId": "20150716-074236-1046891530-15050-30616-S0",
"id": "kibana.f2a929ba-2bb7-11e5-a9ad-ae227afcbacd",
"taskStatus": "TASK_RUNNING",
"appId": "/kibana",
"host": "dc1-docker-dev-mesos-master1.dev.dc1.kelkoo.net.node.consul",
"ports": [
31306
],
"version": "2015-07-16T12:41:12.875Z"
}
With this id
key which used to be taskId
.
It's a recent modification of marathon-consul :
Incriminated (no offence, You are doing a great jog @BrianHicks ) commit :
CiscoCloud/marathon-consul@c7b968b#diff-a3520fb65206c9e232f10baaff310c7a
Change the HAproxy template and use this id
key instead of taskId
. It's easy and simple but I'm against this id
key which is a too generic name (we already got slaveId so taskId is not a long name in comparison ).
OR
Adapt the incriminated commit (poke @BrianHicks ) and stick with taskId
.
OR
go talk with marathon people and make them use consistent name for every taskId
. Slowest solution since it has to be discuss and approve on marathon side and to be release in marathon.
/launch.sh: line 68: openssl: command not found
When letting launch.sh generate pem file, the openssl binary is not found. Should be added.
Hi,
I love this project - it works perfectly together with my Consul environment! Thank you so much for your work!
One thing I'd like in this project is the ability to enable/disable HA statistics. I can add it myself in a PR but I'd like to know if there is any reason for it not already being implemented?
Hi,
I just tried your image. I was taking a look around and was trying out the /reload.sh
which fails to run. If I call with debug enabled I get the following.
/ $ DEBUG=true sh reload.sh
+ HAPROXY_MODE=consul
+ HAPROXY_DOMAIN=haproxy.service.consul
+ CONSUL_TEMPLATE=/usr/local/bin/consul-template
+ CONSUL_CONFIG=/consul-template/config.d
+ CONSUL_CONNECT=consul.service.consul:8500
+ CONSUL_MINWAIT=2s
+ CONSUL_MAXWAIT=10s
+ CONSUL_LOGLEVEL=info
+ function update_configuration {
reload.sh: line 1: function: not found
It seems as something is not working as expected in the script, but I haven't figured out what exactly.
After my first try I was very interested in the solution of almost zero package drop on reloading the configuration. So I took a look into the lines beginning at #48.
There you make use of nl-qdisc-add --dev=lo --parent=1:4 --id=40: --update plug --buffer
which is also recommended by some other people dealing with haproxy and low package drop while reloading.
But the command nl-qdisc-add
could not be executed because the Dockerfile is missing to install the right package (libnl3-cli
). After installing it is still not possible to make use of it, because the shell is complaining about missing permissions. As you can see below.
Error: Unable to add qdisc: Operation not permitted
Adding qdisc plug dev lo id 40: parent 1:4
refcnt 0no options
Error: Unknown qdisc "plug--release-indefinite"
Do you have an idea, recommendations, tips how I can get this to work? Any help would be very appreciated.
Cheers,
Volker
Using the default HAPROXY_MODE of 'consul', I believe I'm seeing what appears to be breakage when parsing for .Name to configure backend servers.
I'm using marathon-consul to get Marathon data in to consul. It works great and I can curl GET query consul for this Marathon data all day long.
An excerpt from haproxy.cfg:
backend api.184.integration.project.platform.domain_backend
backend database.184.integration.project.platform.domain_backend
backend frontend.184.integration.project.platform.domain_backend
backend project-api_backend
server mesos-master01.infrastructure.domain 172.32.x.x:31132
backend project-database_backend
server mesos-master01.infrastructure.domain 172.32.x.x:32861
backend project-frontend_backend
server mesos-master01.infrastructure.domain 172.32.x.x:32072
I'm using SERVICE_NAME to name the service (http://gliderlabs.com/registrator/latest/user/services/) and had been wondering if this is what broke the configuration, until I did a test using SERVICE_NAME but using a more simple name, one which did not represent a FQDN.
If I don't use SERVICE_NAME, or if I don't name the service to contain periods, my backends are then assigned a server
as expected.
I've been using haproxy-consul for some weeks now and I still don't get the requirement of specifying the HAPROXY_DOMAIN environment variable for consul use.
Wouldn't it be easier to just define the acl in the tmpl file like this:
acl host_{{ .Name }} hdr(host) -m beg {{ .Name }}.
use_backend {{ .Name }}_backend if host_{{ .Name }}
Notice the dot following the second occurrence of {{ .Name }}
. It is here to ensure that services don't get confused (for example "mail" and "mailinglist").
I'm not sure though how this would work on the marathon counterpart but I don't believe HAPROXY_DOMAIN should be a requirement for at least consul.
I went even further and extended the tmpl file for multiple domains use, using consul's tags for storing the domains under the form <domain>_<tld>
.
Here are the relevant parts:
# Generated automatically by consul-template
#tagged services, tag is use for domain information
{{ range $tag, $services := services | byTag }}
{{ range $services}}
acl host_{{ .Name }}_{{ $tag}} hdr(host) -i {{ .Name }}.{{ $tag |replaceAll "_" "." }}
use_backend {{ .Name }}_{{$tag}}_backend if host_{{ .Name }}_{{ $tag }}
{{ end }}
{{ end }}
#not tagged services, the url starting part is used
{{range services}}{{if eq (.Tags |len) 0 }}
acl host_{{ .Name }} hdr(host) -m beg {{ .Name }}.
use_backend {{ .Name }}_backend if host_{{ .Name }}
{{ end }}{{ end }}
#backend definition for tagged services
{{ range $tag, $services := services | byTag }}{{range $services}}
backend {{ .Name }}_{{$tag}}_backend{{ range service (print $tag "." .Name) }}
server {{ .Node }} {{ .Address }}:{{ .Port }}{{ end }}
{{ end }}{{ end }}
#backend definition for untagged services
{{ range services }}{{ if eq (.Tags |len) 0 }}
backend {{ .Name }}_backend{{ range service .Name }}
server {{ .Node }} {{ .Address }}:{{ .Port }}{{ end }}
{{ end }}{{ end }}
There is a lot of room for improvement ;-)
I had random issues of my load balancer returning 503 errors on available services. The configuration was ok, the server seemed fine. I couldn't grasp the cause of the issue until I noticed that several instances of haproxy were running in the container.
Here is a ps
from within a failing container:
PID USER TIME COMMAND
1 root 0:00 {launch.sh} /bin/bash /launch.sh
7 root 0:06 /usr/local/bin/consul-template -config /consul-template/config.d -log-level info -wait 2s:10s -consul 127.0.0.1:8500
17 root 0:22 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf
21 root 0:00 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf 17
25 root 0:00 /usr/sbin/haproxy -D -p /var/run/haproxy.pid -f /haproxy/haproxy.cfg -sf 21
26 root 0:00 sh
31 root 0:00 ps
So the new instance, bearing the correct configuration, is living amoung the old instances. All of them are listening and answering to requests on 0.0.0.0:80. This gives random error because the correct instance could be the one to answer the call. But as time goes by, more zombie instances are living, making the correct answer more and more improbable.
I will try to fix this issue, so far I had to kill and start new containers for it to work...
Is this project still alive by the way ? I see PR waiting and the last commit is 4 months old...
Need a method add https support.
It looks like marathon 0.11 uses Id instead of taskId. This causes <no value>
to be set in the haproxy.cfg file when marathon mode is enabled.
I am receiving the same messages all the time, but i am not sure what the problem is:
2016/02/11 07:27:11 [WARN] ("key(service/haproxy/timeouts/client)") Consul returned no data (does the path exist?)
2016/02/11 07:27:11 [WARN] ("key(service/haproxy/timeouts/connect)") Consul returned no data (does the path exist?)
2016/02/11 07:27:12 [WARN] ("key(service/haproxy/maxconn)") Consul returned no data (does the path exist?)
2016/02/11 07:27:14 [WARN] ("key(service/haproxy/timeouts/server)") Consul returned no data (does the path exist?)
2016/02/11 07:28:11 [WARN] ("key(service/haproxy/timeouts/client)") Consul returned no data (does the path exist?)
2016/02/11 07:28:13 [WARN] ("key(service/haproxy/timeouts/connect)") Consul returned no data (does the path exist?)
2016/02/11 07:28:15 [WARN] ("key(service/haproxy/timeouts/server)") Consul returned no data (does the path exist?)
2016/02/11 07:28:16 [WARN] ("key(service/haproxy/maxconn)") Consul returned no data (does the path exist?)
Any ideas?
Hi again! :)
I was reading up on existing PRs after creating my own and saw some issues being fixed that I'd very much like to use in my environment.
I could help out with reviewing these changes if that could speed up merges?
Cheers!
Runnung docker stop haproxy-consul
followed by docker start haproxy-consul
results in the following error in the logs:
ln: /consul-template/template.d/haproxy.tmpl: File exists
It looks like this line in launch.sh needs to remove the link before creating it or needs to test for the existence of the link:
ln -s /consul-template/template.d/${HAPROXY_MODE}.tmpl /consul-template/template.d/haproxy.tmpl
The same problem happens if the docker daemon or the VM it runs on are restarted.
Nice... hit too quickly. Following up shortly.
I'm running the container as a upstart service , when stopping container an starting it again the launch.sh skript tries to link the haproxy template again and the "ln" command fails which causes the container to stop.
https://github.com/CiscoCloud/haproxy-consul/blob/master/launch.sh#L54
easy fix for this is to add "-sf" instead of "-s" to the link command.
or labels. What would be the most appropriate thing to expect here?
Ref: mantl/mantl#241
Haproxy suffers from the issue where reloading the configuration leads to TCP connections being dropped, and requests failing. There's a number of workarounds for the problem, for example: http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
It would be nice if this haproxy-consul Docker image could have inbuilt support for zero-downtime reloads!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.