qubitproducts / bamboo Goto Github PK

View Code? Open in Web Editor NEW

794.0 72.0 215.0 3.6 MB

HAProxy auto configuration and auto service discovery for Mesos Marathon

License: Apache License 2.0

JavaScript 10.94% Go 78.12% Shell 5.58% HTML 3.64% CSS 1.73%

ceh infra

bamboo's Issues

TCP Load Balancing

Load balancing of TCP services using haproxy

default haproxy ReloadCommand locks up

The haproxy reload script frequently locks up, prevent config updates for haproxy. I found two related commits: 3bf7fd5 d1d3b65

Note how the second one changes from reading the PID file to using pidof. On our server where the reload hangs, pidof reports two running haproxy processes, while there is only one PID in the file - which seems to be the issue mentioned by @sttts in the first commit. The existence of the extra PID, in combination with the while loop from the first commit causes updates to hang: While haproxy reloads sucessfully, the while will never terminate, so further updates will not trigger reloads anymore.

$ cat /var/run/haproxy.pid
25296
$ pidof haproxy
25296 22555

$ ps aux|grep haproxy
haproxy  22555  0.0  0.1  30512  3072 ?        Ss   09:27   0:01 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 22492
1000     22707  0.0  0.0   9356   644 pts/2    R+   13:57   0:00 grep --color=auto haproxy
root     25293  0.3  0.0   4400   636 ?        S    10:34   0:41 sh -c PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done
haproxy  25296  0.0  0.1  30696  3072 ?        Ss   10:34   0:02 haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf 22555

I wonder why the production.example.json was updated in that second commit (cc @activars): to me, it seems that reading the PIDs from the PID file is the correct way.

Rest API for triggering a reload event

EscapedId

Hi,

any specific reason for the colons in

EscapedId: strings.Replace(appId, "/", "::", -1),
?

The colons mess up my host acl

acl ::socketchat-host hdr(host) -i ::socketchat.example.io

Best,

Mike

Docker Container Fails to Deploy

Hi,

I'm attempting to deploy bamboo in a docker container as described in the documentation. I've built the docker image, but it is unable to find the config file when I run it. Here's what I'm seeing:

[root@bamboo bamboo-0.2.9]# ls -l config/bamboo_test.json
-rw-r--r--. 1 root root 642 Mar  6 20:51 config/bamboo_test.json
[root@bamboo bamboo-0.2.9]# docker run -t -i --rm -p 8000:8000 -p 80:80 \
     -e MARATHON_ENDPOINT=http://xxx.xxx.xxx.xxx:8080 \
     -e BAMBOO_ENDPOINT=http://xxx.xxx.xxx.xxx:8000 \
     -e BAMBOO_ZK_HOST=xxx.xxx.xxx.xxx:2181 \
     -e BAMBOO_ZK_PATH=/bamboo \
     -e BIND=":8000" \
     -e CONFIG_PATH="config/bamboo_test.json" \
     -e BAMBOO_DOCKER_AUTO_HOST=true \
     bamboo
sed: can't read config/bamboo_test.json: No such file or directory

Thanks in advance for the help.

hardcoded marathon URL makes HA tricky

Just wondered if it's possible to look up the current marathon master via zookeeper somehow?

As it stands a single URL seems to be the only way to find marathon, which breaks
if that marathon node is down.

Provide a string replace function for template renderer

Allow creating entries using PUT

Currently, to get a "create or update" behaviour, clients have to implement it themselves, because POSTing with an existing id will fail, as will putting with a non-existing id. It would be convenient if PUT could be used for creating new configurations too.

panic: runtime error: index out of range

I deployed bamboo on ubuntu 14.04 (master) and created a new app:
marathon Id: "jenkins"
domain: "jenkins.rnatest.brightcove.com"
Nothing seems to update in the UI and looking a the logs i see:

2014/09/09 21:11:04.189694 [ip-10-146-205-81/tF2Zg1BkAD-000001] Started POST "/api/state/domains" from 50.197.95.1:34552
2014/09/09 21:11:04.192254 [ip-10-146-205-81/tF2Zg1BkAD-000001] Returning 200 in 2.499162ms
2014/09/09 21:11:04.295404 [ip-10-146-205-81/tF2Zg1BkAD-000002] Started GET "/api/state" from 50.197.95.1:34552
2014/09/09 21:11:04.438776 [ip-10-146-205-81/tF2Zg1BkAD-000002] panic: runtime error: index out of range
2014/09/09 21:11:04.439196 [ip-10-146-205-81/tF2Zg1BkAD-000002] Returning 500 in 143.765582ms
2014/09/09 21:11:06.030963 [ip-10-146-205-81/tF2Zg1BkAD-000003] Started GET "/" from 50.197.95.1:34552
2014/09/09 21:11:06.031108 [ip-10-146-205-81/tF2Zg1BkAD-000003] Returning 304 in 95.871us
2014/09/09 21:11:06.231419 [ip-10-146-205-81/tF2Zg1BkAD-000004] Started GET "/dist/main-app.js" from 50.197.95.1:52581
2014/09/09 21:11:06.231478 [ip-10-146-205-81/tF2Zg1BkAD-000004] Returning 304 in 31.071us
2014/09/09 21:11:06.231700 [ip-10-146-205-81/tF2Zg1BkAD-000005] Started GET "/dist/main.css" from 50.197.95.1:59688
2014/09/09 21:11:06.231743 [ip-10-146-205-81/tF2Zg1BkAD-000005] Returning 304 in 22.723us
2014/09/09 21:11:06.231938 [ip-10-146-205-81/tF2Zg1BkAD-000006] Started GET "/dist/main-libs.js" from 50.197.95.1:34552
2014/09/09 21:11:06.231980 [ip-10-146-205-81/tF2Zg1BkAD-000006] Returning 304 in 21.994us
2014/09/09 21:11:06.490896 [ip-10-146-205-81/tF2Zg1BkAD-000007] Started GET "/api/state" from 50.197.95.1:59688
2014/09/09 21:11:06.642818 [ip-10-146-205-81/tF2Zg1BkAD-000007] panic: runtime error: index out of range
2014/09/09 21:11:06.643170 [ip-10-146-205-81/tF2Zg1BkAD-000007] Returning 500 in 152.238341ms
2014/09/09 21:11:07.015312 [ip-10-146-205-81/tF2Zg1BkAD-000008] Started GET "/" from 50.197.95.1:59688
2014/09/09 21:11:07.017793 [ip-10-146-205-81/tF2Zg1BkAD-000008] Returning 200 in 2.421628ms
2014/09/09 21:11:07.189494 [ip-10-146-205-81/tF2Zg1BkAD-000009] Started GET "/dist/main-libs.js" from 50.197.95.1:59688
2014/09/09 21:11:07.195451 [ip-10-146-205-81/tF2Zg1BkAD-000010] Started GET "/dist/main-app.js" from 50.197.95.1:52581
2014/09/09 21:11:07.195998 [ip-10-146-205-81/tF2Zg1BkAD-000011] Started GET "/dist/main.css" from 50.197.95.1:34552
2014/09/09 21:11:07.297142 [ip-10-146-205-81/tF2Zg1BkAD-000010] Returning 200 in 101.650699ms
2014/09/09 21:11:07.297552 [ip-10-146-205-81/tF2Zg1BkAD-000011] Returning 200 in 101.5153ms
2014/09/09 21:11:07.863142 [ip-10-146-205-81/tF2Zg1BkAD-000009] Returning 200 in 673.584726ms
2014/09/09 21:11:08.155449 [ip-10-146-205-81/tF2Zg1BkAD-000012] Started GET "/fonts/ionicons.ttf?v=1.5.2" from 50.197.95.1:52581
2014/09/09 21:11:08.155677 [ip-10-146-205-81/tF2Zg1BkAD-000013] Started GET "/api/state" from 50.197.95.1:59688
2014/09/09 21:11:08.156843 Marathon: State changed
2014/09/09 21:11:08.545464 [ip-10-146-205-81/tF2Zg1BkAD-000012] Returning 200 in 389.94999ms
2014/09/09 21:11:08.742842 [ip-10-146-205-81/tF2Zg1BkAD-000014] Started GET "/favicon.ico" from 50.197.95.1:52581
2014/09/09 21:11:08.742926 [ip-10-146-205-81/tF2Zg1BkAD-000014] Returning 404 in 41.233us
2014/09/09 21:11:08.776081 [ip-10-146-205-81/tF2Zg1BkAD-000013] panic: runtime error: index out of range
2014/09/09 21:11:08.776480 [ip-10-146-205-81/tF2Zg1BkAD-000013] Returning 500 in 620.766409ms
2014/09/09 21:11:08.834330 Starting Goji on [::]:8000
2014/09/09 21:11:10.854234 [ip-10-146-205-81/ZJyf9oeUzk-000001] Started GET "/api/state/domains" from 127.0.0.1:50246
2014/09/09 21:11:10.855212 [ip-10-146-205-81/ZJyf9oeUzk-000001] Returning 200 in 930.746us
2014/09/09 21:11:38.142537 Marathon: State changed
2014/09/09 21:11:38.472626 Starting Goji on [::]:8000
2014/09/09 21:12:08.153508 Marathon: State changed
2014/09/09 21:12:08.424940 Starting Goji on [::]:8000
2014/09/09 21:12:38.143537 Marathon: State changed
2014/09/09 21:12:38.471640 Starting Goji on [::]:8000
2014/09/09 21:13:08.636313 Marathon: State changed
2014/09/09 21:13:08.951384 Starting Goji on [::]:8000
2014/09/09 21:13:38.144468 Marathon: State changed
2014/09/09 21:13:38.418398 Starting Goji on [::]:8000
2014/09/09 21:14:08.143968 Marathon: State changed
2014/09/09 21:14:08.447555 Starting Goji on [::]:8000
2014/09/09 21:14:36.398245 [ip-10-146-205-81/I7hhxpqesc-000001] Started GET "/" from 50.197.95.1:13641
2014/09/09 21:14:36.401455 [ip-10-146-205-81/I7hhxpqesc-000001] Returning 200 in 3.141018ms
2014/09/09 21:14:36.645516 [ip-10-146-205-81/I7hhxpqesc-000002] Started GET "/dist/main-libs.js" from 50.197.95.1:13641
2014/09/09 21:14:36.645762 [ip-10-146-205-81/I7hhxpqesc-000003] Started GET "/dist/main-app.js" from 50.197.95.1:60232
2014/09/09 21:14:36.649878 [ip-10-146-205-81/I7hhxpqesc-000004] Started GET "/dist/main.css" from 50.197.95.1:24151
2014/09/09 21:14:36.832260 [ip-10-146-205-81/I7hhxpqesc-000003] Returning 200 in 186.448961ms
2014/09/09 21:14:36.832417 [ip-10-146-205-81/I7hhxpqesc-000004] Returning 200 in 182.492761ms
2014/09/09 21:14:37.412150 [ip-10-146-205-81/I7hhxpqesc-000002] Returning 200 in 766.568162ms
2014/09/09 21:14:37.759763 [ip-10-146-205-81/I7hhxpqesc-000005] Started GET "/api/state" from 50.197.95.1:13641
2014/09/09 21:14:37.763757 [ip-10-146-205-81/I7hhxpqesc-000006] Started GET "/fonts/ionicons.ttf?v=1.5.2" from 50.197.95.1:24151
2014/09/09 21:14:37.902539 [ip-10-146-205-81/I7hhxpqesc-000005] panic: runtime error: index out of range
2014/09/09 21:14:37.903141 [ip-10-146-205-81/I7hhxpqesc-000005] Returning 500 in 143.334318ms
2014/09/09 21:14:38.154143 [ip-10-146-205-81/I7hhxpqesc-000006] Returning 200 in 390.338903ms
2014/09/09 21:14:38.438154 [ip-10-146-205-81/I7hhxpqesc-000007] Started GET "/favicon.ico" from 50.197.95.1:24151
2014/09/09 21:14:38.438222 [ip-10-146-205-81/I7hhxpqesc-000007] Returning 404 in 31.211us
2014/09/09 21:14:38.601105 Marathon: State changed
2014/09/09 21:14:38.892008 Starting Goji on [::]:8000
2014/09/09 21:15:08.148493 Marathon: State changed
2014/09/09 21:15:08.416597 Starting Goji on [::]:8000
2014/09/09 21:15:38.203455 Marathon: State changed
2014/09/09 21:15:38.506437 Starting Goji on [::]:8000
2014/09/09 21:16:08.147463 Marathon: State changed
2014/09/09 21:16:08.449516 Starting Goji on [::]:8000

Looks like there is a problem but the error message isn't descriptive enough to diagnose it.

Healthcheck on different port?

With portIndex, Marathon health checks can be configured to run against any port associated with the task:

{
  "path": "/api/health",
  "portIndex": 0,
  "protocol": "HTTP",
  "gracePeriodSeconds": 30,
  "intervalSeconds": 30,
  "timeoutSeconds": 30,
  "maxConsecutiveFailures": 3
}

The health check port isn't currently available within Bamboo - it's simply assumed that it's equal to $task.Port. Consequently, configuring a Marathon app with a health check on a different port results in the app always failing HAProxy's checks.

Is there a way we could get something like $task.HealthCheckPort?

server {{ $app.EscapedId}}...check inter 30000 check port {{ $task.HealthCheckPort }}

./bamboo.go:142: unknown lumberjack.Logger field 'Filename' in struct literal

Can't build bamboo deb package

Index out of range with jobs without ports

With the new 0.2.1 release, Mesos 0.2, Marathon 0.7.0rc3, I get the backtrace shown below just after startup. I have a Docker task in Marathon without ports. I registered that with the old bamboo version.

2014/09/19 06:48:47.715526 Using environment override MARATHON_ENDPOINT=http://10.0.0.1:5080
2014/09/19 06:48:47.718157 Using environment override BAMBOO_ENDPOINT=http://10.0.0.1:9000
2014/09/19 06:48:47.719109 Using environment override BAMBOO_ZK_HOST=10.0.0.1:2181,10.0.0.2:2181,10.0.0.3:2181
2014/09/19 06:48:47.719512 Using environment override BAMBOO_ZK_PATH=/bamboo
2014/09/19 06:48:47.853548 Starting Goji on [::]:8000
2014/09/19 06:48:47.865740 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0mStarted �[35;1mPOST �[0m�[34m"/api/marathon/event_callback" �[0mfrom 10.0.0.1:53546
2014/09/19 06:48:47.868047 subscribe_event => 2014-09-19T06:48:47.807Z
2014/09/19 06:48:48.039152 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0m�[31;1mpanic: runtime error: index out of range�[0m
/opt/go/src/github.com/zenazn/goji/web/middleware/recoverer.go:24 (0x55dbed)
    com/zenazn/goji/web/middleware.func.006: debug.PrintStack()
/usr/lib/go/src/pkg/runtime/panic.c:248 (0x4149f6)
    panic: runtime·newstackcall(d->fn, (byte*)d->args, d->siz);
/usr/lib/go/src/pkg/runtime/panic.c:482 (0x41529d)
    panicstring: runtime·panic(err);
/usr/lib/go/src/pkg/runtime/panic.c:433 (0x4150b7)
    panicindex: runtime·panicstring("index out of range");
/opt/go/src/github.com/QubitProducts/bamboo/services/marathon/marathon.go:151 (0x5e3f45)
    com/QubitProducts/bamboo/services/marathon.createApps: simpleTasks = append(simpleTasks, Task{ Host: task.Host, Port: task.Ports[0] })
/opt/go/src/github.com/QubitProducts/bamboo/services/marathon/marathon.go:194 (0x5e406c)
    com/QubitProducts/bamboo/services/marathon.FetchApps: apps := createApps(tasks, marathonApps)
/opt/go/src/github.com/QubitProducts/bamboo/services/haproxy/haproxy.go:19 (0x575f61)
    com/QubitProducts/bamboo/services/haproxy.GetTemplateData: apps, _ := marathon.FetchApps(config.Marathon.Endpoint)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_handler.go:52 (0x48a3de)
    com/QubitProducts/bamboo/services/event_bus.handleHAPUpdate: templateData := haproxy.GetTemplateData(conf, conn)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_handler.go:36 (0x48a0a8)
    com/QubitProducts/bamboo/services/event_bus.(*Handlers).MarathonEventHandler: handleHAPUpdate(h.Conf, h.Zookeeper)
/opt/go/src/github.com/QubitProducts/bamboo/bamboo.go:52 (0x401d3d)
    com/QubitProducts/bamboo/services/event_bus.*Handlers.MarathonEventHandler.fm: eventBus.Register(handlers.MarathonEventHandler)
/usr/lib/go/src/pkg/runtime/asm_amd64.s:339 (0x425832)
    call32: CALLFN(call32, 32)
/usr/lib/go/src/pkg/reflect/value.go:474 (0x5428ab)
    Value.call: call(fn, ptr, uint32(size))
/usr/lib/go/src/pkg/reflect/value.go:345 (0x54199d)
    Value.Call: return v.call("Call", in)
/opt/go/src/github.com/QubitProducts/bamboo/services/event_bus/event_bus.go:66 (0x489d6a)
    com/QubitProducts/bamboo/services/event_bus.(*EventBus).Publish: fn.Call(args[:])
/opt/go/src/github.com/QubitProducts/bamboo/api/event_subscription.go:29 (0x484e46)
    com/QubitProducts/bamboo/api.(*EventSubscriptionAPI).Callback: sub.EventBus.Publish(event)
/opt/go/src/github.com/QubitProducts/bamboo/bamboo.go:77 (0x401f84)
    com/QubitProducts/bamboo/api.*EventSubscriptionAPI.Callback.fm: goji.Post("/api/marathon/event_callback", eventSubAPI.Callback)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/router.go:113 (0x557c00)
    com/zenazn/goji/web.netHTTPWrap.ServeHTTPC: h.Handler.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/web/atomic.go:1 (0x55ad32)
    com/zenazn/goji/web.(*netHTTPWrap).ServeHTTPC: // +build !appengine
/opt/go/src/github.com/zenazn/goji/web/router.go:210 (0x55849f)
    com/zenazn/goji/web.routeMachine.route: rm.routes[si].handler.ServeHTTPC(*c, w, r)
/opt/go/src/github.com/zenazn/goji/web/router.go:251 (0x55880f)
    com/zenazn/goji/web.(*router).route: methods, ok := rm.route(c, w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware.go:100 (0x5596d5)
    com/zenazn/goji/web.func.002: router.route(&cs.C, w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/options.go:70 (0x55db3e)
    com/zenazn/goji/web/middleware.func.004: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/recoverer.go:29 (0x55dd23)
    com/zenazn/goji/web/middleware.func.007: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/logger.go:27 (0x55d8ec)
    com/zenazn/goji/web/middleware.func.002: h.ServeHTTP(lw, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware/request_id.go:68 (0x55dfdb)
    com/zenazn/goji/web/middleware.func.008: h.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/opt/go/src/github.com/zenazn/goji/web/middleware.go:54 (0x554ee5)
    com/zenazn/goji/web.(*cStack).ServeHTTP: s.m.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/web/mux.go:77 (0x555fff)
    com/zenazn/goji/web.(*Mux).ServeHTTP: stack.ServeHTTP(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1496 (0x4450f3)
    (*ServeMux).ServeHTTP: h.ServeHTTP(w, r)
/opt/go/src/github.com/zenazn/goji/graceful/middleware.go:46 (0x561b99)
    com/zenazn/goji/graceful.func.002: h.ServeHTTP(&fancyWriter{bw}, r)
/usr/lib/go/src/pkg/net/http/server.go:1220 (0x443c50)
    HandlerFunc.ServeHTTP: f(w, r)
/usr/lib/go/src/pkg/net/http/server.go:1597 (0x4458fe)
    serverHandler.ServeHTTP: handler.ServeHTTP(rw, req)
/usr/lib/go/src/pkg/net/http/server.go:1167 (0x4438a7)
    (*conn).serve: serverHandler{c.server}.ServeHTTP(w, w.req)
/usr/lib/go/src/pkg/runtime/proc.c:1394 (0x418d00)
    goexit: runtime·goexit(void)
2014/09/19 06:48:48.065102 �[30;1m[f061086f5cad/MZc8sXsCMT-000001] �[0mReturning �[31;1m500�[0m in �[32m199.234144ms�[0m

Does bamboo can support https protocol?

I have successful deploy the bamboo, the http instance are works like a charm. so i have a another request, how can i config https service?

service on Debian Wheezy

I built the pkg on Debian Wheezy and installed it successfully. However there is something wrong with the start script as my system tells me the files dont exist how do I get the service to start?

HAProxy Restart coordination

Bamboo does not detect/receive modification from marathon

Hello.

I'm testing bamboo on a 1 master/3 slaves mesos cluster. Marathon, zk, mesos master, haproxy and bamboo are all on master server.

When I scale up or down a app in Marathon, I have to clic on "update" button in bamboo ui for bamboo to update haproxy configuration. The number of instances is reflected correctly, but the log does not show anything, and haproxy is not reconfigured.

Here is the production.json:
{
"Marathon": {
"Endpoint": "http://localhost:8080",
"Zookeeper": {
"Host": "localhost:2181",
"Path": "/marathon/state",
"ReportingDelay": 5
}
},

"Bamboo": {
"Endpoint": "http://localhost:8000",
"Zookeeper": {
"Host": "localhost",
"Path": "/marathon-haproxy/state",
"ReportingDelay": 5
}
},

"HAProxy": {
"TemplatePath": "/root/gopath/src/github.com/QubitProducts/bamboo/config/haproxy_template.cfg",
"OutputPath": "/usr/local/etc/haproxy/haproxy.cfg",
"ReloadCommand": "read PIDS < /var/run/haproxy.pid; haproxy -f /usr/local/etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
},

"StatsD": {
"Enabled": false,
"Host": "localhost:8125",
"Prefix": "bamboo-server.development."
}
}

Is there any other configuration to debug te (marathon | zk) /bamboo communication?

Thanks.

Auto create missing path in ZK setting or print a meaningful error message

chore: move dependencies to Godep and commit it to the repository

It's better to keep a local copy of the dependency source code in Godep.
this ensure the build process is consistent and independent of the remote changes.

Make sure test and build process still work

Healthcheck improvments

Support multiple healthcheck per app
Support TCP, HTTP, COMMAND check specified in Marathon
Try not re-invent the wheel by getting health check information from marathon

Also related to #56 #62

Lumberjack "Logger" struct giving build error

Hi there.

When building bamboo.go, I got a build error on line 152(Filename: logPath)
According to these docs there is no "Filename".
Changing "Filename" to "Dir" fixed the error for me :)

haproxy not updated

I run the latest (as of today) bamboo version and run into the following issue:

I update my marathon app (scaling or other changes, tried various combinations)
Changes show up on http://localhost:8000/api/state
haproxy does not get updated
I receive 503 Service Unavailable
I change ACL in UI and change it back
haproxy is now updated

config:

{
  "Marathon": {
    "Endpoint": "http://localhost:8080"
  },

  "Bamboo": {
    "Endpoint": "http://10.218.146.232:8000",
    "Zookeeper": {
      "Host": "10.218.146.232:2181",
      "Path": "/marathon-haproxy/state",
      "ReportingDelay": 5
    }
  },

  "HAProxy": {
    "TemplatePath": "/opt/bamboo/config/haproxy_template.cfg",
    "OutputPath": "/etc/haproxy/haproxy.cfg",
    "ReloadCommand": "PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
  },

  "StatsD": {
    "Enabled": false,
    "Host": "localhost:8125",
    "Prefix": "bamboo-server.development."
  }
}

Allow automatic configuration of Host based routing

This is dependent on this issue in marathon: mesosphere/marathon#599

It would be great if I could configure Bamboo to automatically configure apps to hostnames via specifying metadata to Marathon.

For example, if I could specify a tag in Marathon of bamboo.hostname: my-app.example.com and have bamboo automatically configure that application when it sees it.

This would allow us to automatically configure and provision the load balancing for services based off of metadata for the application itself, rather than making a separate api call to provision load balancing.

bamboo - newbie question

i am reading alot about mesos and marathon in specific since i'm interested in using mesos for managing long lived micro services. I would like to know what bamboo gives that mesos / marathon in it's latest version does not provide out of the box.

also - do you think that this project could be converged into marathon at some point or do you see this project continues to grow side by side.

finally - what is the current status of bamboo (known production installations etc) and what is your roadmap?

thanks for the very well and organised documentation!

Chen.

Support DNS configuration without DNS provider

This avoids TTL for some cases when DNS provider is less helpful.

URI encoded marathon application ids are double encoded

The README about the REST API says that marathon application ids have to be URI encoded. The given examples in the documentation are actually double encoded. I’ve tried both, single encoded and double encoded marathon app ids, the REST API only works with double encoded ids.

Example from documentdation: curl -i -X DELETE http://localhost:8000/api/services/%252Fapp-1
Decoded: http://localhost:8000/api/services/%2Fapp-1
Decoded again: http://localhost:8000/api/services//app-1

Marathon timeout during HTTP POST to bamboo on bigger installations

The http handler of Bamboo does a lot of work, e.g. querying Zookeeper, reloading haproxy. This can take seconds on non-trivial installations.

Marathon sometimes sends a series of events to Marathon, each with its own POST request. This obviously times out (timeout 10 sec in Marathon). Hence, we are easily loosing events in Bamboo.

Add a UI button to show the current HAproxy config

Also a UI element to show if HAProxy is running on localhost (green == process started) ?

Allow multiple template file rendering

In my use case i use bamboo for update HAproxy and for Redis failover.
Today i need to have one instance of Bamboo for HAproxy and one instance of Bamboo for redis failover.
It will be nice if i can use only once instance of Bamboo to udpate HAproxy and Redis failover.

Support loadbalancing strategy configuration via UI

design storage
design UI

Multiple HAProxy Instances Started on Deployment

Using Mesos 0.20.1 and Marathon 0.7.0 I commonly see multiple HAProxy instances after a deployment event. It looks like Marathon doesn't batch events so Bamboo ends up restarting HAProxy multiple times in rapid succession. This leads to HAProxy not being able properly reload and thus multiple instances.

bamboo to read marathon endpoint from zookeeper

Is it possible to define MARATHON_ENDPOINT by specifying zookeeper path like you can when running for example mesos frameworks?

Path could be:
zk://10.0.1.45:2181,10.0.1.47:2181,10.0.1.46:2181/marathon

run bamboo in supervisord to recover from fails

Hi,

here is where I am with adding supervisord to docker.

jwerak@bf662e5

Currently I am just triggering bamboo bin and I suppose it runs haproxy.

Is there a way we could run haproxy separately as another target in supervisord.conf?

Haproxy reload error output not verbose enough and misleading

The output on an haproxy reload error is the following:

2014/10/20 13:02:31.794232 exit status 1
2014/10/20 13:02:31.794513 HAProxy: Configuration updated

The exit status has no context and the second line is plain wrong.

Question: about deploying apps with bamboo

How does it handle the a/b deploys with marathon? Does it wait for all the new nodes to be spun up the updated or does it do bleeding deploys?

Support Marathon 0.7

Utilize /v2/tasks JSON format API

qzk.sinkSelfEvents panic

I'm not quite sure why or what the error is but I'm getting a panic in qzk.sinkSelfEvents:

goroutine 30 [chan receive]:
github.com/samuel/go-zookeeper/zk.(*Conn).request(0xc210068750, 0xc200000004, 0x641ea0, 0xc2100598e0, 0x641ee0, ...)
    /tmp/src/github.com/samuel/go-zookeeper/zk/conn.go:595 +0x8f
github.com/samuel/go-zookeeper/zk.(*Conn).GetW(0xc210068750, 0xc210059e80, 0x1b, 0x0, 0x0, ...)
    /tmp/src/github.com/samuel/go-zookeeper/zk/conn.go:638 +0x15f
github.com/QubitProducts/bamboo/qzk.func·004()
    /tmp/src/github.com/QubitProducts/bamboo/qzk/qzk.go:82 +0x5d
created by github.com/QubitProducts/bamboo/qzk.sinkSelfEvents
    /tmp/src/github.com/QubitProducts/bamboo/qzk/qzk.go:98 +0xfb

I've been trying with different Zookeeper settings and tried to purge all data in both /marathon as well as /bamboo (/bamboo/state is set as DomainMapping path).

Do you have any pointers or can I provide anything else to help investigating?

Thanks!

Missing app in Marathon

I have trouble adding app to bamboo from gui.

I've added app called /luigi-test and it says Missing app in Marathon.

I am running bamboo as docker container with command:

/usr/bin/docker run --rm --name bamboo -p 8000:8000 -p 80:80 -e MARATHON_ENDPOINT=10.0.1.149:8081 -e BAMBOO_ENDPOINT=10.0.1.151:8000 -e BAMBOO_ZK_HOST="10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181" -e BAMBOO_ZK_PATH=/bamboo quay.io/appuri/bamboo -bind=":8000" -config="config/production.example.json"

docker container logs:
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56 Starting update loop
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052755 Using environment override MARATHON_ENDPOINT=10.0.1.149:8081
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052823 Using environment override BAMBOO_ENDPOINT=10.0.1.151:8000
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052891 Using environment override BAMBOO_ZK_HOST=10.0.1.149:2181,10.0.1.151:2181,10.0.1.150:2181
Nov 01 22:08:56 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/01 22:08:56.052949 Using environment override BAMBOO_ZK_PATH=/bamboo

When app added:
From bamboo logs:
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653968 Domain mapping: Stated changed
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.653983 Queuing an haproxy update.
Nov 02 06:05:37 ip-10-0-1-151.us-west-2.compute.internal bash[16882]: 2014/11/02 06:05:37.655489 HAProxy: Same content, no need to reload

From marathon, only relevant info I've found:

Nov 02 06:06:29 ip-10-0-1-149.us-west-2.compute.internal bash[6143]: [2014-11-02 06:06:29,066] INFO 10.0.1.151 - - [02/Nov/2014:06:06:29 +0000] "GET /v2/tasks HTTP/1.1" 200 252 "-" "curl/7.26.0" (mesosphere.chaos.http.ChaosRequestLog:15)

I can add marathon callback and call to /v2/tasks returns following:
curl -X GET -H "Content-Type: application/json" 10.0.1.149:8081/v2/tasks
{"tasks":[{"appId":"/luigi-test","id":"luigi-test.70583548-5f32-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31001],"startedAt":"2014-11-02T06:07:11.580Z","stagedAt":"2014-10-29T06:11:36.580Z","version":"2014-10-29T06:11:36.379Z","servicePorts":[9001]},{"appId":"/marathon-tcp-haproxy-dev","id":"marathon-tcp-haproxy-dev.9e92e270-606f-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31187],"startedAt":"2014-11-02T06:07:11.602Z","stagedAt":"2014-10-30T20:02:04.602Z","version":"2014-10-29T09:20:19.048Z","servicePorts":[30000]},{"appId":"/demo-dev","id":"demo-dev.f56bbeb8-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-149.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.563Z","stagedAt":"2014-10-28T21:25:37.088Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]},{"appId":"/demo-dev","id":"demo-dev.f69d39da-5ee8-11e4-8166-56847afe9799","host":"ip-10-0-1-151.us-west-2.compute.internal","ports":[31000],"startedAt":"2014-11-02T06:07:11.572Z","stagedAt":"2014-10-28T21:25:39.089Z","version":"2014-10-28T21:25:34.948Z","servicePorts":[9000]}]}%

I am using mesos 0.20.1
marathon 0.7.3
latest bamboo

Do you have any advice on how I could debug this further?

haproxy zombies in docker container

Every reload cmd leaves the previous haproxy process as zombie. Because neither bamboo nor docker waits for them, they don't go away.

In a real unix system (if bamboo is installed as package) init is waiting for the exited processes and this doesn't happen.

no change in marathon but haproxy reloads

Hi,

haproxy reloads every 30 seconds even if there are no changes in marathon.

2014/09/11 13:24:45 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:24:50.935589 Marathon: State changed
2014/09/11 13:24:51.104599 HAProxy: Configuration updated
2014/09/11 13:24:51.142967 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:25:15 qzk.go:51: Got event. Delaying post
2014/09/11 13:25:15 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:25:20.936585 Marathon: State changed
2014/09/11 13:25:21.102402 HAProxy: Configuration updated
2014/09/11 13:25:21.135208 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:25:45 qzk.go:51: Got event. Delaying post
2014/09/11 13:25:45 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:25:50.933389 Marathon: State changed
2014/09/11 13:25:51.095557 HAProxy: Configuration updated
2014/09/11 13:25:51.132177 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)
2014/09/11 13:26:15 qzk.go:51: Got event. Delaying post
2014/09/11 13:26:15 qzk.go:55: No further debouncing. Posting event
2014/09/11 13:26:20.933709 Marathon: State changed
2014/09/11 13:26:21.107363 HAProxy: Configuration updated
2014/09/11 13:26:21.144372 Exec cmd: haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $(cat /var/run/haproxy.pid)

Best,

Mike

serviceports aren't available in templates

Our current setup involves running each app on a standard port on the haproxy layer
(this is how marathons old haproxy_cfg used to generate configs).

Apps are deployed with a specific port as a way to 'tag' them for a given service.

Would be useful to expose that port for Bamboo templates too
(I notice Bamboo currently only exposes 1 port for tasks so I propose to keep that behaviour).

bamboo not using service ports

I think current bamboo implementation does not take into account tasks service port(s).

Let me demonstrate it on one example with 2 exposed service ports:
Lets say I have following marathon app:

{
    "id": "marattest-dev",
    "cmd": "curl -q -o /app.py https://gist.githubusercontent.com/veverjak/b8607526f071f296bd8c/raw/3545a22d421d1f041ca5a8b305ea00f7edf967c6/get_file_content.py && python /app.py /etc/ip",
    "instances": 3,
    "cpus": 0.1,
    "mem": 10,
    "constraints": [],
    "uris": ["/.dockercfg"],
    "taskRateLimit": 1,
    "container": {
      "type": "DOCKER",
      "volumes": [
            {
                "containerPath": "/var/lib/scripts/create_configs.sh",
                "hostPath": "/var/lib/scripts/create_configs.sh",
                "mode": "RO"
            },
            {
                "containerPath": "/usr/bin/etcdctl",
                "hostPath": "/usr/bin/etcdctl",
                "mode": "RO"
            },
            {
                "containerPath": "/etc/ip",
                "hostPath": "/etc/ip",
                "mode": "RO"
            }
      ],
      "docker": {
        "image": "quay.io/appuri/get_file_content",
        "portMappings": [
          { "containerPort": 5000, "hostPort": 0, "servicePort": 9001, "protocol": "tcp" },
          { "containerPort": 5001, "hostPort": 0, "servicePort": 9002, "protocol": "tcp" }

        ],
        "network": "BRIDGE"
      }
    }
}

result for /v2/tasks gives:

{
  "tasks": [
    {
      "appId": "/marattest-dev",
      "id": "marattest-dev.46b9ab9c-6738-11e4-b85e-1645b8de8fb1",
      "host": "ip-10-0-1-151.us-west-2.compute.internal",
      "ports": [
        31003,
        31004
      ],
      "startedAt": "2014-11-08T11:38:08.063Z",
      "stagedAt": "2014-11-08T11:13:33.038Z",
      "version": "2014-11-08T11:13:29.557Z",
      "servicePorts": [
        9001,
        9002
      ]
    },
    {
      "appId": "/marattest-dev",
      "id": "marattest-dev.c2f5c97a-6736-11e4-b85e-1645b8de8fb1",
      "host": "ip-10-0-1-149.us-west-2.compute.internal",
      "ports": [
        31006,
        31007
      ],
      "startedAt": "2014-11-08T11:38:08.051Z",
      "stagedAt": "2014-11-08T11:02:42.474Z",
      "version": "2014-11-08T11:02:39.555Z",
      "servicePorts": [
        9001,
        9002
      ]
    },
    {
      "appId": "/marattest-dev",
      "id": "marattest-dev.4752421f-6738-11e4-b85e-1645b8de8fb1",
      "host": "ip-10-0-1-149.us-west-2.compute.internal",
      "ports": [
        31008,
        31009
      ],
      "startedAt": "2014-11-08T11:38:08.022Z",
      "stagedAt": "2014-11-08T11:13:34.037Z",
      "version": "2014-11-08T11:13:29.557Z",
      "servicePorts": [
        9001,
        9002
      ]
    },

If I am using default marathon bash script to create haproxy.cfg, I get:

listen marattest-dev-9001
  bind 0.0.0.0:9001
  mode tcp
  option tcplog
  balance leastconn
  server marattest-dev-3 ip-10-0-1-151.us-west-2.compute.internal:31003 check
  server marattest-dev-2 ip-10-0-1-149.us-west-2.compute.internal:31006 check
  server marattest-dev-1 ip-10-0-1-149.us-west-2.compute.internal:31008 check

listen marattest-dev-9002
  bind 0.0.0.0:9002
  mode tcp
  option tcplog
  balance leastconn
  server marattest-dev-3 ip-10-0-1-151.us-west-2.compute.internal:31004 check
  server marattest-dev-2 ip-10-0-1-149.us-west-2.compute.internal:31007 check
  server marattest-dev-1 ip-10-0-1-149.us-west-2.compute.internal:31009 check

If I want the same from bamboo, it generates:

## listen ::marattest-dev_0
##   bind *:0
##   mode http
##
##   balance leastconn
##   option forwardfor
##
##         server ::marattest-dev-ip-10-0-1-149.us-west-2.compute.internal-31006 ip-10-0-1-149.us-west-2.compute.internal:31006
##         server ::marattest-dev-ip-10-0-1-151.us-west-2.compute.internal-31003 ip-10-0-1-151.us-west-2.compute.internal:31003
##         server ::marattest-dev-ip-10-0-1-149.us-west-2.compute.internal-31008 ip-10-0-1-149.us-west-2.compute.internal:31008
#

It is because listen endpoint is created only per app and first app port is considered to be the only service port which I didn't define because I expected same behavior like default bash script has.

Workaround for me will be defining ports: [] per app and use only one port, which is fine for now, but I believe this can be easily fixed by adding proper structure in Create Apps.

bamboo is proxying even to hosts that are not started by marathon yet

When I scale my app up and there are 4 of 10 apps running bamboo creates config for all of them.
Currently I do not have health checks (that is how I've found out about this) and I am aware I should have those implemented.
IMO this could be handled by bamboo which would create haproxy configuration only for the apps which are currenly up.

Is this info available from marathon?
Is marathon sending callbacks about another app up?

HAProxy http_check configuration setting treats non-empty 404 errors as healthy indicator

Using a version of Bamboo that includes commit 6791f26, it comes with a change to the HAProxy configuration template backend section that treats a HTTP response with a non-empty body as a positive health signal. The configuration change in questions is this:

http-check expect rstring .*

What this means is that even if you specify an non-existing URI in the httpchk option leading to a 404 response, you will still end up with a health check considered valid if the response contains some string like Not Found. In fact, this is the exact scenario that we ran into using Bamboo.

I couldn't fully understand from the git commit message what the reason was to make the change, so I am uncertain how a proper fix would look like. The http-check regular expression could be extended to cover 2xx and 3xx responses (as in ^[23]\d{2}.*), but that seems no different to leaving out the entire parameter (as httpchkdefaults to considering 2xx/3xx as valid).

Maybe @activars (the commit author) can shed some light on what the intention for the change was in the first place.

Thanks!

unknown lumberjack.Logger field

Hi,

im new to go so maybe just a simple error on my side.

Downloaded all the git stated in bamboo.go (zookeeper etc.) and did go bamboo.go which results in:

# command-line-arguments
./bamboo.go:119: unknown lumberjack.Logger field 'Filename' in struct literal

Best,

Mike

When zookeeper goes away, bamboo dies

Not sure if bug, but when running haproxy in the same container as bamboo, haproxy dies too and all requests all terminated.
It'd be cool if bamboo waited for zookeeper to be available doing nothing with haproxy config in the meantime.
Second solution would be running haproxy in separate container. If that's preferred - feel free to close the issue.
Some logs: https://gist.github.com/gregory90/00658376bc08dbe979da

Bamboo on every mesos slave and Marathon callback

Not sure if bug, but...
When running Bamboo on every mesos slave should I have callback in Marathon for each instance of Bamboo?
I thought one callback will be enough, but updates doesn't seem to propagate to other instances.

Custom key/value map to specify custom attributes for template rendering

Currently, the mode needs to be hardcoded e.g. "mode tcp" or "mode http". Can we make it configurable by adding tags for a given service so that we can use "$services.mode" to get the value ?

We can probably store key/value map to make it generic e.g. $services.param["mode"] to make any key referenced in ha proxy template.

multiple master configuration

Hi,

sorry for posting this question, but I cannot get it to work properly:

I have a 3 master setup and the following config.json:

{
  "Marathon": {
    "Endpoint": "http://localhost:8080"
  },

  "Bamboo": {
    "Endpoint": "http://masterN:8000", # this is the local IP of this master
    "Zookeeper": {
      "Host": "master0:2181,master1:2181,master2:2181",
      "Path": "/marathon-haproxy/state",
      "ReportingDelay": 5
    }
  },

  "HAProxy": {
    "TemplatePath": "/opt/bamboo/config/haproxy_template.cfg",
    "OutputPath": "/etc/haproxy/haproxy.cfg",
    "ReloadCommand": "PIDS=`pidof haproxy`; haproxy -f /etc/haproxy/haproxy.cfg -p /var/run/haproxy.pid -sf $PIDS && while ps -p $PIDS; do sleep 0.2; done"
  },

  "StatsD": {
    "Enabled": false,
    "Host": "localhost:8125",
    "Prefix": "bamboo-server.development."
  }
}

explanation: as a bamboo endpoint I used each master's ip address, marathon endpoint is localhost and the zookeeper hosts are my three master ip addresses. Now my website switches between available and 503 in a millisecond manner. How do I make this setup work?

Order of acl rules

The web ui allows to define haproxy ACLs, but not the order they are applied, i.e. the order of use_backend lines in the config. I propose to add an order either by drag'n'drop in the UI or by a priority value in order to allow use cases like these:

route requests with a certain header to canary instances of an app
route unknown domains to a default app.

qubitproducts / bamboo Goto Github PK

bamboo's Issues

Recommend Projects

Recommend Topics

Recommend Org