grafana / carbon-relay-ng Goto Github PK

View Code? Open in Web Editor NEW

459.0 149.0 151.0 11.96 MB

Fast carbon relay+aggregator with admin interfaces for making changes online - production ready

License: Other

Makefile 1.26% CSS 0.02% Go 70.08% JavaScript 24.92% HTML 3.00% Dockerfile 0.05% Shell 0.13% Jsonnet 0.54%

monitoring graphite go carbon metrics

carbon-relay-ng's Introduction

carbon-relay-ng

A relay for carbon streams, in go. Like carbon-relay from the graphite project, except it:

performs better: should be able to do about 100k ~ 1M million metrics per second depending on configuration and CPU speed.
you can adjust the routing table at runtime, in real time using the web or telnet interface (this feature has rough edges and is not production ready)
has aggregator functionality built-in for cross-series, cross-time and cross-time-and-series aggregations.
supports plaintext and pickle graphite routes (output) and metrics2.0/grafana.net, as well as kafka, Google PubSub and Amazon CloudWatch.
graphite routes supports a per-route spooling policy. (i.e. in case of an endpoint outage, we can temporarily queue the data up to disk and resume later)
performs validation on all incoming metrics (see below)
supported inputs: plaintext, pickle and AMQP (rabbitmq)

This makes it easy to fanout to other tools that feed in on the metrics. Or balance/split load, or provide redundancy, or partition the data, etc. This pattern allows alerting and event processing systems to act on the data as it is received (which is much better than repeated reading from your storage)

Documentation

Concepts

You have 1 master routing table. This table contains 0-N routes. There's different route types. A carbon route can contain 0-M destinations (tcp endpoints)

First: "matching": you can match metrics on one or more of: prefix, substring, or regex. All 3 default to "" (empty string, i.e. allow all). The conditions are AND-ed. Regexes are more resource intensive and hence should - and often can be - avoided.

All incoming metrics are validated and go into the table when valid.
The table will then check metrics against the blocklist and discard when appropriate.
Then metrics pass through the rewriters and are modified if applicable. Rewrite rules wrapped with forward slashes are interpreted as regular expressions.
The table sends the metric to:
- the aggregators, who match the metrics against their rules, compute aggregations and feed results back into the table. see Aggregation section below for details.
- any routes that matches
The route can have different behaviors, based on its type:
- for grafanaNet / kafkaMdm / Google PubSub routes, there is only a single endpoint so that's where the data goes. For standard/carbon routes you can control how data gets routed into destinations (note that destinations have settings to match on prefix/sub/regex, just like routes):
- sendAllMatch: send all metrics to all the defined endpoints (possibly, and commonly only 1 endpoint).
- sendFirstMatch: send the metrics to the first endpoint that matches it.
- consistentHashing (older carbon consistent hashing behavior)/consistentHashing-v2 (experimental new behavior). (see config docs and PR 447for details)
- round robin: the route is a RR pool (not implemented)

carbon-relay-ng (for now) focuses on staying up and not consuming much resources.

For carbon routes:

if connection is up but slow, we drop the data
if connection is down and spooling enabled. we try to spool but if it's slow we drop the data
if connection is down and spooling disabled -> drop the data

kafka, Google PubSub, and grafanaNet have an in-memory buffer and can be configured to blocking or non-blocking mode when the buffer runs full.

carbon-relay-ng's People

Contributors

Stargazers

Watchers

Forkers

gregburek dieterbe mayan-it yo61 pauloconnor jpowers-ot shiaho vladimir-smirnov-sociomantic godeep comradin sergeyignatov rhoml civil inthecloud247 halcyonardency tbielawa gitbbrid transactcharlie throck anderender rtkwlf rtkrruvinskiy voxmedia dforste damienclaveau amjadshabani qnib chansolo henrypfhu ctrlok ptqa tg90nor aidanholsgrove anarcher tcolgate harpreet-sawhney pawellesniewski 0xd3adb33f ediezh tuomassalo rlodge cloudxtreme chenm11 alphagov dancech isavoy lolorc zillow directionless catap llaurent maggnus isavcic jnjackins qinguoan manticorecao bzed paulcapestany choffee otsuarez ehlerst popsikle guillaumeautran tehlers320 replay alrocky ad05bzag duologic optionalg yunstanford spheromak joemiller scrichar terraboops fengyunpan2 mexx77 teekaay hjdr4 woodsaj neurale olanna12 bodhi-space hotschedules teads alexsorokoletov chakra-coder mbethke frankegoesdown codelingobot korservick criteo-forks niteshjain smourapina shammishailaj vbichov meltwater-ateam hnakamur tjm2019 fullcontact vsreelasya

carbon-relay-ng's Issues

Any issue putting multiple carbon-relay-ng in a hardware load balancer?

I was thinking of standing up 2 carbon-relay-ng instances in a Citrix Netscaler VIP, pointing to 6 carbon daemons.

Would there be any problem with this configuration?

Currently I'm using the older, regular, carbon-relay.

First_only not working properly

In golang the iteration order for map is uncertain, even if it is certain, it will not the same as the order in configure file.

make sure browser doesn't cache old .js

after updating js code, recompiling and running, browser keeps using the old js.
this bit me for example in #26

post REST & SPA comments

#21 brought a revamped admin UI as a single page app, based on a rest api exposed by the go program. neat stuff, and it got merged.

however there's a few small remarks I'ld like to addressed:

when no fields are filled in, and you click "add route", it doesn't show error messages like the previous codebase did.
when a route comes back up, it stays red, until you reload the page. might be nice to automatically update the online state. (if it's not too much of a hassle)
removeRoute needs a comment explaining why we create a new map[string]string

cc @pwielgolaski

should ship with html templates built in

@pauloconnor that makes it easier to deploy
we should probably use https://github.com/jteeuwen/go-bindata for this
see https://www.youtube.com/watch?v=stFw0f3WEsk from 6:00 onwards, where this is explained.

Feature Request for consistent hashing

Hello,

with some regret I read you had to close the consistent hashing bug. I would like to open this one as an enhancement/feature request for this feature, as it is the one last thing that prevents me from using carbon-relay-ng instead of the original relay-agent.

Sadly Im not a good enough developer to fix this issue, instead I tried some of the existing packages for consistent-hashing and fed them the metrics from my graphite cluster (1M+ metrics), but sadly none of them distributed the metrics in such a way as the pyhton/graphite implementation does.

I had a look at:
"github.com/golang/groupcache/consistenthash"
"github.com/ngerakines/ketama"
"github.com/stathat/consistent"

I failed to make it run with this package, due to a missing understanding how to initialise it with my two backends: "github.com/dgryski/go-ketama".

If you know of more packages, Ill happily feed my metrics into them to check the distribution ;)

Kind regards,
Marcus

put on stdout which admin addresses (tcp and http) we're listening on

18:29:49.871118 carbon-relay-ng.go:349: initializing routes...
18:29:49.878296 carbon-relay-ng.go:378: listening on 0.0.0.0:2003
18:29:49.878684 routing.go:98: influxdb (re)connecting to 127.0.0.1:2004
18:29:49.878777 routing.go:98: carbon-default (re)connecting to out:2005
18:29:49.879632 routing.go:113: influxdb connected
18:29:49.879859 routing.go:113: carbon-default connected

if we have output like the above, we might as well mention http and tcp admin endpoints

when a route is down, other route will not receive any data

when I closed a route, but do not remove it from carbon-relay-ng, my other routes will not receive any data.

bug: shouldn't allow creating route with already existing key

Leaks memory

Relay seems to be leaking memory. It consumes all 2GB of RAM we gave him until OOMK kicks in.
Routing rules are pretty simple:

init = [
     "addBlack .net",
     "addRoute sendAllMatch core prefix=org.company  12.56.176.41:2014 spool=false pickle=true  12.56.176.42:2014 spool=false pickle=true",
     "addRoute sendAllMatch other prefix=stats  12.56.176.51:2014 spool=false pickle=true  12.56.176.52:2014 spool=false pickle=true"
]

build fail

I tried to build carbon-relay-ng as described in [1], but it just fails.
I tried it with ubuntu-precise and ubuntu-trusty in a vagrant box, both
did not work.

Here the command which fails:

root@vagrant-ubuntu-trusty-64:~/go/src/github.com/graphite-ng/carbon-relay-ng#
./make.sh
github.com/graphite-ng/carbon-relay-ng/admin
admin/admin-http.go:141: undefined: AssetDir

dependencies were installed without problems on trusty
root@vagrant-ubuntu-trusty-64:/go/src/github.com/graphite-ng/carbon-relay-ng#
ls $GOPATH
bin lib pkg src
root@vagrant-ubuntu-trusty-64:/go/src/github.com/graphite-ng/carbon-relay-ng#
ls /root/go/src/
github.com
root@vagrant-ubuntu-trusty-64:~/go/src/github.com/graphite-ng/carbon-relay-ng#
ls /root/go/src/github.com/
BurntSushi Dieterbe elazarl gorilla graphite-ng jteeuwen kisielk
rcrowley

I verified that 5a1a5f3 is the last commit that builds for me.

[1] https://github.com/graphite-ng/carbon-relay-ng#

I hope you can help me.

make: Nothing to be done for `all'.

I'm having problems installing on Ubuntu 14.04 with Go v1.2. Any help is greatly appreciated.

# export GOPATH=/tmp/foo
# export PATH=$PATH:$GOPATH/bin
# go get -d github.com/graphite-ng/carbon-relay-ng
# go get github.com/jteeuwen/go-bindata/...
# cd "$GOPATH/src/github.com/graphite-ng/carbon-relay-ng"
# git checkout v0.5
Note: checking out 'v0.5'.

You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.

If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
                                                                                                                                                                                                    [0/4613]
  git checkout -b new_branch_name

HEAD is now at fa7022b... document new releases / versioning approach
# make
make: Nothing to be done for `all'.

same endpoint addresses with spooling -> overlapping spool names cause spool issues

if you have the same address multiple times as endpoint (in the same or different routes), those are each an endpoint and conn but will use a spool file with the same name, which will conflict and cause all kinds of issues. like

2015/04/17 15:08:44 ERROR: diskqueue(spool_localhost_2005) failed to sync - rename spool/spool_localhost_2005.diskqueue.meta.dat.tmp spool/spool_localhost_2005.diskqueue.meta.dat: no such file or directory

long term, might want to use the same conn and spool for these multiple endpoints, but this would require more coordination logic. short term, we should just name them better, somehow.

multiple statsdaemon senders can cause issues

interesting situation happened yesterday:
we have 1 main statsdaemon, a whole bunch of collectd's (around 400), some diamonds, and some ssh tunnels writing metrics into the relay.
yesterday we wanted to run some local statsdaemons on some servers, instead of the central one, so they just took some existing metric traffic , didn't really add much new. just 7 more clients. somehow this caused our pipeline to block. not sure that carbon-relay-ng is at fault, but it feels that way :?
we do about 4k metrics/s with peaks at 20k/s when statsd flushes. presumably most clients send spread out over time, statsd sends on the minute "on the dot", maybe that's related

web authentication

Implementing a change like this is useful for protecting (non-admins) from changing configurations

/table hangs after aggregated stats are sent.

I'm using carbon-relay-ng with the following conf

init = [
     'addBlack interface-lo', 
     'addBlack cpu-',
     'addAgg avg ^(.*)$ 5m.$1 300 120',
     'addRoute sendAllMatch carbon-default  127.0.0.1:2004 spool=true pickle=false',
]

After a few minutes I see the 5 minute metrics sent, and after that the UI breaks because the /tables endpoint hangs. I grabbed a goroutine dump and got this:

goroutine 73 [running]:
runtime/pprof.writeGoroutineStacks(0x7f147d234d90, 0xc2080ba6e0, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:511 +0x8d
runtime/pprof.writeGoroutine(0x7f147d234d90, 0xc2080ba6e0, 0x2, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:500 +0x4f
runtime/pprof.(*Profile).WriteTo(0xa73d80, 0x7f147d234d90, 0xc2080ba6e0, 0x2, 0x0, 0x0)
    /usr/local/go/src/runtime/pprof/pprof.go:229 +0xd5
net/http/pprof.handler.ServeHTTP(0xc208ec3571, 0x9, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
    /usr/local/go/src/net/http/pprof/pprof.go:169 +0x35f
net/http/pprof.Index(0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
    /usr/local/go/src/net/http/pprof/pprof.go:181 +0x15e
net/http.HandlerFunc.ServeHTTP(0x921e48, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
    /usr/local/go/src/net/http/server.go:1265 +0x41
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
    /usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba6e0, 0xc20801f380)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba140)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 1 [chan receive, 11 minutes]:
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain.Wait(0x7f147d20bf90, 0xc2080404a8, 0xc208031ce0, 0x0, 0x0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain/goagain.go:204 +0x3d4
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain.AwaitSignals(0x7f147d20bf90, 0xc2080404a8, 0x0, 0x0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/rcrowley/goagain/legacy.go:13 +0x44
main.main()
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:264 +0x1593

goroutine 25 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209b92af0, 0x47, 0x47)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040048, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 6 [syscall, 11 minutes]:
os/signal.loop()
    /usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
    /usr/local/go/src/os/signal/signal_unix.go:27 +0x35

goroutine 8 [select, 11 minutes]:
github.com/graphite-ng/carbon-relay-ng/badmetrics.(*BadMetrics).manage(0xc20803cea0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/badmetrics/badMetrics.go:65 +0x715
created by github.com/graphite-ng/carbon-relay-ng/badmetrics.New
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/badmetrics/badMetrics.go:42 +0x178

goroutine 9 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cf800, 0x3e, 0x40)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.func·010()
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:52 +0xb8
created by main.NewTable
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:54 +0x3a0

goroutine 10 [chan send, 2 minutes]:
github.com/graphite-ng/carbon-relay-ng/aggregator.(*Aggregator).Flush(0xc2080c0480, 0x5536dbd4)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:137 +0x4c3
github.com/graphite-ng/carbon-relay-ng/aggregator.(*Aggregator).run(0xc2080c0480)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:180 +0x52e
created by github.com/graphite-ng/carbon-relay-ng/aggregator.New
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/aggregator/aggregator.go:107 +0x64a

goroutine 11 [select]:
github.com/graphite-ng/carbon-relay-ng/nsqd.(*DiskQueue).ioLoop(0xc208074160)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/nsqd/diskqueue.go:566 +0x6df
created by github.com/graphite-ng/carbon-relay-ng/nsqd.NewDiskQueue
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/nsqd/diskqueue.go:87 +0x409

goroutine 12 [chan receive, 11 minutes]:
main.func·006(0xc2087b01e0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/slowchan.go:12 +0x87
created by main.NewSlowChan
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/slowchan.go:17 +0xfe

goroutine 13 [chan receive]:
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics.(*meterArbiter).tick(0xa7da80)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics/meter.go:221 +0x52
created by github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics.NewMeter
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/Dieterbe/go-metrics/meter.go:40 +0x1f7

goroutine 14 [select, 11 minutes]:
main.(*Spool).Writer(0xc208068c60)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:84 +0x9d0
created by main.NewSpool
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:68 +0x5f2

goroutine 15 [select, 11 minutes]:
main.(*Spool).Buffer(0xc208068c60)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:117 +0x271
created by main.NewSpool
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/spool.go:69 +0x60f

goroutine 16 [select]:
main.(*Destination).relay(0xc208091e00)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/destination.go:258 +0x13ed
created by main.(*Destination).Run
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/destination.go:158 +0x3b5

goroutine 18 [IO wait]:
net.(*pollDesc).Wait(0xc208816300, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc208816300, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc2088162a0, 0x0, 0x7f147d20ad98, 0xc208dee740)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080404a8, 0xc208032df0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
main.accept(0xc2080404a8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:75 +0x33
created by main.main
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:240 +0x14e6

goroutine 19 [IO wait, 11 minutes]:
net.(*pollDesc).Wait(0xc2088164c0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2088164c0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc208816460, 0x0, 0x7f147d20ad98, 0xc208099d88)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080404b0, 0xc208020380, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net.(*TCPListener).Accept(0xc2080404b0, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:244 +0x4c
github.com/graphite-ng/carbon-relay-ng/telnet.ListenAndServe(0xc208032ea0, 0xc, 0x0, 0x0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/telnet/telnet.go:49 +0xea
main.adminListener(0xc208032ea0, 0xc, 0x0, 0x0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_telnet.go:108 +0x9bf
main.func·002()
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:252 +0x3c
created by main.main
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:257 +0x1534

goroutine 20 [IO wait]:
net.(*pollDesc).Wait(0xc20882ffe0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc20882ffe0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).accept(0xc20882ff80, 0x0, 0x7f147d20ad98, 0xc208dee7b8)
    /usr/local/go/src/net/fd_unix.go:419 +0x40b
net.(*TCPListener).AcceptTCP(0xc2080405e8, 0x4ae08e, 0x0, 0x0)
    /usr/local/go/src/net/tcpsock_posix.go:234 +0x4e
net/http.tcpKeepAliveListener.Accept(0xc2080405e8, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1976 +0x4c
net/http.(*Server).Serve(0xc2087b14a0, 0x7f147d20d1e8, 0xc2080405e8, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1728 +0x92
net/http.(*Server).ListenAndServe(0xc2087b14a0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1718 +0x154
net/http.ListenAndServe(0xc208032f40, 0xc, 0x0, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:1808 +0xba
main.HttpListener(0xc208032f40, 0xc, 0xc208034410)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:211 +0xaf3
created by main.main
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:261 +0x1573

goroutine 21 [chan receive]:
main.(*keepSafe).keepClean(0xc208828690)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/keepsafe.go:29 +0x77
created by main.NewKeepSafe
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/keepsafe.go:23 +0x162

goroutine 22 [IO wait, 11 minutes]:
net.(*pollDesc).Wait(0xc2088163e0, 0x72, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:84 +0x47
net.(*pollDesc).WaitRead(0xc2088163e0, 0x0, 0x0)
    /usr/local/go/src/net/fd_poll_runtime.go:89 +0x43
net.(*netFD).Read(0xc208816380, 0xc208834800, 0x400, 0x400, 0x0, 0x7f147d20ad98, 0xc208826298)
    /usr/local/go/src/net/fd_unix.go:242 +0x40f
net.(*conn).Read(0xc2080405f0, 0xc208834800, 0x400, 0x400, 0x400, 0x0, 0x0)
    /usr/local/go/src/net/net.go:121 +0xdc
main.(*Conn).checkEOF(0xc2087fa000)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:110 +0x9f
created by main.NewConn
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:92 +0xcba

goroutine 23 [select]:
main.(*Conn).HandleData(0xc2087fa000)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:172 +0x1901
created by main.NewConn
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:94 +0xcd4

goroutine 24 [select]:
main.(*Conn).HandleStatus(0xc2087fa000)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:149 +0x3bd
created by main.NewConn
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/conn.go:95 +0xcee

goroutine 26 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209b93680, 0x41, 0x41)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2080401d0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 27 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20dc0, 0x40, 0x40)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040810, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 28 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20fc0, 0x40, 0x40)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208040e98, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 29 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e21040, 0x3e, 0x3e)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fecb98, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 30 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094555e0, 0x41, 0x41)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fed248, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 31 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455900, 0x44, 0x44)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc208fed938, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 32 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094559f0, 0x42, 0x42)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6030, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 33 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20740, 0x3e, 0x3e)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6168, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 34 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e203c0, 0x37, 0x37)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6170, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 35 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e204c0, 0x3c, 0x3c)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6468, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 36 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455d60, 0x47, 0x47)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a65d0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 37 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc209455bd0, 0x4c, 0x4c)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a65d8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 38 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20a00, 0x38, 0x38)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2090a6990, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 40 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfa40, 0x38, 0x38)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc20987b338, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 41 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfa80, 0x3b, 0x3b)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209918a30, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 42 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfac0, 0x37, 0x37)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2099195e8, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 43 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfd00, 0x39, 0x39)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc2099bd930, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 44 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094cfe40, 0x3d, 0x3d)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209a57660, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 45 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc2094556d0, 0x42, 0x42)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b04770, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 46 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20140, 0x37, 0x37)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b9ab80, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 47 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e201c0, 0x40, 0x40)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36f38, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 48 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20a80, 0x37, 0x37)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36f20, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 49 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20c40, 0x3f, 0x3f)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c374b0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 50 [chan send, 2 minutes]:
main.(*Table).Dispatch(0xc208034410, 0xc208e20f40, 0x38, 0x38)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209b04bd0, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

goroutine 51 [chan send]:
main.(*Table).Snapshot(0xc208034410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:115 +0x3f8
main.listTable(0x7f147d234860, 0xc2080ba5a0, 0xc20801f110, 0x0, 0x0, 0xc20883dae8)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:57 +0x4d
main.handler.ServeHTTP(0x921c68, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:30 +0x55
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux.(*Router).ServeHTTP(0xc208035ef0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux/mux.go:98 +0x297
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
    /usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba5a0, 0xc20801f110)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba000)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 52 [chan send, 2 minutes]:
main.(*Table).Snapshot(0xc208034410, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:115 +0x3f8
main.listTable(0x7f147d234860, 0xc2080ba780, 0xc20801f2b0, 0x0, 0x0, 0xc20883fae8)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:57 +0x4d
main.handler.ServeHTTP(0x921c68, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:30 +0x55
github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux.(*Router).ServeHTTP(0xc208035ef0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/_third_party/github.com/gorilla/mux/mux.go:98 +0x297
net/http.(*ServeMux).ServeHTTP(0xc20803c7e0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
    /usr/local/go/src/net/http/server.go:1541 +0x17d
net/http.serverHandler.ServeHTTP(0xc2087b14a0, 0x7f147d234860, 0xc2080ba780, 0xc20801f2b0)
    /usr/local/go/src/net/http/server.go:1703 +0x19a
net/http.(*conn).serve(0xc2080ba0a0)
    /usr/local/go/src/net/http/server.go:1204 +0xb57
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:1751 +0x35e

goroutine 71 [chan send]:
main.(*Table).Dispatch(0xc208034410, 0xc20a2865c0, 0x3c, 0x3c)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/table.go:74 +0x2ec
main.handle(0xc209c36018, 0xc208032df0, 0xc, 0xc208032ea0, 0xc, 0xc208032f40, 0xc, 0xc208032ff0, 0x5, 0x0, ...)
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:123 +0x420
created by main.accept
    /Users/shanse1/code/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:80 +0xe8

Cancelling a deletion of a route deletes it anyway

Since the last merge, when I delete a route in the web-interface a press cancel, it gets deleted anyway.

Docker image for carbon-relay-ng

Hi,

I would just let you know that I have created a small (15.46 MB) Docker image for carbon-relay-ng (since I couldn't find one in the registry).

You can find it here:
https://registry.hub.docker.com/u/yasn77/docker-carbon-relay-ng/

Thanks for the great work!

Yasser

Concatenated entries

Hi,

I am using carbon-relay-ng and every once and a while I get a concatenated entry in my graphite database. The entry comes into carbon-relay-ng ok, but when it is forwarded to the carbon-cache server occasionally an entry becomes concatenated.

For example I get an entry like:

c01-01/:
cpuload
cputotals
ctxint
disktotals
inodeinc01-01
inodeinfo
meminfo
nettotals
pageinfo
swapinfo
tcpinfo

where the "indoeinc01-01" is wrong.

Below, 192.168.1.102 is the client. 192.168.1.101/10.0.0.1 is the carbon-relay-ng server (has multiple interfaces), and 10.0.0.6 is the carbon-cache server.

Watching with tcpflow, I see the packet come into carbon-relay-ng OK:

$ grep -a "c01-01.cpuload.avg5 0 1359480120" 192.168.001.102.39231-192.168.001.101.02003
c01-01.cpuload.avg5 0 1359480120

but when it goes out to the carbon-cache server it's not the same as when it came in:

$ grep -a inodeinc01-01 ./*
./010.000.000.001.60749-010.000.000.006.02003:c01-01.inodeinc01-01.cpuload.avg5 0 1359480120

Any thoughts on that? I'm not sure where to look next.

Thanks.

Please clarify 'zero-downtime restarts'

I understand from the README that "[carbon-relay-ng] can be restarted without dropping packets (needs testing)" but I am unclear on some things. Could you perhaps assist?

Exactly how does this zero-downtime restart work? Obviously you're not describing a 'service carbonrelayng restart' type scenario.
The README says this needs further testing, can you describe how to test this functionality?

Thanks!

Newline inserted after every pickled message

The code currently sends a newline after every message, even when the messages are pickled. The receiver doesn't expect this.

dropping packets to one endpoint can impact other endpoints (routes?)

with a config like

addRoute sendAllMatch carbon-default  host1:2005 spool=true  host2:2004 spool=true

when host1 becomes slow and starts dropping packets (seen by dest ... nonBlockingSend -> dropping due to slow conn messages and in grafana), sometimes host2 exhibits the same behavior, a drop in traffic at the same time and following the same pattern.
we must always make sure the pipeline can't hang, so i traced through the code (starting at table.Dispatch) and made sure none of the route/destpoint/conn code can block the processing. especially of importance is the Destination.relay loop. in there we actually have the explicit flush and shutdown handling which might take a while, but the first is seemingly never called, and the latter is definitely never called during runtime.
so i installed a profile hook by importing _ "net/http/pprof", loading up a relay and blasting it with input traffic to cause it to start dropping traffic, and then collected a profile:

clearly we spend too much time logging. for comparison, the other functions invoked by relay are around 0.01s ~ 0.04s.

my current thinking is:

we can get too busy executing logging logic (specifically, the "warning.. dropped" messages), causing the pipe to hang, which blocks delivery to other endpoints
in fact, it might even cause more dropped packets to the bad route

i've tried replacing these high-volume warning messages to info (which is appropriate, since we use it to trace what happens to individual metric messages), if we tell a loglevel below warning might cause a lot of traffic, we're good. so i ran again with warning level and tested again (this time using the /debug/vars2 because no more warn messages on screen). There's definitely a lot less time spent in logger.Info, but still some, and also significant time spent in logger.Debug. Not sure why the latter, as I changed to Info, or maybe because we now process so much traffic that the Debug stuff also starts adding up.

No tags/versions

The README specifies that there is a version v0.5 and that the master is a work-in-progress.

However, there are no git tags/releases in the repo or changelog.

How do we use the stable version?

can't import "routing" package at install

Following the readme to install :

go get github.com/graphite-ng/carbon-relay-ng

/opt/data/apps/go/src/github.com/graphite-ng/carbon-relay-ng/carbon-relay-ng.go:7:2: local import "./routing" in non-local package

Here is my Go version :

go env
GOARCH="amd64"
GOBIN=""
GOCHAR="6"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/opt/data/apps/go"
GORACE=""
GOROOT="/usr/local/go/bin"
GOTOOLDIR="/usr/local/go/bin/pkg/tool/linux_amd64"
TERM="dumb"
CC="gcc"
GOGCCFLAGS="-g -O2 -fPIC -m64 -pthread"
CXX="g++"
CGO_ENABLED="1"

go version
go version go1.2.2 linux/amd64

performance improvements

the last few commits (less allocations, faster path for disabled logging, RCU-backed table config instead of mutex based, amortize cost of field splitting for aggregators) have given me about 30% decrease in cpu usage. I wonder what your results are.

Here are some more ideas to shave off more cpu time:

less channels where we can avoid them, use Dispatch function instead of .in channel.
where we need channel api, buffer up metrics (use batches and/or buffered channels to amortize overhead). I believe buffered channels need less locking than sychronous? but batching often seems to need selects, which come with a cost too.
use RCU instead of locks in routing and perhaps elsewhere
instead of splitting/parsing metrics in multiple places (validation, aggregation, pickle encoder, ...) do it in 1 place and send the parsed/splitted version packed along with the buf, or something
aggregators: don't use regex, or auto figure out what substring we can apply before doing expensive regex match, or use pcre binding, or cache key->parse results, or use something else than regex. we could even check the match on the key in the buf without splitting the buf in fields, as bytes.Fields(buf) is pretty expensive
do matching before send to aggregator channel, though we don't want to break encapsulation too much (matcher.Match in caller would be ok, not so much regex)
lower the overhead of select in Destination.relay, Conn.HandleStatus and Conn.HandleData, perhaps also using RCU here

ready for more testing

just a message for all watchers that the master branch, which contains the new version, is ready for more testing! try it out and let us know what you think.

spooling is broken

endpoint went down, we spool to disk, endpoint comes back up, we unspool, but not correctly: some data gets lost as can be seen.
there's usually an interleaved pattern of data that makes it back, as can be seen here.
sometimes preceeded by a period of full data loss, sometimes not.

Can't build with latest master

When I follow the readme instructions and run make I get:

go-bindata admin_http_assets
go build
# github.com/graphite-ng/carbon-relay-ng
./carbon-relay-ng.go:110: undefined: metrics20.ValidatePacket
./conn.go:140: c.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
./conn.go:179: c.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
./destination.go:215: conn.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:93: s.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:102: s.numBuffered.Inc undefined (type metrics.Gauge has no field or method Inc)
./spool.go:121: s.numBuffered.Dec undefined (type metrics.Gauge has no field or method Dec)
make: *** [build] Error 2

Some kind of dependency issue.

Performance issue

When I run official code from master branch(commit dc9ebd0) with official configuration ./carbon-relay-ng carbon-relay-ng.ini
My CPU is getting crazy and consuming 100%.

I have no port open for endpoints defined in configuration, but still it should not consume 100% (it sounds like active loop).

I run it on mac.

spooling to 1 endpoint stops sending data to other endpoint

i had this when influxdb had an issue, the relay starts writing to spool, but it also didn't send to carbon anymore.

routes can block and hang the process

in the loop where we read from to_unspool and route.ch we call process_packet which can take a while to return (because we write to unbuffered conn, because of udp sends, and i think because of writing to diskqueue.
this is probably a cause of #32 and #33
we should decouple this and not block any of the routing.
https://github.com/tv42/topic/blob/master/topic.go#L42-53 is an interesting idea i'm taking inspiration from.
i'm working on a fix in next.

zero-downtime restart should keep routing table

routing table may have been altered via admin interface.

some metrics missing and/or duplicated in output stream

in everything-looks-good mode, still some metrics don't arrive at the destination, and sniffing traffic reveals that some seem to be duplicated. possibly related to the bufwriter, should try without that and see.

bogus "connection refused" error when trying to edit route

had carbon on port 2003 with route pointing to it,
restarted carbon to listen on port 2005
try to use web UI to update the route with port 2005, but it gave me the message "something something (probably the route): connection refused"
this should have worked. maybe because 2003 was now connection refused?

http UI doesn't accept hostname/dns for addr

@pauloconnor currently it checks with a regex for ip:port, and doesn't let me submit anything else
it should also support hostnames and dns names instead of just an ip.

no pickle option in webUI

I can only change the config file and restart carbon-relay

go get -> undefined Asset

Hello,

i have this message when i try to install carbon-relay-ng

go get github.com/graphite-ng/carbon-relay-ng

github.com/graphite-ng/carbon-relay-ng

/opt/golang/bin/src/github.com/graphite-ng/carbon-relay-ng/templates.go:17: undefined: Asset

Regards,
Wilfrid

reworking routing for performance and being more powerful

Hi everybody.
I'ld like to hear your thoughts on this. 1 and 2 are fairly obvious, but 3 could really use input on how you are using, or want to use the relay.

regex has much overhead. I don't think much can be done about this, so we should try to avoid regex matching when we can. I'm thinking of introducing "contains" and "starts_with" options for string checks in addition to regex, so that in many cases we could use those instead of regex and it will perform much better.
specifically, if the regex pattern is "", we shouldn't even do the regex Match() like we currently do.
I've noticed that first_only, while useful for some setups (see E below), prevents us from "also sending traffic elsewhere". so i started thinking about getting rid of first_only and making the routing/dispatching more powerful. that said, it's a neat mechanism because you can make routing decisions with just a bool check, no pattern checking needed.
so, taking a step back, i'ld like to collect use cases of how people want to use carbon-relay-ng, especially since i know some people are interested (and working on) sharding/consistent hashing, and round robin, and i've been pondering how to keep routing/dispatching performant, yet viable for various/all use cases. (ideally, without running multiple carbon-relay-ng instances if we can)

I see the following use cases

A consistent hashing / round robin to a pool of servers, for loadbalancing and HA (each pool is just one route)
B mirroring the same load to multiple machines, possibly a subset of traffic using regex pattern (for example to test influxdb next to graphite)
C sending a subset of metrics to anomaly detection
D sending all or some - of the metrics to carbon-tagger
E poor man's load balancing: sending certain metrics to server A, some others to B and maybe some others to C, using regexes to control what goes where (maybe some storage is faster then others), and using first_only to make sure we don't store metrics on more than one system.

am i missing a case? first_only is really only used for E, but it doesn't really work anymore as soon as you want to send some metrics also elsewhere like in B, C or D.

What I'm thinking is, to keep the system powerful, and still efficient, without overcomplicating things too much, we should implement E as a multi-endpoint route just like we are approaching round robin and hashing/sharding. the key is that these are multiple endpoints within one route, instead of multiple routes in the "global level"
this way, on the global routing/dispatching level, we check the pattern (and string check) of every route, and if it matches, the route gets it. but a route can be a special one with multiple endpoints,
and send traffic to those endpoints using a dedicated mechanism:

round robin
sharded
further pattern matching within the route, with first_only

thoughts?
cc @robinbowes @rcrowley @pauloconnor @willowpet @pwielgolaski @shiaho @vladimir-smirnov-sociomantic @prune998 @curtisgithub

expose [route] performance metrics

ideally we're able to expose them via expvars, and also as a carbon stream.
rcrowley/go-metrics#68 would be nice for that.

once we have the expvars, we can also show them in the web ui.

not go-gettable

Fails to build (go install github.com/graphite-ng/carbon-relay-ng) with messages:

src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:205: undefined: Asset
src/github.com/graphite-ng/carbon-relay-ng/admin_http.go:205: undefined: AssetDir

destination shows red in UI even though it's up and working fine

http UI should display whether spooling is enabled for a route

master fails to build

github.com/graphite-ng/carbon-relay-ng

./admin_http.go:151: undefined: Asset
./admin_http.go:151: undefined: AssetDir

Pickle listener

Is there support for listners for plain text and pickle on relay-ng? I currently have things that send in both to an ordinary Carbon relay daemon which listens on both 2003/2004. I can see that I can send to pickle port as well so I presume that there is support for plain/pickle on the single listen address?

Routes list is not in any order

There is map used for storing routes, but as it is stated here http://blog.golang.org/go-maps-in-action
"Iteration order" is not guaranteed to be the same from one iteration to the next. I wonder if it makes sense to make it more under control, especially when I want to forward only to first matching.

configuration for active/passive clusters.?

I've been reading configuration rules and I can understand how can't I define a group of 2 graphite-carbon servers working together in mode active/passive, something like.

clusterA = serverA_1:2003 (active) , serverA_2:2003 (passive)
clusterB = serverB_2:2003 (active), serverB_2:2003 (passive)

where I should send all metrics matching a regex "^metricsA.*" only to the first server (active) or the second one if primary fails ( passive) .

something like:

match "^metricsA.*" send to ClusterA
match "^metricsB.*" send to clusterB

addRoute sendAllMatch regex="^metricsA.*"  serverA_1:2003  serverA_2:2003
addRoute sendAllMatch regex="^metricsB.*"  serverB_1:2003  serverB_2:2003

I think there is no way to indicate a active/passive balancing mode , isn't it?

building error looking for go-bindata

following your instructions there is an error when compiling , it seems like path is not correct or can not find go-bindata

root@nono:~/go/src/github.com/graphite-ng/carbon-relay-ng# make
go-bindata admin_http_assets
make: go-bindata: No se encontró el programa
make: *** [build] Error 127

How can I fix this problem?

http UI incorrectly says "listening on ..." when it failed to listen

i had this happen, port was already in use by other app, carbon-relay-ng still said the http admin ui was listening. this is because we ignore the error from http.ListenAndServe which we should of course, not do.
@pauloconnor

Error when building

When building latest version of carbon-relay-ng like mentioned in the readme I get this error when running ./make.sh:

routing/routing.go:153: undefined: parseDataPoint

I tried building on CentOS 6.5 and MacOSX 10.9.5

panic: runtime error: invalid memory address or nil pointer dereference

in relay() when c is nil, we print the error and sleep, and then proceed with a write on the c which is still nil, causing a panic.

full output here:
(note to self: influx:2003 was down at first but than came up, but that shouldn't be related to this)

[dieter.plaetinck@dfvimeographite3 ~]$ ./carbon-relay-ng -l="0.0.0.0:2005" "=:2015" "=influx:2003"
17:06:05.408269 carbon-relay-ng.go:128: listening on 0.0.0.0:2005
17:06:05.409048 carbon-relay-ng.go:156: dial tcp 10.90.128.10:2003: connection refused
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x0 pc=0x402583]

goroutine 6 [running]:
runtime.panic(0x53c620, 0x6d7928)
        /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.write(0xc2100504e0, 0x51, 0x51, 0x0, 0x51)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:181 +0x233
main.relay(0xc210050180, 0xc21001e5d0)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:165 +0x1ef
created by main.main
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:115 +0x559

goroutine 1 [chan receive]:
github.com/rcrowley/goagain.Wait(0x7f95609f6440, 0xc210000130, 0x7f9560855da8, 0x401c65, 0x20)
        /home/dieter/go/src/github.com/rcrowley/goagain/goagain.go:204 +0x222
github.com/rcrowley/goagain.AwaitSignals(0x7f95609f6440, 0xc210000130, 0x0, 0x0)
        /home/dieter/go/src/github.com/rcrowley/goagain/legacy.go:13 +0x43
main.main()
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:143 +0x8e5

goroutine 3 [syscall]:
os/signal.loop()
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:21 +0x1e
created by os/signal.init·1
        /usr/lib/go/src/pkg/os/signal/signal_unix.go:27 +0x31

goroutine 4 [chan receive]:
main.relay(0xc2100500c0, 0xc21001e480)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:164 +0x1c3
created by main.main
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:115 +0x559

goroutine 5 [syscall]:
runtime.goexit()
        /usr/lib/go/src/pkg/runtime/proc.c:1394

goroutine 7 [IO wait]:
net.runtime_pollWait(0x7f95609f61d8, 0x72, 0x0)
        /usr/lib/go/src/pkg/runtime/netpoll.goc:116 +0x6a
net.(*pollDesc).Wait(0xc210051300, 0x72, 0x7f95609f4f88, 0xb)
        /usr/lib/go/src/pkg/net/fd_poll_runtime.go:81 +0x34
net.(*pollDesc).WaitRead(0xc210051300, 0xb, 0x7f95609f4f88)
        /usr/lib/go/src/pkg/net/fd_poll_runtime.go:86 +0x30
net.(*netFD).accept(0xc2100512a0, 0x5af3e8, 0x0, 0x7f95609f4f88, 0xb)
        /usr/lib/go/src/pkg/net/fd_unix.go:382 +0x2c2
net.(*TCPListener).AcceptTCP(0xc210000130, 0xc21004a040, 0x2, 0x2)
        /usr/lib/go/src/pkg/net/tcpsock_posix.go:233 +0x47
main.accept(0xc210000130, 0xc21004a040, 0x2, 0x2)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:34 +0x27
created by main.main
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:134 +0x8c5

goroutine 9 [runnable]:
main.dispatch(0xc2100504e0, 0x51, 0x51, 0xc21004a040, 0x2, ...)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:48 +0xfd
main.handle(0xc210000188, 0xc21004a040, 0x2, 0x2)
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:78 +0x409
created by main.accept
        /home/dieter/workspaces/eclipse/carbon-relay-ng/carbon-relay-ng.go:39 +0xf7

grafana / carbon-relay-ng Goto Github PK

carbon-relay-ng's Introduction

carbon-relay-ng

Documentation

Concepts

carbon-relay-ng's People

Contributors

Stargazers

Watchers

Forkers

carbon-relay-ng's Issues

github.com/graphite-ng/carbon-relay-ng

github.com/graphite-ng/carbon-relay-ng

Recommend Projects

Recommend Topics

Recommend Org