readytalk / metrics-statsd Goto Github PK

gauge will overwrite old values in statsd which make distributed counters impossible

Add Documentation to README to explain what this library is for (interop with statsd for systems in mixed environments) and why it reports everything as gauges (seeing Issues #21 #20 #15, organicveggie#6, organicveggie#4).

Metrics has changed packages to io.dropwizard.metrics. We should update and create a new version to take advantage of changes in the new line.

Hi,

Please help to understand an issue I have seen.
My app collects timers, counters, and meters and reports them through different reporters.
if the graphite reporter is used directly, the collected metrics are reported to graphite as the correct metric types.
But if I use the statsd reporter, all the collected metrics are reported to statsd as gauges, e.g. all the message to statsd end with "|g". This is confirmed from the statsd logs.
Could anyone help me to resolve the problem? Thanks.

Hi, at totally random times I get this exception in one of my production machines:

ERROR com.codahale.metrics.ScheduledReporter - Exception thrown from StatsDReporter#report. Exception was suppressed. java.lang.IllegalArgumentException: unresolved address at java.net.DatagramPacket.setSocketAddress(DatagramPacket.java:316) at java.net.DatagramPacket.<init>(DatagramPacket.java:144) at java.net.DatagramPacket.<init>(DatagramPacket.java:178) at com.readytalk.metrics.DatagramSocketFactory.createPacket(DatagramSocketFactory.java:30) at com.readytalk.metrics.StatsD.send(StatsD.java:98) at com.readytalk.metrics.StatsDReporter.reportGauge(StatsDReporter.java:252) at com.readytalk.metrics.StatsDReporter.report(StatsDReporter.java:175) at com.codahale.metrics.ScheduledReporter.report(ScheduledReporter.java:162) at com.codahale.metrics.ScheduledReporter$1.run(ScheduledReporter.java:117) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

I captured with tcpdump the DNS packets to see where is the failure comes from, and I found a small lag from the dns server with resolving the name of the statsd server, I replaced the configuration to IP address instead of hostname and once again this error occured, this time I could see that it's trying to reach AWS hostname of the statsd server for the ip:

x.x.x.x

it is

ip-x-x-x-x.eu-west-1.compute.internal
It's important to mention I'm running on AWS EC2 m5 type machines and using Route53 private-zones as DNS server.
no idea how it infers this IP, any thoughts how can I suppress this error ?

1. Can add retries to the dns server in order to not throw after first try and retry the dns name resolution?
2. Can I disable the name resolve and communicate via IP only ?

The default settings of statsite is that it computes various percentiles, max, min etc.

The problem is that metrics-statsd sends also various percentiles to StatsD/statsite and so you end with percentiles of percentiles in Graphite.

It would be nice to be able to configure the reporter to send raw data instead of aggregated.

I could prepare the pull request if you would mind accepting it.

Btw. our use-case is that we sends application metrics from more instances of the application. These instances are running in a cloud so we don't have fixed identifiers of the running instances. We also don't want to add a random hash to metric name because it would cause problems in Whisper (DB of Graphite) - it doesn't like higher number of files.

The idea here is to support a "suffixedWith" option, which works similarly to "prefixedWith", for situations where it is necessary to append something to the names of metrics being reported to statsd.

PR: #41

Hi, can you enable maven central sync so we don't have to add a non-default repository to our builds?

Instructions can be found here
https://bintray.com/docs/usermanual/uploads/uploads_syncingartifactswithmavencentral.html

Thanks!

The StatsD class is erroneously formatting all values as gauge type (|g). It should follow the protocol and format the values appropriately.

Would it be possible to get the bintray project added to jcenter or sonatype/maven central? I suspect that jcenter will be less depression-inducing... :)

Double.NaN is not handled. It is probably a bug in Dropwizard Metrics, but metrics-statsd should anyhow strictly enforce statsd message format.

Bad line: NaN,g in msg "foo:NaN|g

Because of the peculiarity with how graphite interprets anything that ends with ".count" we should save users from themselves and switch that for ".samples" when we see it at the end of a leaf name that will be reported directly.

Version: "com.readytalk" % "metrics3-statsd" % "4.1.0"

The wire format of counter metrics is being sent to statsD with a "|g" suffix. It's meant to be a "|c". This blog details the proper format for counters:
http://blog.johngoulah.com/2012/10/looking-under-the-covers-of-statsd/

Is this intended? The "|g" can be seen in the send() method here:
https://github.com/ReadyTalk/metrics-statsd/blob/master/metrics-statsd-common/src/main/java/com/readytalk/metrics/StatsD.java