Sometimes I see that value in the QueryAttribute::collectMetrics is the instance of the LinkedHashMap instead of CompositeData or Number/String/Date and jmxtrans is unable to parse it.

Hi,
it is possible to define an alias name for an attribute in the json configuration, for example:
{code}
{
"queries": [
{
"objectName": "ProductConnect:type=Statistic,name=shopResponseStatistic",
"resultAlias": "shopResponseStatistic",
"attributes": ["ResendCounter", "ResponseCodeRange200"],
"aliasNames": ["request.failed", "request.success"]
},
{code}

Kind regards
Peter

hello,

i am trying to use the embedded-jmxtrans, unfortunately there no influxDb writer implementing org.jmxtrans.embedded.output.OutputWriter .

there is any solution to connect influx db writer ?

Add a plain Java web application (without Spring) embedded-jmxtrans launcher based on a ServletContextListener.

Hello,

I am looking out for ways to write code that could be executed when certain jvm metric crosses its configured threshold value. For e.g. If the cpu usage crosses a configured threshold value say 90% I want to take frequent thread dumps that could then be used for root cause analysis.

For this I was thinking of extending the configured output writer (for e.g. graphiteWriter) and executing custom code to parse the metric value and perform custom action (in this case take thread dumps).
But there is one problem - This metric needs to be monitored more frequently then others. i.e. every 2 secs we check if the cpu crosses 90% if so take thread dumps and send the metric to graphite. If not do not take any action.

Is there a way to configure embedded jmxtrans to send a jvm metric more frequently than others?

Thanks!

I would like to query for a bunch of metrics which all share the same Type and some common name, e.g. I would have:

{
  "queries": [
    {
      "objectName": "my.mbean:name=cluster.*,*",
      "resultAlias": "%name%",
      "attributes": [
        "Value"
      ]
    }]
}

The metrics collected are supposed to be sent to graphite, but due to the org.jmxtrans.embedded.ResultNameStrategy the won't appear like cluster.some.hierarchy, rather than cluster_some_hierarchy. This is kind of breaking the graphite namespace as previously hierarchical metrics will share the same top level, as all levels are joined into one. This happens only if I use placeholders, if I would directly write the exact same name with dots, they're not replaced.
Would it be possible to have the dot replacement optional?

I would like to incorporate jmxtrans in my application that would connect to several nodes in cloud environment and fetch JMX enabled metrics. Is that doable via json configuration? If not, how this can be achieved via API? Thanks in advance.

EmbeddedJmxTrans.start() spawns ScheduledExecutors directly, using the JDK Executors static factory.
AFAIK this is discouraged as it may cause issues in J2EE containers.

At least SpringEmbeddedJmxTrans should delegate the creation of a TaskScheduler to Spring, and Spring may be configured to delegate thread creation to the container, when needed.

Any idea what this might be? A class file incompatibility between jdk6 and jdk7 maybe?

scala: error while loading EmbeddedJmxTrans, class file '/home/bkaplan/.ivy2/cache/org.jmxtrans.embedded/embedded-jmxtrans/jars/embedded-jmxtrans-1.0.9.jar(org/jmxtrans/embedded/EmbeddedJmxTrans.class)' is broken
(class java.lang.RuntimeException/bad constant pool tag 9 at byte 10)

The #hostname# and #hostname_canonical# should be a dot path structure like 'com.company.host' and not the escaped path 'com_company_host'.
see https://github.com/jmxtrans/embedded-jmxtrans/wiki/Expression-Language

The correct path where escaped in ResultNameStrategy#resolveExpression() by calling ResultNameStrategy#appendEscapedNonAlphaNumericChars().

@Test
public void testCanonicalHostName() throws UnknownHostException {
     String expectedHostname = InetAddress.getLocalHost().getCanonicalHostName();
     String actualHostname = new ResultNameStrategy().resolveExpression("#canonical_hostname#");
     Assert.assertEquals(expectedHostname, actualHostname);
}

During shutdown we are seeing the following exception.. could be related to the commons-pool upgrade?

2015-07-08 16:38:57,224 WARN  [jmxtrans-export-1] (org.jmxtrans.embedded.output.GraphiteWriter) Failure to send result to graphite server 'HostAndPort{host='graphite.xxx', port=2003}' with null
java.lang.IllegalStateException: Pool not open
    at org.apache.commons.pool2.impl.BaseGenericObjectPool.assertOpen(BaseGenericObjectPool.java:662)
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:338)
    at org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:278)
    at org.jmxtrans.embedded.output.GraphiteWriter.write(GraphiteWriter.java:144)
    at org.jmxtrans.embedded.Query.exportCollectedMetrics(Query.java:208)
    at org.jmxtrans.embedded.EmbeddedJmxTrans$2.run(EmbeddedJmxTrans.java:229)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)

hi there -- thanks for EmbeddedJmxTrans! it's perfect for our uses ;)

One issue arose recently however. We provisioned a replacement graphite host in order to scale up, switched its DNS records, and kept an eye on the clients to ensure they switched over to the new host.

One set of clients that didn't were the ones using EmbeddedJmxTrans, instead reporting this stacktrace every minute:

2013-06-21 15:08:00,483 [jmxtrans-export-1] WARN  org.jmxtrans.embedded.output.GraphiteWriter  - Fa
ilure to send result to graphite server 'graphite.example.com/10.2.3.4:2003' with null
java.net.ConnectException: Connection refused
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at java.net.Socket.<init>(Socket.java:375)
        at java.net.Socket.<init>(Socket.java:218)
        at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFacto
ry.java:53)
        at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFacto
ry.java:39)
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
        at org.jmxtrans.embedded.output.GraphiteWriter.write(GraphiteWriter.java:129)
        at org.jmxtrans.embedded.Query.exportCollectedMetrics(Query.java:208)
        at org.jmxtrans.embedded.EmbeddedJmxTrans$2.run(EmbeddedJmxTrans.java:152)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
        at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:317)
        at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:150)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:98)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.runPeriodic(ScheduledThreadPoolExecutor.java:180)
        at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:204)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

sure enough, reading https://github.com/jmxtrans/embedded-jmxtrans/blob/master/src/main/java/org/jmxtrans/embedded/output/GraphiteWriter.java , it appears the InetSocketAddress is resolved once at startup, then cached forever.

    graphiteServerSocketAddress = new InetSocketAddress(host, port);

For correctness and resilience against IP address changes, the host (string) and port should be cached forever, but the graphiteServerSocketAddress should be recreated on every call to write(), in order to re-look-up the DNS records correctly. DNS caching should take care of the rest.

Could you please lower the logging level, when no explicit type is given?

On the one hand I would like to see the info logs for messages like

logger.info("Unsupported metric type '{}' for result {}, export it as counter", result.getType(), result);

But when everything is OK and it should default to counter this line is filling up my log and makes it unreadable:

logger.info("Unspecified type for result {}, export it as counter");

We've AtomicLong counters exposed with JMX but their value is not sent when using LibratoWriter.

Root cause at

        g.writeNumberField("measure_time", counter.getEpoch(TimeUnit.SECONDS));
        if (counter.getValue() instanceof Integer) {
            g.writeNumberField("value", (Integer) counter.getValue());
        } else if (counter.getValue() instanceof Long) {
            g.writeNumberField("value", (Long) counter.getValue());
        } else if (counter.getValue() instanceof Float) {
            g.writeNumberField("value", (Float) counter.getValue());
        } else if (counter.getValue() instanceof Double) {
            g.writeNumberField("value", (Double) counter.getValue());
        }
        g.writeEndObject();

AtomicLong type is not tested, the value is not sent.

We fixed this in our application code by wrapping our AtomicLong object with Long getter when registered in MBean Server.

I'm not sure how to patch this in the LibratoWriter:
adding an "if" for AtomicLong will work but not for LongAdder or any other future or not known Number subclasses.

So may be we only could throw an exception or log an error ?

When outputWriter attribute "settings.enabled" is defined, output writer is always disabled even if value is "true" ({settings:{"enabled":"xyz"}}).

Fixed by #27

It would be nice to have a simple WAR project (maybe even publishied to Maven Central) that can be published to any servlet container out of the box and automatically read configuration from the application server's working directory. I've noticed there is already an application that shows to add embedded-jmxtrans into existing application, but in case of many applications, it would be better to just have an indepedent app that can be deployed for container-wide metric collection.

§ https://github.com/jmxtrans/embedded-jmxtrans/blob/master/pom.xml#L247

I wanted to try out basic getting started example with embedded-jmxtrans. So I add the below code

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:jmxtrans="http://www.jmxtrans.org/schema/embedded"
       xmlns:context="http://www.springframework.org/schema/context"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-3.0.xsd
       http://www.jmxtrans.org/schema/embedded http://www.jmxtrans.org/schema/embedded/jmxtrans-1.0.xsd
       http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-3.0.xsd">

    <context:annotation-config/>

    <jmxtrans:jmxtrans>
        <jmxtrans:configuration>classpath:jvmmonitoring/jmxtrans/jmxtrans.json</jmxtrans:configuration>
        <jmxtrans:configuration>classpath:org/jmxtrans/embedded/config/tomcat-6.json</jmxtrans:configuration>
        <jmxtrans:configuration>classpath:org/jmxtrans/embedded/config/jmxtrans-internals.json</jmxtrans:configuration>
        <jmxtrans:configuration>classpath:org/jmxtrans/embedded/config/jvm-sun-hotspot.json</jmxtrans:configuration>
    </jmxtrans:jmxtrans>


</beans>

public class EmbeddedJMXTrans {
    public static void main(String[] args) {
        System.setProperty("spring.profiles.active", "dev");
        System.setProperty("spring.profiles.default", "dev");
        System.setProperty("spring.liveBeansView.mbeanDomain", "dev");
        ApplicationContext context = new ClassPathXmlApplicationContext("jvmmonitoring/jmxtrans/spring-beans.xml");
        while (true) {

        }
    }
}

The while loop was added to keep the application up till the jvm statistics are printed on console. Here is the jmxtrans.json file

{
    "queries": [
        {
            "objectName": "java.lang:type=Memory",
            "resultAlias": "jvm.memory",
            "attributes": [
                {
                    "name": "HeapMemoryUsage",
                    "keys": ["committed", "used"]
                },
                {
                    "name": "NonHeapMemoryUsage",
                    "keys": ["committed", "used"]
                }
            ]

        },
        {
            "objectName": "java.lang:type=Runtime",
            "resultAlias": "jvm.runtime",
            "attributes": [
                "Uptime"
            ]

        },
        {
            "objectName": "java.lang:type=GarbageCollector,name=*",
            "resultAlias": "jvm.gc.%name%",
            "attributes": [
                "CollectionCount",
                "CollectionTime"
            ]
        },
        {
            "objectName": "java.lang:type=Threading",
            "resultAlias": "jvm.thread",
            "attributes": [
                "ThreadCount"
            ]

        },
        {
            "objectName": "java.lang:type=OperatingSystem",
            "resultAlias": "jvm.os",
            "attributes": [
                "CommittedVirtualMemorySize",
                "FreePhysicalMemorySize",
                "FreeSwapSpaceSize",
                "OpenFileDescriptorCount",
                "ProcessCpuTime",
                "SystemLoadAverage"
            ]

        }
    ],
    "outputWriters": [
        {
          "@class": "org.jmxtrans.embedded.output.ConsoleWriter"
        }
    ]
}

On enabling debug level logs I found that jmxtrans spring bean wasn't created because of a spring circular reference error

DEBUG o.s.b.f.s.DefaultListableBeanFactory 1426 - Ignoring bean creation exception on FactoryBean type check: org.springframework.beans.factory.BeanCurrentlyInCreationException: Error creating bean with name 'jmxtrans': Requested bean is currently in creation: Is there an unresolvable circular reference?

The complete spring logs are shared here - http://pastebin.com/p2VeNEzE

What could be missing?

The use case is when you are fine tuning writing of queries, it would be really nice to be able to turn on "polling" of the configuration file, for the configuration to be picked up at runtime instead of just once at startup. This would be similar to the polling feature in logback.

I will probably add this at some point, I just wanted to get the issue out here so we can see if there is any support for it.

How can I configure embedded-jmxtrans in a programmatic way like it is shown for jmxtrans here ?

Librato Metrics writer relying on the HTTP API http://dev.librato.com/v1/post/metrics

It would be benefitial if JMXtrans reported non-valid configuration and invalid property names

Support Graphite Pickle Protocol according to http://graphite.readthedocs.org/en/latest/feeding-carbon.html#the-pickle-protocol

Is it correct assumption that jmxtrans and embedded-jxmtrans use different OutputWriter impementation that are not reusable? I don't see that embedded-jxmtrans depends on jmxtrans. Would it make sense to create a common core?

I am trying to use StatsDWriter and for some metrics I get the following warning:

2015/08/05-11:01:01.339 WARN [jmxtrans-export-1] org.jmxtrans.embedded.output.StatsDWriter - Given data too big (115bytes) for the buffer size (93bytes), skip it: stats.sys.jvm.loc...

The metric is the following:
stats.sys.jvm.localhost.mongodb.connection_pools.localhost.27017.55c1d0513e2a02c639c596e2.Statistics.MinSize:0|c

I tried to play with bufferSize parameter but the result is the same.

"outputWriters": [
{
"@Class": "org.jmxtrans.embedded.output.StatsDWriter",
"settings": {
"host": "${statsd.host}",
"port": "${statsd.port}",
"namePrefix": "stats.",
"enabled": "${statsd.enabled:true}",
"bufferSize" : 1024
}
}
]

I am using embedded-jmxtrans 1.1.0.

Write a Getting Started Guide

1. Maven
2. Spring configuration with
   • jvm + tomcat + internal-jmxtrans mbeans
   • outputwriter: Console + graphite
3. few graphite screenshots

Support Graphite Carbon Pickle Protocol as described in http://graphite.readthedocs.org/en/0.9.10/feeding-carbon.html#the-pickle-protocol

Remove noisy line break in ConsoleWriter and Slf4jWriter outputs

At the moment it is unclear what is happening inside jmxtrans and what is the exact configuration it takes in. It would be much better if jxmtrans reported it's effective configuration upon startup on DEBUG level, and probably showed even more information on TRACE level when actual queries are performed.

I'm struggling for couple of hours already to get simple ConsoleWriter to at least print something and I'm not sure if the configuration is wrong or something else. The only message I get from it is:

DEBUG o.j.e.s.EmbeddedJmxTransBeanDefinitionParser - Add configuration from attribute classpath:jmxtrans.json

Having the enabled attribute at the same level as @classwhen all other attributes (host, port, ...) are declared under the settingsnode is confusing.
Move the enabled attribute under the settings attribute.

Hi,

we want to use it in embedded Jetty, do you have a working example for that?

Thx,
Sascha

commons-pool does not support TTL on cached object.

Find a way to add support for TTL tweaking commons-pool or using another pool.

Hello,
embedded-jmxtrans sometimes blocks in stop when we have network problems, or the graphite server is in a zombie state.
The call of new Socket(inetSocketAddress.getAddress(), inetSocketAddress.getPort()); opens the socket with a timeout of 0 = infinite.

It would be better to open the socket with:
Socket socket = new Socket();
socket.connect(new InetSocketAddress(this.host, this.port), this.timeout);
and define a timeout value by a configuration or system property.

The following stacktrace shows the blocking thread
{code}
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
- locked <0x000000079774bc88> (a java.net.SocksSocketImpl)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
at java.net.Socket.connect(Socket.java:579)
at java.net.Socket.connect(Socket.java:528)
at java.net.Socket.(Socket.java:425)
at java.net.Socket.(Socket.java:241)
at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFactory.java:53)
at org.jmxtrans.embedded.util.pool.SocketWriterPoolFactory.makeObject(SocketWriterPoolFactory.java:39)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1220)
at org.jmxtrans.embedded.output.GraphiteWriter.write(GraphiteWriter.java:129)
at org.jmxtrans.embedded.Query.exportCollectedMetrics(Query.java:208)
at org.jmxtrans.embedded.EmbeddedJmxTrans.exportCollectedMetrics(EmbeddedJmxTrans.java:220)
at org.jmxtrans.embedded.EmbeddedJmxTrans.stop(EmbeddedJmxTrans.java:186)
{code}

Kind regards
Peter

JMX does not provide a metric to determine jvm's cpu usage. The ManagementFactory.getOperatingSystemMXBean().getSystemLoadAverage() jmx attribute gives the overall system cpu load. I came across this link and was looking for a way to integrate it with embedded-jmxtrans.

Is there a way to send custom jvm metrics like jvm's cpu usage along with other configured ones with jmxtrans? Any interfaces/class I classes to implement/extend?

Thanks

Make embedded-jmxtrans threads daemon

Copy of jmxtrans/jmxtrans#216

@mheffner said

I'm from the Librato team. It would help us to include a descriptive User-Agent header in the LibratoWriter that includes "jmxtrans" and the version number of jmxtrans, eg. "jmxtrans/1.2.3". This would assist in our debugging when working with users that are sending data with jmxtrans to Librato.
We have a full write-up here: http://support.metrics.librato.com/knowledgebase/articles/175825-how-to-set-a-custom-user-agent-when-integrating-wi