jmxtrans / jmxtrans Goto Github PK

Hi

in KeyOutWriter, logger is static (and its name is unique/hard-coded as "KeyOutWriter")
but each instance (i.e. each query in the configuration files) :

• can define a different filename
• add a new appender to the same logger each time is initialized

the result is that i get duplicate line for each metrics that get sent to the writer

in my case, 1 server def with 12 queries (each of them using the same file for KeyOutWriter) :
each line is then duplicated 12 times!

i can see/suggest 2 ways to fix that :
quick fix : do not support multiple filename (only create the appender on the first instance!)
long fix : non-static logger field, static cache of logger by output filename not to duplicate loggers

Sometime, JMXTrans can't start because different jobs have the same generated ID.

In fact, the used algorithm is not good :

String name = server.getHost() + ":" + server.getPort() + "-" + System.currentTimeMillis();

[06 Mar 2012 11:28:09] [main] 1      ERROR (com.googlecode.jmxtrans.JmxTransformer:106) - Error scheduling job for server: Server [host=xxxxxxxx.org, port=32001, url=null, cronExpression=null, numQueryThreads=4]
org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'xxxxxxxx.org:32001-1331029689802' and group: 'DEFAULT', because one already exists with this identification.
    at org.quartz.simpl.RAMJobStore.storeTrigger(RAMJobStore.java:314)
    at org.quartz.simpl.RAMJobStore.storeJobAndTrigger(RAMJobStore.java:194)
    at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:801)
    at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243)
    at com.googlecode.jmxtrans.JmxTransformer.scheduleJob(JmxTransformer.java:295)
    at com.googlecode.jmxtrans.JmxTransformer.processServersIntoJobs(JmxTransformer.java:256)
    at com.googlecode.jmxtrans.JmxTransformer.startupSystem(JmxTransformer.java:173)
    at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:102)
    at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:84)

I'm trying to use GraphiteWriter OOB after cloning locally, running "ant dist" and running jmxtrans.sh from the target/jmxtrans-2013xxxxx directory. I'm getting this:

[22 May 2013 22:19:30] [pool-1-thread-2] 350 ERROR (com.googlecode.jmxtrans.util.JmxUtils$ProcessQueryThread:163) - Error executing query
java.util.NoSuchElementException: Could not create a validated object, cause: ValidateObject failed
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1256)
at com.googlecode.jmxtrans.model.output.GraphiteWriter.doWrite(GraphiteWriter.java:147)
at com.googlecode.jmxtrans.util.JmxUtils.runOutputWritersForQuery(JmxUtils.java:336)
at com.googlecode.jmxtrans.util.JmxUtils.processQuery(JmxUtils.java:206)
at com.googlecode.jmxtrans.util.JmxUtils$ProcessQueryThread.run(JmxUtils.java:161)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:680)

Not sure why - using the last precomplied build from site works fine, and keyout writer is fine too...

c3p0 datasource beans does not contain name in the ObjectName. It is something like: com.mchange.v2.c3p0:type=PooledDataSource*, where * is replaced by some random string and changes per application restart.
I am using 4 different datasources in one application and want to keep track them by their datasourceName. How do I achieve this?

Hi,

• I use JMXTrans to send numeric metric values to Graphite.
• Graphite supports receiving Integer values only.
• Some JMX attributes contain decimal values like 0.48, or String values.

Does JMXTrans provide a mechanism by which I'm able to transform JMX attributes to values that would make sense to Graphite? For example, I may like to multiply decimal values by 100 to scale 0.48 to 48.

I've read the wiki docs, but haven't found such a feature.

When servers that are monitored with jmxtrans go down due to maintenance window or any other reason it seems that jmxtrans starts to leak threads ending finally in an OutOfMemoryException.

Attached is an Graphite graph for threads used by jmxtrans itself, showing the leak. As you can see as roughly at 1:10 am an we started a maintenance window with some of our services being shut down.

As a temporary fix I am restarting jmxtrans every hour :-(.

As a side note to find out which JMX connection cannot be established you need to run jmxtrans in DEBUG mode because the line of the java.net.ConnectException does not give any details.

The jmxtrans version we use:
[root@myserver ~]# rpm -qa|grep jmxtrans
jmxtrans-20130909.170310.bd95c1f283-0.noarch

When I start jmxtrans on my Red Hat box, I have:

:/apps/ajmxtrans/bin# java.lang.UnsatisfiedLinkError: /tmp/jpathwatch-nativelib-v-0-94-libjpathwatch-native.so: /tmp/jpathwatch-nativelib-v-0-94-libjpathwatch-native.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at name.pachler.nio.file.impl.NativeLibLoader.extractAndLoadLibrary(NativeLibLoader.java:397)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:247)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:105)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
java.lang.UnsatisfiedLinkError: /tmp/jpathwatch-nativelib-v-0-94-5682615090656347523libjpathwatch-native.so: /tmp/jpathwatch-nativelib-v-0-94-5682615090656347523libjpathwatch-native.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at name.pachler.nio.file.impl.NativeLibLoader.extractAndLoadLibrary(NativeLibLoader.java:397)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:257)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:105)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
java.lang.UnsatisfiedLinkError: no jpathwatch-native in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at name.pachler.nio.file.impl.NativeLibLoader.loadDefaultLibrary(NativeLibLoader.java:84)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:267)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:105)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
java.lang.UnsatisfiedLinkError: /tmp/jpathwatch-nativelib-v-0-94-libjpathwatch-native.so: /tmp/jpathwatch-nativelib-v-0-94-libjpathwatch-native.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at name.pachler.nio.file.impl.NativeLibLoader.extractAndLoadLibrary(NativeLibLoader.java:397)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:247)
at name.pachler.nio.file.impl.Unix.(Unix.java:86)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:211)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
java.lang.UnsatisfiedLinkError: /tmp/jpathwatch-nativelib-v-0-94-2818599835536657461libjpathwatch-native.so: /tmp/jpathwatch-nativelib-v-0-94-2818599835536657461libjpathwatch-native.so: failed to map segment from shared object: Operation not permitted
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1803)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1699)
at java.lang.Runtime.load0(Runtime.java:770)
at java.lang.System.load(System.java:1003)
at name.pachler.nio.file.impl.NativeLibLoader.extractAndLoadLibrary(NativeLibLoader.java:397)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:257)
at name.pachler.nio.file.impl.Unix.(Unix.java:86)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:211)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
java.lang.UnsatisfiedLinkError: no jpathwatch-native in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1734)
at java.lang.Runtime.loadLibrary0(Runtime.java:823)
at java.lang.System.loadLibrary(System.java:1028)
at name.pachler.nio.file.impl.NativeLibLoader.loadDefaultLibrary(NativeLibLoader.java:84)
at name.pachler.nio.file.impl.NativeLibLoader.loadLibrary(NativeLibLoader.java:267)
at name.pachler.nio.file.impl.Unix.(Unix.java:86)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:211)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)
Aug 8, 2012 3:23:08 PM name.pachler.nio.file.ext.Bootstrapper newWatchService
WARNING: null
java.lang.UnsatisfiedLinkError: name.pachler.nio.file.impl.Unix.getIntDefine(Ljava/lang/String;)I
at name.pachler.nio.file.impl.Unix.getIntDefine(Native Method)
at name.pachler.nio.file.impl.Unix.(Unix.java:87)
at name.pachler.nio.file.impl.LinuxPathWatchService.(LinuxPathWatchService.java:211)
at name.pachler.nio.file.ext.Bootstrapper.newWatchService(Bootstrapper.java:80)
at name.pachler.nio.file.FileSystems$1.newWatchService(FileSystems.java:40)
at com.googlecode.jmxtrans.util.WatchDir.(WatchDir.java:33)
at com.googlecode.jmxtrans.JmxTransformer.startupWatchdir(JmxTransformer.java:259)
at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:138)
at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:107)
at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:92)

It works fine on some other boxes. I presume that the problem is due to the fact that Jpathwatch install native libs in /tmp/.
Some security stuff doesn't allow to load *.so files in /tmp/, such as SELinux.
The workaround could be to set this native librarty in another directory.

I am running on IBM's Java and it seems that JPS is not part of their JDK.
I tried looking through the jmxtrans code to understand the dependency it has on JPS and I cannot figure that out either (although I see it in the startup checks but when I run, I can run without it

What is it needed for?

thanks,

I've cloned a repository and run ./jmxtrans.sh eample.json. I get "Error: Unable to access jarfile jmxtrans-all.jar". Is it missing in github repository?

Hi,

I'm trying to run master locally but when I do ./jmxtrans.sh start I get this error:

vagrant@precise32:/code/jmxtrans$ ./jmxtrans.sh start
vagrant@precise32:/code/jmxtrans$ Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/commons/cli/CommandLineParser
Caused by: java.lang.ClassNotFoundException: org.apache.commons.cli.CommandLineParser
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
Could not find the main class: com.googlecode.jmxtrans.JmxTransformer. Program will exit.

Before running this, I ran ant dist.

The ant build uses the git revision number to create a version. This is not compatible with Debian which requires versions to start with a number.

Have 205 json config files with identical layouts based using common queries such as heap, memory and threads. Noticing jmxtrans is using 263 threads to get the job done (ps -eLo, ps -p -T) and send the metrics to graphite. Using one host per json file. Is there a more efficient way to query or setup the config for more efficiency?

To support easy release building, Debian and RPM packages should be built using a CI tool, not locally. I added travis-ci support for my fork experimentally but it does not support building .deb/.rpm files yet.

Is there another way than building releases locally? We could wait for travis-ci support, though because it integrates beautifully with github.

python ./yaml2jmxtrans.py

File "./yaml2jmxtrans.py", line 100
if present(host, sep, port) = host_name.partition(":")
^
SyntaxError: invalid syntax


I think you just wanted to say: host, sep, port = host_name.partition(":") ?

Would it be possible to create a new release for jmxtrans?

Background: We are running into #60 using the latest release, but using latest source it run ok. Would be nice if we could use a stable release instead of a snapshot.

Hi!

For instance, I have com.organization.myproject with many registered beans.
Some of them have attributes, some haven't.

Is it possible to write queries to catch all metrics for:

• com.organization.myproject
• java.lang
• com.mongodb
• etc

The general idea for use it with graphite.
I think the killer feature of such monitoring is zero-configuration.

I don't care about what exactly metric I want to grub, services with graphite use push mechanism instead of pull. And this is great. Any redeploy/change immediately appears in graphite.

But, in case with jmx, jmxtrans - is a puller. That's why it will be great to catch all or part of tree. No reconfiguration needed, no restarts of jmxtrans needed, etc...

PS. I'm very new with JMX/JMXTrans. In documentation, examples uses mask (*) in class/type name, but attributes are defined explicitly.

Could you add a writer for Librato (https://metrics.librato.com/) writer ?
There is one in embedded-jmxtrans (https://github.com/jmxtrans/embedded-jmxtrans/wiki/Librato-Writer) but it's not compatible.

In regards to the following bug http://code.google.com/p/jmxtrans/issues/detail?id=9#makechanges did it get fixed? I cant see any reference to it in the code?

If not I'll see if i can sort it and issue a pull request.

Hi!
Often, we want to monitor all of the attributes of an object. We're able to do that using the JSON configuration by simply omitting the "attr" attribute:

{
"outputWriters" : [ {
"@Class" : "com.googlecode.jmxtrans.model.output.GraphiteWriter",
"settings" : {
"port" : 2003,
"host" : "localhost"
}
} ],
"obj" : "java.lang:type=GarbageCollector,name=*"
},

When "attr" is not specified in the YAML, yaml2jmxtrans.py set "attr" to null:

{
 "outputWriters": [
  {
   "@class": "com.googlecode.jmxtrans.model.output.GraphiteWriter", 
   "settings": {
    "typeNames": [
     "name"
    ], 
    "host": "localhost", 
    "port": 2003
   }
  }
 ], 
 "resultAlias": null, 
 "obj": "java.lang:type=GarbageCollector,name=*", 
 "attr": null
},

Can the ability NOT to specify an attribute be added to the yaml2jmxtrans.py ?
That is,
After processing the query
IF there is no "attr" defined
THEN don't include it in the JSON output

I have encountered an error when using distinct URL-defined servers in several json files. The error I receive is similar to what is below:

[03 Oct 2013 13:40:26] [main] 399    DEBUG (com.googlecode.jmxtrans.JmxTransformer:437) - Scheduled job: null:null-1380832826521-7373346390 for server: Server [host=null, port=null, url=service:jmx:remoting-jmx://www.example.com:4697, cronExpression=null, numQueryThreads=4]
[03 Oct 2013 13:40:26] [main] 399    DEBUG (com.googlecode.jmxtrans.JmxTransformer:437) - Scheduled job: null:null-1380832826532-6141082628 for server: Server [host=null, port=null, url=service:jmx:remoting-jmx://www.example.com:4497, cronExpression=null, numQueryThreads=4]
[03 Oct 2013 13:40:26] [main] 400    ERROR (com.googlecode.jmxtrans.JmxTransformer:148) - Error scheduling job for server: Server [host=null, port=null, url=service:jmx:remoting-jmx://www.example.com:4997, cronExpression=null, numQueryThreads=4]
com.googlecode.jmxtrans.util.LifecycleException: Error scheduling job for server: Server [host=null, port=null, url=service:jmx:remoting-jmx://www.example.com:4997, cronExpression=null, numQueryThreads=4]
    at com.googlecode.jmxtrans.JmxTransformer.processServersIntoJobs(JmxTransformer.java:400)
    at com.googlecode.jmxtrans.JmxTransformer.startupSystem(JmxTransformer.java:307)
    at com.googlecode.jmxtrans.JmxTransformer.start(JmxTransformer.java:145)
    at com.googlecode.jmxtrans.JmxTransformer.doMain(JmxTransformer.java:110)
    at com.googlecode.jmxtrans.JmxTransformer.main(JmxTransformer.java:95)
Caused by: org.quartz.ObjectAlreadyExistsException: Unable to store Trigger with name: 'null:null-1380832826532' and group: 'DEFAULT', because one already exists with this identification.
    at org.quartz.simpl.RAMJobStore.storeTrigger(RAMJobStore.java:314)
    at org.quartz.simpl.RAMJobStore.storeJobAndTrigger(RAMJobStore.java:194)
    at org.quartz.core.QuartzScheduler.scheduleJob(QuartzScheduler.java:801)
    at org.quartz.impl.StdScheduler.scheduleJob(StdScheduler.java:243)
    at com.googlecode.jmxtrans.JmxTransformer.scheduleJob(JmxTransformer.java:435)
    at com.googlecode.jmxtrans.JmxTransformer.processServersIntoJobs(JmxTransformer.java:396)
    ... 4 more

As you can see, this is because the CronTrigger is using the server's host and port to create the name. The code to fix this is theoretically quite simple, but I'm getting some other errors and need to get to the bottom of those before I can fully verify the update.

The clearest path to resolve is to replace the following code in JmxTransformer:

((CronTrigger) trigger).setName(server.getHost() "-" + server.getPort() + "-" + Long.valueOf(System.currentTimeMillis()).toString());

with

((CronTrigger) trigger).setName(server.getUrl() + "-" + Long.valueOf(System.currentTimeMillis()).toString());

Do you see any objections to this? I'll submit a pull request when I have the other issues resolved.

OpenTSDBWriter is not resilient to lost socket connections. Not sure exactly what is causing it, but we lose connection to the server and then no more data is sent to OpenTSDB.

I'll fix it; I'm in the middle of a lot of updates to support a new TCollectorUdpWriter (almost done), so this will naturally fit into that update.

I'm looking to add a few JARs to the classpath to handle other URL formats than default - and it seems like the "url" option seems to work. I'm trying to connect to Terracotta (adding the tc.jar) and jboss eap 6.0 (which needs to add a few files). I can connect using jConsole, after adding the missing jars to the classpath...

Apologies if this isn't the right place for this - please advise otherwise.

Thx - pd

I was wondering how can I add several monitoring groups to a config file?

here is my current json file

I want to specify multiple group names for monitoring in ganglia on a server behined a nat box

{
"servers" : [ {
"host" : "10.10.31.23",
"alias" : "10.10.31.23",
"port" : "1099",
"queries" : [ {
"obj" : "java.lang:type=GarbageCollector,name=ConcurrentMarkSweep",
"outputWriters" : [ {
"@Class" : "com.googlecode.jmxtrans.model.output.GangliaWriter",
"settings" : {
"groupName" : "server web group",
"host" : "10.10.31.23",
"port" : 8649,
"slope" : "both",
"units" : "bytes",
"tmax" : 60,
"dmax" : 900,
"sendMetadata": 5
}
} ]
} ]
} ]
}

Hi!

Can the yaml2jmxtrans.py be patched to allow overriding the OutputWriters configuration?

This includes migrating build.xml to pom.xml and migrating debian/rules to mvn clean package command.

When i start the jmxtrans and for around next 15 minutes the gmond is receiving the metrics properly but after that gmond is not receiving the metrics.

I started the code in debug mode and i can see log saying it is emitting, but gmond is not receiving.

..snip..
[11 Jul 2012 12:53:36] [ServerScheduler_Worker-4] 1380500 DEBUG (com.googlecode.jmxtrans.model.output.GangliaWriter:268) - Emitted metric HintedHandoff.PendingTasks, type DOUBLE, value 13 for host: 127.0.0.1:cassandra1007.xxxxx.com
...snip...

Could you please help me solve this issue ? (between, i am not a Java developer, else i would have tried for a patch :-)

Hi,

The best way possible for this tool, for me, is to create a query that defined something like the top "directories" that JConsole shows you when you connect to JVM JMX, and it will dump it.
For instance, If I have hbase running, I would like to sample "hbase., jvm., hadoop.*.
This makes it super easy to define a query.

Forcing me to write all the attribute names is very time consuming

jmxtrans currently uses log4j.DailyRollingFileAppender which creates a new log file every 24 hours, but isn't capable of compressing or deleting the old ones.

This build-up of old files results in some nasty hacks like: https://github.com/bryanwb/chef-jmxtrans/blob/master/recipes/default.rb#L86-91

logback supports both of these features and has in many ways superseded log4j. Would you consider switching this project over?

Here is a implementation of this feature as a diff against GraphiteWriter:

https://gist.github.com/ms2scale/7040467

OpenTSDBWriter supports creating a single tag from ObjectName using the typeNames setting. It would be very helpful to be able to create multiple tags.

For instance, the following is from an ActiveMQ subscription:

org.apache.activemq:BrokerName=broker1,Type=Subscription,persistentMode=Non-Durable,destinationType=Queue,destinationName=replay.requests,clientId=ID_client1-34827-1392975881222-1_0,consumerId=ID_client1-34827-1392975881222-0_0_0_0

Using typeNames of ["destinationName", "consumerId"] results in one tag named "destinationNameconsumerId" with a value of "replay.requests_ID_client1-34827-1392975881222-0_0_0_0".

I want to have destinationName and consumerId as separate tags in the metric.

The tags setting may look promising, but it only supports hard-coded tag values.

From a technical perspective, it seems like splitting the tags created by typeNames makes the most sense. However, that would break backward-compatibility.

How about a new setting mergeTypeNamesTags which defaults to true? Setting to false will cause the new, desired operation - creation of multiple tags instead of individual ones. I don't see a reason to want both merged and split.

Hello:

We are using jmxtrans with Graphite, but we are also trying to front it with Statsd.

We can see jmxtrans sending data on tcp, but not udp, which is what statsd is looking for.

How do we force jmxtrans to use udp in an outputwriter?

We are using jmxtrans with Kafka and running into some concerns with the metric naming produced by OpenTSDBWriter from the JMX query result names. For example, using the class name and attribute name to form the metric name does not work well with Kafka because many of the MBeans use the same, generic classname of the Gauge class. Note the same naming pattern works very well with ActiveMQ though.

I'm planning to solve this by supporting an expression to be defined as a setting to define the naming rule, if needed. OGNL is a language recommended to me for this purpose. Any thoughts or concerns with using OGNL as an expression langauge?

We recently plugged JMXTrans into our appservers. Immediately following this we saw strange GC behavior. Our app servers only allow Full GC's via RMI every 70 minutes. Every 70 minutes with JMXTrans polling we had full GC's. This is horrible as they are stop the world GC's. Can you confirm or deny that JMXTrans would cause a Full GC?

Using the startup scripts provided by the RPM [1], remote startup via SSH hangs indefinitely on CentOS 6.4. Note that jmxtrans does start, but that the startup script doesn't return.

For example:

# ssh remotehost service jmxtrans start
Starting jmxtrans: [  OK  ]
(at this point it hangs, eventually we tire and issue Ctrl-C)
^CKilled by signal 2.
(jmxtrans is running)

Startup scripts work locally.

Calling out the script directly -- ala /etc/init.d/jmxtrans -- results in the same.

[1] jmxtrans-20121016.145842.6a28c97fbb-0.noarch.rpm

Hi,

Is there any plan to use Maven for builds instead of ant?

Hi there,

I'm working with Raphaël Marmier and I've added some modifications to the dotted key parameter. As the keys are defined in the json for the queries, I've put the "allowDottedKeys" parameter in there instead of in the application options.
It simplifies the code a lot and avoid to have a static switch in the JmxUtils class.

Can I then have a commit authorisation on the AllowDotsInKeys branch?

Thank you very much.

jmxtrans.sh script is written assuming jps and java are located under /usr/bin/

There should be a way to define JAVA_HOME where these files could be read from if they're located in a location othre than /usr/bin

We got lots of processes running that we want to monitor using jmxtrans. Some of them are often restarted due deployments, others can be stopped because different reasons.

Jmxtrans now logs a full stacktrace when a connection error occurs. It would be great to log just a INFO or WARN log entry each time it fails connecting:

[20 Sep 2012 11:25:02] [ServerScheduler_Worker-5] 28483  ERROR (com.googlecode.jmxtrans.jobs.ServerJob:39) - Error
java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 33.33.33.10; nested exception is: 
        java.net.ConnectException: Connection timed out]
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:357)
        at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:267)
        at com.googlecode.jmxtrans.util.JmxUtils.getServerConnection(JmxUtils.java:413)
        at com.googlecode.jmxtrans.util.JmxConnectionFactory.makeObject(JmxConnectionFactory.java:30)
        at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
        at com.googlecode.jmxtrans.jobs.ServerJob.execute(ServerJob.java:36)
        at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
        at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: javax.naming.ServiceUnavailableException [Root exception is java.rmi.ConnectException: Connection refused to host: 33.33.33.10; nested exception is: 
        java.net.ConnectException: Connection timed out]
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:118)
        at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:203)
        at javax.naming.InitialContext.lookup(InitialContext.java:411)
        at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1924)
        at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1891)
        at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:274)
        ... 7 more
Caused by: java.rmi.ConnectException: Connection refused to host: 33.33.33.10; nested exception is: 
        java.net.ConnectException: Connection timed out
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:619)
        at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:216)
        at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
        at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:340)
        at sun.rmi.registry.RegistryImpl_Stub.lookup(Unknown Source)
        at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:114)
        ... 12 more
Caused by: java.net.ConnectException: Connection timed out
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:337)
        at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:198)
        at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:391)
        at java.net.Socket.connect(Socket.java:579)
        at java.net.Socket.connect(Socket.java:528)
        at java.net.Socket.<init>(Socket.java:425)
        at java.net.Socket.<init>(Socket.java:208)
        at sun.rmi.transport.proxy.RMIDirectSocketFactory.createSocket(RMIDirectSocketFactory.java:40)
        at sun.rmi.transport.proxy.RMIMasterSocketFactory.createSocket(RMIMasterSocketFactory.java:146)
        at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:613)
        ... 17 more

There's been a request to package this in Homebrew: Homebrew/legacy-homebrew#18076

Is there a newer stable release than the 1 year old tag, or is that still recommended?

When I try to use a property value such as "${some.prop}/static.file" I get an IndexOutOfBoundsException in PropertyResolver.resolveProps. THis should instead just resolve the variable part and substitute the rest as it is.

My platform:
jmxtrans running on Linux x86, talking to some WL Servers, when i discover the issue, i have 5 threads per query, this causes in the log this kind of errors:

...
[30 ene 2012 14:11:30] [WorkerThread-LastJob] 2665 DEBUG (com.googlecode.jmxtrans.util.JmxUtils:102) - ----- Creating 4 query threads
[30 ene 2012 14:11:30] [pool-1-thread-4] 2698 ERROR (com.googlecode.jmxtrans.util.JmxUtils$ProcessQueryThread:163) - Error executing query
java.lang.IllegalStateException: Cannot register two instances of WorkContextAccessController
at weblogic.workarea.spi.WorkContextAccessController.(WorkContextAccessController.java:24)
at weblogic.workarea.spi.WorkContextAccessController$1.(WorkContextAccessController.java:65)
at weblogic.workarea.spi.WorkContextAccessController.getAccessController(WorkContextAccessController.java:65)
at weblogic.workarea.spi.WorkContextAccessController.isAccessAllowed(WorkContextAccessController.java:38)
at weblogic.workarea.WorkContextLocalMap.put(WorkContextLocalMap.java:50)
at weblogic.workarea.WorkContextLocalMap.put(WorkContextLocalMap.java:62)
at weblogic.workarea.WorkContextMapImpl.put(WorkContextMapImpl.java:52)
at weblogic.management.context.JMXContextHelper$DefaultJMXContextAccessImpl.putJMXContext(JMXContextHelper.java:89)
at weblogic.management.context.JMXContextHelper.putJMXContext(JMXContextHelper.java:36)
at weblogic.management.remote.common.RMIConnectionWrapper.initializeJMXContext(ClientProviderBase.java:1020)
at weblogic.management.remote.common.RMIConnectionWrapper.getMBeanInfo(ClientProviderBase.java:1030)
at javax.management.remote.rmi.RMIConnector$RemoteMBeanServerConnection.getMBeanInfo(RMIConnector.java:1052)
at com.googlecode.jmxtrans.util.JmxUtils.processQuery(JmxUtils.java:181)
at com.googlecode.jmxtrans.util.JmxUtils$ProcessQueryThread.run(JmxUtils.java:161)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
[30 ene 2012 14:11:30] [pool-1-thread-1] 2756 DEBUG (com.googlecode.jmxtrans.util.JmxUtils:195) - Executing queryName: com.bea:Location=AdminServer,Name=AdminServer,ServerRuntime=AdminServer,Type=JVMRuntime from query: Query
..

This seems that jmxconnection is not thread safe, i solve with only one thread per query, this causes me, a lot of headache becaouse some of the metrics are lost and not reach the writters.

This can happen with the WL RMI/JMX connector, i test it with different protocols (rmi, iiop, t3), all the same.

Define a query (in the json) which the "obj" contains an invalid object. For example:
"java.lang:type=GarbageCollector2,name=*"
(I placed 2 to represent a spelling error).

After you restart jmxtrans with this json, there's no error in the log indicating this object in non existent.
When you fix the spelling error and restart jmxtrans the log indicates this object was retrieved.

I just ran into this issue - shutting down JMXtrans hangs. I think it's a Quartz bug.

Here's the shutdown thread that is blocked:

"Thread-2" - Thread t@14
   java.lang.Thread.State: TIMED_WAITING
        at java.lang.Object.wait(Native Method)
        - waiting on <66cf0eb> (a java.lang.Object)
        at org.quartz.simpl.SimpleThreadPool.shutdown(SimpleThreadPool.java:361)
        at org.quartz.core.QuartzScheduler.shutdown(QuartzScheduler.java:640)
        at org.quartz.impl.StdScheduler.shutdown(StdScheduler.java:207)
        at com.googlecode.jmxtrans.JmxTransformer.stopServices(JmxTransformer.java:195)
        - locked <5f23510d> (a com.googlecode.jmxtrans.JmxTransformer)
        at com.googlecode.jmxtrans.JmxTransformer.access$000(JmxTransformer.java:56)
        at com.googlecode.jmxtrans.JmxTransformer$ShutdownHook.run(JmxTransformer.java:679)

Here's a jira entry for Quartz that looks like this problem:
https://jira.atlassian.com/browse/JRA-2517

Perhaps upgrading Quartz is the best solution. That also raises a desire to move from ant-based builds to maven-based builds. Thoughts?

Another thing - the jmxtrans.sh script could be improved to timeout after a reasonable period (I recommend 15 seconds) and then use SIGKILL to forcefully terminate the process.

By the way - I don't know how hard it is to reproduce this problem. I'll try.

Hey there, as given on Installation page, I am trying to find any jmxtrans_*_amd64.deb file. However I got one which does not have amd64 in its name and it's not working on my server running Ubuntu 12.04.
Please provide a proper way of installation.

It's difficult to write Writer implementations without a git tag representing the version which was released.

I had installed jmxtrans locally. I am trying to connect to the same.

The url published by jmxtrans is as below:

url=service:jmx:rmi:///jndi/rmi://10.148.1.3:1105/jmxrmi

On the server side (where jmx is running):

$ ps -ef | grep solr
root 15200 1 0 Feb07 ? 00:02:02 /usr/share/jdk7/bin/java -Djava.util.logging.config.file=/myarg/search-service/releases/current//conf/logging.properties -server -Xmx4096m -Xms4096m -Dsolr.solr.home=/myarg/search-service/releases/current/solrhome/ -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=1105 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Djava.rmi.server.hostname=10.148.1.3 -Djava.endorsed.dirs=/opt/software/tomcat7/endorsed -classpath /opt/software/tomcat7/bin/bootstrap.jar:/myarg/search-service/releases/current//bin/tomcat-juli.jar -Dcatalina.base=/myarg/search-service/releases/current/ -Dcatalina.home=/opt/software/tomcat7 -Djava.io.tmpdir=/myarg/search-service/releases/current//temp org.apache.catalina.startup.Bootstrap start
504 23407 23384 0 00:05 pts/0 00:00:00 grep solr

telnet to the remote port works fine. ( telnet 10.148.1.3 1105 ) .

It fails with the following error:

[ServerScheduler_Worker-8] 367810285 ERROR (com.googlecode.jmxtrans.jobs.ServerJob:39) - Error
java.rmi.ConnectException: Connection refused to host: 10.148.1.3; nested exception is:
java.net.ConnectException: Conne
ction timed out
at sun.rmi.transport.tcp.TCPEndpoint.newSocket(TCPEndpoint.java:601)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:198)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:184)
at sun.rmi.server.UnicastRef.invoke(UnicastRef.java:110)
at javax.management.remote.rmi.RMIServerImpl_Stub.newClient(Unknown Source)
at javax.management.remote.rmi.RMIConnector.getConnection(RMIConnector.java:2327)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:277)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:248)
at com.googlecode.jmxtrans.util.JmxUtils.getServerConnection(JmxUtils.java:351)
at com.googlecode.jmxtrans.util.JmxConnectionFactory.makeObject(JmxConnectionFactory.java:31)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
at com.googlecode.jmxtrans.jobs.ServerJob.execute(ServerJob.java:36)
at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
Caused by: java.net.ConnectException: Connection timed out
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
at java.net.Socket.connect(Socket.java:529)

Not sure what the issue is, since telnet seems to connect fine. ( Opened up firewall for TCP / 1105 for communication as well). Anything else that we are missing here. Thanks !

Hello,

I am just starting to use jmxtrans and maybe this is a noob question...
Is it possible to ask jmxtrans automatically put the hostname if it's on localhost? for example in below config, it will be prefect I don't have to give a hard coded host name

 {
  "servers": [

 {
 "numQueryThreads": 2,
 "host": "esb-dev1",
 "username": "",
 "password": "",
  "port": 21001,

  ......

Many thanks

James

I want to use ansible to install the deb package. Is it possible to bypass the memory prompt ?

Hi,

I have a few values I want to get more frequently, and a large number that I want to get less frequently. I can't get them all at the same rate as the graphite server can't handle the IO. Currently, I'm considering running a second instance to allow this, but it would be nice if this were something I could configure on a per query basis. I've forked the code and will try to work on a patch but I'm not terribly experienced with Java so if you can do it more quickly, that would be very much appreciated. Thanks for jmxtrans, BTW, it's been really useful and has worked really well.

Steve

I think that in jmxtrans.sh -Djmxtrans.log.dir=$LOG_DIR shoud be part of JMXTRANS_OPTS instead of JAVA_OPTS otherwise when JAVA_OPTS is specified jmxtrans.log.dir is not set.