Heavy Memory usage up to 11GB about signalfx-collectd-plugin HOT 7 CLOSED

Hi @diegopacheco,

Could you share the list of plugins that is configured in your collectd? If you think it's caused by the signalfx collectd plugin (this repo), could you isolate to just that plugin (and the write_http plugin) to confirm it is, in fact, the source of the memory leak?

That being said, a quick glance at the code and I can already tell that https://github.com/signalfx/signalfx-collectd-plugin/blob/master/src/signalfx_metadata.py#L285 is probably wrong and should just be if self.metrics[t].

from signalfx-collectd-plugin.

Hi @diegopacheco,

I noticed connectivity issues in the log snippet that you shared. Could you please open a support ticket through support.signalfx.com and share your configuration files and logs for this host? We would like to take a closer look at the issue. Please provide a link to this github issue in the support ticket.

Thank you!

from signalfx-collectd-plugin.

@mpetazzoni

This is a long time issue. Since for 0.29 we are experience Collected using lots of memory. We thought this was fixed at 0.30. Here are all plugins I'm using for this host:

10-aggregation-cpu.conf  
10-jmx.conf  
10-signalfx.conf  
10-write_http-plugin.conf  
11-redis.conf  
12-rocksdb.conf  
20-javageneric.conf

But looks like you already find the culprit - when can you guys release a fix?

from signalfx-collectd-plugin.

@charliesignalfx

Done. Here is the issue: https://support.signalfx.com/hc/en-us/requests/3576

from signalfx-collectd-plugin.

Hi @diegopacheco,

Thanks for providing the additional info. Engineering has taken a closer look at the issue, and we think what you're seeing is not a memory leak, but rather the normally high level associated with the JMX plugin.

We checked for a leak in resource consumption on this plugin using our own long-running test environment, and observed that virtual memory usage of the agent and this plugin on those long-running instances has been consistent, not increasing or accelerating, for over 3 months. We found that virtual memory usage is highest on those test instances that are using JMX-based plugins, at roughly the same value that you reported. You included the JMX plugin in the list of plugins you're using (10-jmx.conf), and we think this is the reason for the high usage you're seeing.

Our engineer also examined the statement that @mpetazzoni pointed out in his earlier comment. We'd characterize this as an unusual style choice, but not a bug, and not related to the issue you reported.

Have you observed any sudden changes or increases in resource usage? Is virtual memory usage wildly different across several hosts with the same configuration? If there's anything else we're missing, please let us know and we'll be happy to investigate further.

from signalfx-collectd-plugin.

@beccatortell

I found JMX plugin pretty buggy. Some time ago my team were passing by mistake properties of mbeans that does not exist and that make plugin suck lots of rams. A long time ago I talked with one of you QA Engineer "RAM" and I asked him if he had something to valid JMX collectD config in order to avoid this - he said was not aware of the problem. So whats the fix them?

from signalfx-collectd-plugin.

Hi @diegopacheco,

It's typical in our test environments to see this level of high virtual memory usage associated with the collectd-java plugin. This is not a leak, but rather the expected behaviour of collectd-java.

Because the issue you're seeing is not related to the SignalFx metadata plugin, I'm going to close this issue and continue the conversation over in the Support ticket that you filed with us. Thanks for reporting it.

from signalfx-collectd-plugin.