patant / fluent-plugin-loggly Goto Github PK

View Code? Open in Web Editor NEW

13.0 13.0 24.0 32 KB

Fluentd plugin for output to loggly

License: Apache License 2.0

Ruby 100.00%

fluent-plugin-loggly's People

Contributors

Stargazers

Watchers

Forkers

rtaylor-r7 dekked webmonarch htgc atlassian maltej salekseev jorge-rodriguez okkez chris-codaio njam cosmo0920 zazabe aboyett weaveworks baosi1234 xlogin tjochums cryptophobia

fluent-plugin-loggly's Issues

Dependency error for net-http-persistent

Hi our work with the Loggly team and we just got this bug report from a customer. We are posting it here because it looks like we need to update a dependency in this library. I pasted his email below. What's the best way to address this? Would you be open to a pull request that we submit?

"I just pushed a new build of our AMI that uses fluentd to log to Loggly,
and things stopped working. Some diagnosis later, I discovered that a new
version of the 'net-http-persistent' has been released (3.0.0 as opposed to
2.9.4, which was working before), and it looks like the 'initialize'
function of that module doesn't like what the loggly module passes to it.
Here's a stacktrace to show what I'm seeing:

2016-10-13 17:27:17 +0000 [error]: unexpected error
error_class=ArgumentError error=#<ArgumentError: wrong number of
arguments (2 for 0)>
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:505:in
initialize' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly_buffered.rb:43:in new'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly_buffered.rb:43:in
start' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/out_copy.rb:55:in block in start'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/out_copy.rb:54:in
each' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/plugin/out_copy.rb:54:in start'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/agent.rb:70:in
block in start' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/agent.rb:69:in each'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/agent.rb:69:in
start' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/root_agent.rb:108:in start'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/engine.rb:237:in
start' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/engine.rb:187:in run'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:570:in
run_engine' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:162:in block in start'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:366:in
call' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:366:in main_process'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:339:in
block in supervise' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:338:in fork'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:338:in
supervise' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/supervisor.rb:156:in start'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/lib/fluent/command/fluentd.rb:173:in
<top (required)>' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in require'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in
require' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.26/bin/fluentd:5:in <top (required)>'
2016-10-13 17:27:17 +0000 [error]:
/opt/td-agent/embedded/bin/fluentd:23:in load' 2016-10-13 17:27:17 +0000 [error]: /opt/td-agent/embedded/bin/fluentd:23:in <top (required)>'
2016-10-13 17:27:17 +0000 [error]: /usr/sbin/td-agent:7:in load' 2016-10-13 17:27:17 +0000 [error]: /usr/sbin/td-agent:7:in

I have a temporary solution - downgrade the net-http-persistent gem in my
environment - and I know this isn't strictly Loggly's problem, but I am
not Ruby-friendly and was wondering if an adjustment could be made to the
Loggly fluentd plugin to account for this change in that module's function
signature."

Unknown output plugin 'loggly'

I'm able to successfully install the fluent-plugin-loggly but when I actually start td-agent, I receive the following error:

2015-09-10 00:16:20 +0000 [error]: fluent/supervisor.rb:313:rescue in main_process: config error 
file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'loggly'. Run 'gem search -rd 
fluent-plugin' to find plugins"

Here's a full transcript of what I'm doing. Is this a bug in the plugin, or am I doing something wrong?

root@6366f4a521db:/# ruby -v
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-linux]
root@6366f4a521db:/# gem install fluent-plugin-loggly
Successfully installed fluent-plugin-loggly-0.0.6
1 gem installed
Installing ri documentation for fluent-plugin-loggly-0.0.6...
Installing RDoc documentation for fluent-plugin-loggly-0.0.6...
root@6366f4a521db:/# /etc/init.d/td-agent start
 * Starting td-agent  td-agent                                                                                                                                                                       
2015-09-10 00:18:49 +0000 [error]: fluent/supervisor.rb:313:rescue in main_process: config error file="/etc/td-agent/td-agent.conf" error="Unknown output plugin 'loggly'. Run 'gem search -rd fluent-plugin' to find plugins"

Here's my td-agent.conf file:

<source>
  type udp
  tag mytag
  format json
  port 5140
</source>

<match **>
  type loggly
  loggly_url https://logs-01.loggly.com/inputs/TOKEN_CODE/tag/fluentd
</match>

Also, I'm running fluentd in a Docker container, and here's the Dockerfile:

FROM phusion/baseimage:0.9.16

RUN apt-get update && \
    apt-get -y upgrade && \
    # Install the supported version of fluentd, "td-agent"
    curl https://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add - && \
    echo "deb [arch=amd64] http://packages.treasuredata.com/2/ubuntu/trusty/ trusty contrib" > /etc/apt/sources.list.d/treasure-data.list && \
    apt-get update && \
    apt-get install -y --force-yes td-agent && \
    apt-get install -y ruby && \
    gem install fluent-plugin-loggly && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

ADD files/td-agent.conf /etc/td-agent/td-agent.conf

ADD files/init.d/td-agent /etc/init.d/td-agent
RUN chmod 0700 /etc/init.d/td-agent

EXPOSE 5140

CMD ["/etc/init.d/td-agent", "start"]

http errors from Loggly are swallowed, so fluentd does not retry

The code logs errors but does not raise them upwards, so anything transient like a timeout or internal error at Loggly means the data is thrown away.

If this is an official Loggly repo, then please put it under github/loggly

If this is an official Loggly repo, then please put it under https://github.com/loggly

Otherwise, please make it very clear that this is not an official repo.

HTTP proxy server setting

How do I set HTTP proxy server setting ?

Buffered records are combined in Loggly?

I'm sending test records to Fluentd with:

echo "{\"hello\":$(shuf -i1-10 -n1)}" | netcat -q 0 127.0.0.1 5170

These entries are sent to Loggly using the "buffered" plugin:

<match **>
  @type loggly_buffered
  loggly_url https://logs-01.loggly.com/inputs/<api-key>/tag/test-reto
  buffer_type memory
  flush_interval 1s
</match>

The fluentd log shows me that the records are sent to Loggly correctly:

2016-04-05 10:05:36 +0000 [debug]: plugin/out_loggly_buffered.rb:60:write: 4 records sent
2016-04-05 10:05:36 +0000 [debug]: plugin/out_loggly_buffered.rb:65:write: HTTP Response code 200
2016-04-05 10:05:37 +0000 [debug]: plugin/out_loggly_buffered.rb:60:write: 5 records sent
2016-04-05 10:05:37 +0000 [debug]: plugin/out_loggly_buffered.rb:65:write: HTTP Response code 200
2016-04-05 10:05:38 +0000 [debug]: plugin/out_loggly_buffered.rb:60:write: 2 records sent
2016-04-05 10:05:38 +0000 [debug]: plugin/out_loggly_buffered.rb:65:write: HTTP Response code 200
2016-04-05 10:05:39 +0000 [debug]: plugin/out_loggly_buffered.rb:60:write: 1 records sent
2016-04-05 10:05:39 +0000 [debug]: plugin/out_loggly_buffered.rb:65:write: HTTP Response code 200

But then in Loggly these records show up combined/aggregated per second:

Am I doing something wrong?

JSON encoding errors cause infinite retry and data loss

Yajl::Encoder.encode() raises some errors, such as trying to encode "Infinity" values, as exceptions.
These escape to fluentd, which retries, again and again, until either the retry limit is hit and the whole buffer is dropped, or the buffer space is completely full and all buffers are dropped.

Debugging to run this plugin successfully

I've been struggling to get this plugin working successfully. I'd like to be able to debug what's going on, but I can't seem to find the plugin's log entries anywhere.

In the code, I see entries like https://github.com/patant/fluent-plugin-loggly/blob/master/lib/fluent/plugin/out_loggly.rb#L30. But in the /var/log/td-agent/td-agent.log I see none of these entries.

I'm outputting logs both to a local file and to Loggly. The local file works fine but loggly output is not working. Any thoughts on how I can resolve this?

Here's my td-agent.conf (In my actual file, I have #comments which I've removed below):

<source>
  type forward
  port 5140
  bind 0.0.0.0
</source>

<match **>
  type file
  path /tmp/sample.log
</match>

<match **>
  type                loggly_buffered
  loggly_url          https://logs-01.loggly.com/bulk/<token>/tag/fluentd
  output_include_time true  # add 'timestamp' record into log. (default: true)
  buffer_type         file
  buffer_path         /tmp/buffer
  flush_interval      2s
</match>

I've also confirmed that directly calling loggly's API via curl works fine from my box. I've also successfully installed the fluent-plugin-loggly gem by running it in a Docker container with this Dockerfile:

FROM phusion/baseimage:0.9.16

RUN apt-get update && \
    apt-get -y upgrade && \
    # Install the supported version of fluentd, "td-agent"
    curl https://packages.treasuredata.com/GPG-KEY-td-agent | apt-key add - && \
    echo "deb [arch=amd64] http://packages.treasuredata.com/2/ubuntu/trusty/ trusty contrib" > /etc/apt/sources.list.d/treasure-data.list && \
    apt-get update && \
    apt-get install -y --force-yes td-agent && \
    td-agent-gem install fluent-plugin-loggly && \
    apt-get clean && \
    rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*

ADD files/td-agent.conf /etc/td-agent/td-agent.conf

ADD files/init.d/td-agent /etc/init.d/td-agent
RUN chmod 0700 /etc/init.d/td-agent

EXPOSE 5140

CMD ["/etc/init.d/td-agent", "start"]

At this point, I may just use the generic exec plugin, but that seems like a shame given that all of this nice work is done here. Thanks for your input!

Error: wrong number of arguments (given 2, expected 0)

Hi,

My environment is: td-agent 4.4.1 fluentd 1.15.2 (c32842297ed2c306f1b841a8f6e55bdd0f1cb27f)
My fluentd config is:

<match **>
  @type loggly
  loggly_url https://my_app.loggly.com/inputs/my_api_token
</match>

When I start the fluentd, I got this error immediately:

2023-02-02 19:03:40 -0800 [info]: #0 starting fluentd worker pid=137531 ppid=137518 worker=0
2023-02-02 19:03:40 -0800 [error]: #0 unexpected error error_class=ArgumentError error="wrong number of arguments (given 2, expected 0)"
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/net-http-persistent-4.0.1/lib/net/http/persistent.rb:471:in `initialize'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-loggly-0.0.9/lib/fluent/plugin/out_loggly_buffered.rb:43:in `new'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-loggly-0.0.9/lib/fluent/plugin/out_loggly_buffered.rb:43:in `start'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/compat/call_super_mixin.rb:42:in `start'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:203:in `block in start'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:192:in `block (2 levels) in lifecycle'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:191:in `each'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:191:in `block in lifecycle'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:178:in `each'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:178:in `lifecycle'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/root_agent.rb:202:in `start'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:248:in `start'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/engine.rb:147:in `run'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:764:in `block in run_worker'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:1037:in `main_process'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/supervisor.rb:755:in `run_worker'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/lib/fluent/command/fluentd.rb:381:in `<top (required)>'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/2.7.0/rubygems/core_ext/kernel_require.rb:83:in `require'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.15.2/bin/fluentd:15:in `<top (required)>'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/bin/fluentd:23:in `load'
  2023-02-02 19:03:40 -0800 [error]: #0 /opt/td-agent/bin/fluentd:23:in `<top (required)>'
  2023-02-02 19:03:40 -0800 [error]: #0 /usr/sbin/td-agent:15:in `load'
  2023-02-02 19:03:40 -0800 [error]: #0 /usr/sbin/td-agent:15:in `<main>'
2023-02-02 19:03:40 -0800 [error]: #0 unexpected error error_class=ArgumentError error="wrong number of arguments (given 2, expected 0)"
  2023-02-02 19:03:40 -0800 [error]: #0 suppressed same stacktrace
2023-02-02 19:03:40 -0800 [error]: Worker 0 finished unexpectedly with status 1

I wonder if anyone encounter the same issue or is it my config wrong?

When there is no connection to loggly webservice, plugin fails to shutdown

related to td-agent v3.
When there is no connection to loggly webservice due to networking problems, plugin fails to shutdown and the whole td-agent process is stuck for about a minute.

200 Success responses, but no data appears in Loggly dashboard

200 Success responses, but no data appears in Loggly dashboard:

Same data, same everything -- showed up instantly in Logz.io dashboard:

Broken with net-http-persistent 3.0

2016-10-08 01:43:31 +0000 [error]: unexpected error error_class=ArgumentError error=#<ArgumentError: wrong number of arguments (given 2, expected 0)>
  2016-10-08 01:43:31 +0000 [error]: /home/fluent/.gem/ruby/2.3.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:505:in `initialize'
  2016-10-08 01:43:31 +0000 [error]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly.rb:37:in `new'
  2016-10-08 01:43:31 +0000 [error]: /home/fluent/.gem/ruby/2.3.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly.rb:37:in `start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:71:in `block in start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:70:in `each'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:70:in `start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/root_agent.rb:108:in `start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/engine.rb:237:in `start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/engine.rb:187:in `run'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:570:in `run_engine'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:162:in `block in start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:366:in `main_process'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:339:in `block in supervise'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:338:in `fork'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:338:in `supervise'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:156:in `start'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
  2016-10-08 01:43:31 +0000 [error]: /usr/lib/ruby/gems/2.3.0/gems/fluentd-0.12.29/bin/fluentd:5:in `<top (required)>'
  2016-10-08 01:43:31 +0000 [error]: /usr/bin/fluentd:23:in `load'
  2016-10-08 01:43:31 +0000 [error]: /usr/bin/fluentd:23:in `<main>'

Please adjust for ruby2.0

My platform is work with ruby2.0.
I can not install fluent-plugin-loggly.

[root@watcher ruby-2.0.0-p353]# ruby -v
ruby 2.0.0p353 (2013-11-22 revision 43784) [x86_64-linux]
[root@watcher ruby-2.0.0-p353]# gem install fluent-plugin-loggly
ERROR:  Error installing fluent-plugin-loggly:
    fluent-plugin-loggly requires Ruby version ~> 1.9.2.

I hope this plugin upgraded.

Thanks,

Multi process workers

Hello,

Currently i'm using fluent-plugin-loggly and i want to use multi proccess worker, but the loggly plugin can't start.

So how can use this plugin with multi process workers ?

Thanks for this great job 👍

is it possible to add tag depending on match?

  <match your_match>
      type loggly
      loggly_url https://logs-01.loggly.com/inputs/xxx-xxxx-xxxx-xxxxx-xxxxxxxxxx
  </match>

it will post messages to http tag on loggly system by default. or it set tag if /tag/[you-tag] appended to loggly_url
Is it possible to post messages to matched tag?

Error during plugin load

Hi there!
td-agent can't load your pluggin.

$ gem -v
2.4.8

Here are the logs:

2016-10-13 01:29:02 +0300 [info]: starting fluentd-0.12.29
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-mixin-config-placeholders' version '0.4.0'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-kafka' version '0.3.1'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-loggly' version '0.0.8'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-mongo' version '0.7.15'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.5'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-s3' version '0.7.1'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-td' version '0.10.29'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.2'
2016-10-13 01:29:02 +0300 [info]: gem 'fluent-plugin-webhdfs' version '0.4.2'
2016-10-13 01:29:02 +0300 [info]: gem 'fluentd' version '0.12.29'
2016-10-13 01:29:02 +0300 [info]: adding match pattern="**" type="stdout"
2016-10-13 01:29:02 +0300 [info]: adding match pattern="**" type="loggly"
2016-10-13 01:29:02 +0300 [info]: adding source type="forward"
2016-10-13 01:29:02 +0300 [info]: using configuration file: <ROOT>
  <source>
    @type forward
    port 24224
    bind 0.0.0.0
  </source>
  <match **>
    @type stdout
  </match>
  <match **>
    type loggly
    loggly_url https://logs-01.loggly.com/inputs/mytoken
  </match>
</ROOT>
2016-10-13 01:29:02 +0300 [error]: unexpected error error_class=ArgumentError error=#<ArgumentError: wrong number of arguments (2 for 0)>
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/net-http-persistent-3.0.0/lib/net/http/persistent.rb:505:in `initialize'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly.rb:37:in `new'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-loggly-0.0.8/lib/fluent/plugin/out_loggly.rb:37:in `start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:71:in `block in start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:70:in `each'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/agent.rb:70:in `start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/root_agent.rb:108:in `start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/engine.rb:237:in `start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/engine.rb:187:in `run'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:570:in `run_engine'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:162:in `block in start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:366:in `call'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:366:in `main_process'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:339:in `block in supervise'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:338:in `fork'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:338:in `supervise'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/supervisor.rb:156:in `start'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/site_ruby/2.1.0/rubygems/core_ext/kernel_require.rb:54:in `require'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.29/bin/fluentd:5:in `<top (required)>'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `load'
  2016-10-13 01:29:02 +0300 [error]: /opt/td-agent/embedded/bin/fluentd:23:in `<top (required)>'
  2016-10-13 01:29:02 +0300 [error]: /usr/sbin/td-agent:7:in `load'
  2016-10-13 01:29:02 +0300 [error]: /usr/sbin/td-agent:7:in `<main>'
2016-10-13 01:29:02 +0300 [info]: shutting down fluentd
2016-10-13 01:29:02 +0300 [info]: shutting down output type="stdout" plugin_id="object:3ff5058c2bb8"
2016-10-13 01:29:02 +0300 [info]: process finished code=0
2016-10-13 01:29:02 +0300 [warn]: process died within 1 second. exit.

Upgrade to fluentd 0.14 API

Fluentd 0.14 introduced a new plugin API: http://www.fluentd.org/blog/fluentd-v0.14.0-has-been-released

It would be great if I could use the new "dynamic configurations":

support formatting configurations dynamically (e.g., path /my/dest/${tag}/mydata.%Y-%m-%d.log)

This way I could use my fluentd tags as my Loggly tags by setting the URL to:

loggly_url https://logs-01.loggly.com/bulk/xxxxxxxxxxxx/tag/${tag}