tagomoris / fluent-plugin-forest Goto Github PK
View Code? Open in Web Editor NEWLicense: Other
License: Other
Forest creates a new output plugin for each tag / path it sees. When using the file buffer the buffered data is not flushed to the output plugin upon startup (if fluentd failed previously), this is because the output plugins are not initialized until an event matching that tag arrives.
To solve this problem, forest should: 1) scan through all the configured buffer_paths, 2) list all the stored buffers in those paths, 3) regenerate the list of output plugins based on the tags found in the filename of those buffered files (e.g., by calling plant(tag)).
I'm using elasticsearch output. Every morning at 6.30, all machines stop sending data to elasticsearch and S3. I'm using fluent-plugin-forest also. Here is the error:
2015-05-14 20:05:14 +0300 [error]: Cannot output messages with tag 'elasticsearch.esData.events_2015_05_01.file.log'
2015-05-14 20:05:14 +0300 [error]: failed to configure/start sub output elasticsearch: can't create Thread: Resource temporarily unavailable
2015-05-14 20:05:14 +0300 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:110:in initialize' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:110:innew'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:110:in start' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:237:inblock in start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:237:in each' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:237:instart'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-elasticsearch-0.8.0/lib/fluent/plugin/out_elasticsearch.rb:44:in start' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:133:inblock in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in synchronize' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:inplant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in emit' /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:32:innext'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/buffer.rb:184:in block in emit' /opt/td-agent/embedded/lib/ruby/2.1.0/monitor.rb:211:inmon_synchronize'
All events from all systems are coming through to a Forest plugin to push into ElasticSearch using the first two elements of the tag as an index.
The rest of the tag is application specific meta data used by the various members of various teams.
Example of a tag; live.productA.haproxy.access
or staging.productA.nginx.error
.
As each unique tag results in a newly planted connection to ElasticSearch, a considerable number of new connections are established even tho the configuration is identical and the connection can be reused.
Present an additional argument to the matching part of the configuration which defines a 'grove' of similar trees such that even though there may be hundreds of unique trees (based on unique tags) they are grouped into common groves (based on this new config) such that they share a common connection to ES (in this example).
Perhaps something that'd let me do this:
<match **>
@type forest
grove ${tag_parts[0..2]}
subtype elasticsearch
<template>
logstash_format true
logstash_prefix ${tag_parts[0..2]}
hosts elasticsearch.priv.example.com
</template>
</match>
I'd then expect for events tagged into the system, new planted trees only exist for the grove, and not for each tag.
input tag | "grove" |
---|---|
live.product.haproxy.access | live.product.haproxy |
live.product.application.serviceA.event.subkey | live.product.application |
live.product.application.serviceB.event.otherkey | live.product.application |
I believe the change would be to the @mapping
hash, and more specifically around here
I am trying to fetch logs from AWS Cloudwatch and push them to S3. I am running td-agent (2.3.6-0) in my ubuntu 16.04. I am using the following plugins :
The issue is that the logs are not being flushed to S3 by the forest plugin. I have verified that the logs are arriving on my local instance by printing them using stdout. Can anyone help regarding what the problem can be? Below is the config file that I am using (The parts marked with $ are masked for privacy concerns).
<source>
@type cloudwatch_logs
format json
state_file /var/log/td-agent/state/$.msa.lambda.state
include_time_key true
time_key time
time_format %Y-%m-%dT%H:%M:%S.%L%Z
log_group_name /aws/lambda/$
log_stream_name 2 #[^:*]*
use_log_stream_name_prefix true
region $
aws_key_id $
aws_sec_key $
tag $.$.$.msa.lambda.log
</source>
#<match *.*.$.msa.lambda.log>
# @type stdout
#</match>
<match *.*.$.msa.lambda.log>
type copy
deep_copy true
<store>
@type forest
subtype s3
<template>
@type s3
buffer_type file
buffer_path /tmp/td-agent/buffer/${tag}.buffer_
buffer_chunk_limit 64m
buffer_queue_limit 1024
num_threads 2 # the number of threads to flush the buffer
<buffer tag_parts>
flush_at_shutdown true
</buffer>
flush_interval 30s # how long will a compressed file generated on S3
time_format %Y-%m-%dT%H:%M:%S.%L%z
# AWS Common Config
s3_region $
# AWS Credential Config
aws_key_id $
aws_sec_key $
# AWS S3 Config
s3_bucket $
s3_object_key_format %{path}%{time_slice}/${tag_parts[3]}/${tag_parts[2]}/%{index}.%{file_extension}
auto_create_bucket false
path ${tag_parts[0]}/${tag_parts[1]}/backendlog/lambda/middleware/${tag_parts[4]}.${tag_parts[5]}.${tag_parts[6]}.${tag_parts[7]}
time_slice_format %Y-%m-%d/%H #%Y%m%d%H
format json
time_key time
include_time_key true
tag_key log_source
include_tag_key true
utc # timezone
</template>
</store>
</match>
The strange thing is that, if I remove this configuration, forest plugin normally flushes other logs to S3. But as soon as I add this config and restart td-agent, forest plugin stops flushing logs and td-agent.log is stuck at the following line :
2018-12-19 07:44:14 +0000 [info]: listening fluent socket on 0.0.0.0:24224
Are there any conflicts/dependency issues between the plugins fluent-plugin-cloudwatch-logs
and fluent-plugin-forest
?
いつも大変便利にプラグインを利用させていただいております。
その中で、out_forestを用いてファイル書き出しをさせておりますが、書き出し先のファイルが変わるタイミングで、問題が起こることがありますのでご報告させて頂きます。
out_file_alternative、out_fileともに同じ現象を確認しております。
内容が不定的でふわっとした感じで大変申し訳ないのですが、ご確認いただけたら大変嬉しいです。
よろしくお願いいたします。
毎時ローテーションするように設定している場合に、切り替わりのタイミングで書き込み先ファイルが複数に分割されることがあります。
以下例の場合、3ホストからログを受信しておりますが、「access.20120820_13」が3つあり、それぞれのホストのログが別々のファイルに書き出されます。
そして、しばらく時間が経つと、1つに集約されて書き込まれるようになることもあります。
-rw-rw-rw- 1 td-agent td-agent 30824688 Aug 20 12:57 access.20120820_12.log -rw-rw-rw- 1 td-agent td-agent 111709 Aug 20 12:59 error.20120820_12.b4c7aa4a636ea546f.log -rw-rw-rw- 1 td-agent td-agent 2696514 Aug 20 12:59 access.20120820_12.b4c7aa819c350ceec.log -rw-rw-rw- 1 td-agent td-agent 5001 Aug 20 13:04 error.20120820_13.b4c7aa8bae93da00b.log -rw-rw-rw- 1 td-agent td-agent 3054839 Aug 20 13:04 access.20120820_13.b4c7aa8b3bc4f603e.log -rw-rw-rw- 1 td-agent td-agent 3048574 Aug 20 13:04 access.20120820_13.b4c7aa8b368b59743.log -rw-rw-rw- 1 td-agent td-agent 3058537 Aug 20 13:04 access.20120820_13.b4c7aa8b3659bd571.log
forward error: No such file or directory
と、
emit transaction failed error="No such file or directory
が交互に延々繰り返しエラーログに出力されます。 (意図せずtd-agentが再起動され、停まる場合もあります。)
こちらの現象は、access.20120820_15_0.log.gz などが作成されたタイミングで起こることが多いです。
fluent-plugin-file-alternative (0.1.2)
fluent-plugin-forest (0.1.0)
td-agent-1.1.8-0
<source>
type forward
</source>
<match service.**>
type forest
subtype file_alternative
remove_prefix service
<template>
time_slice_format %Y%m%d
compress gz
output_include_time false
output_include_tag false
output_data_type attr:message
add_newline true
</template>
<case access.**>
path /tmp/httpd/service/access.*.log
</case>
<case error.**>
path /tmp/httpd/service/error.*.log
</case>
</match>
2012-08-20 15:11:24 +0900: process finished code=0 2012-08-20 15:11:24 +0900: starting fluentd-0.10.25 2012-08-20 15:11:24 +0900: reading config file path="/etc/td-agent/td-agent.conf" 2012-08-20 15:11:24 +0900: adding source type="forward" 2012-08-20 15:11:24 +0900: adding match pattern="service.**" type="forest" 2012-08-20 15:11:24 +0900: adding match pattern="plack.access.**" type="file_alternative" 2012-08-20 15:11:24 +0900: adding match pattern="plack.info.**" type="file_alternative" 2012-08-20 15:11:24 +0900: adding match pattern="slowfilter.**" type="copy" 2012-08-20 15:11:24 +0900: listening fluent socket on 0.0.0.0:24224 2012-08-20 15:11:24 +0900: out_forest plants new output: file for tag 'access.host01' 2012-08-20 15:11:24 +0900: out_forest plants new output: file for tag 'access.host03' 2012-08-20 15:11:25 +0900: out_forest plants new output: file for tag 'access.host02' 2012-08-20 15:11:37 +0900: out_forest plants new output: file for tag 'error.host01' 2012-08-20 15:12:25 +0900: error on output thread error="No such file or directory - (/tmp/httpd/service/access.20120820_14.b4c7ab61c08839080.log, /tmp/httpd/service/access.20120820_14.q4c7ac64c56a4730d.log)" 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `rename' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `mv' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:155:in `enqueue' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:239:in `block (2 levels) in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:238:in `block in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:232:in `push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:498:in `block (2 levels) in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `each' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `block in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `call' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `enqueue_buffer' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:271:in `block in try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:269:in `try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:120:in `run' 2012-08-20 15:12:25 +0900: unexpected error error="No such file or directory - (/tmp/httpd/service/access.20120820_14.b4c7ab61c08839080.log, /tmp/httpd/service/access.20120820_14.q4c7ac64c56a4730d.log)" 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `rename' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `mv' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:155:in `enqueue' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:239:in `block (2 levels) in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:238:in `block in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:232:in `push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:498:in `block (2 levels) in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `each' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `block in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `call' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `enqueue_buffer' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:271:in `block in try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:269:in `try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:120:in `run' 2012-08-20 15:12:25 +0900: unexpected error while shutting down error="No such file or directory - (/tmp/httpd/service/access.20120820_14.b4c7ab61c08839080.log, /tmp/httpd/service/access.20120820_14.q4c7ac64c56a4730d.log)" 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `rename' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:69:in `mv' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/plugin/buf_file.rb:155:in `enqueue' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:239:in `block (2 levels) in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:238:in `block in push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/buffer.rb:232:in `push' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:498:in `block (2 levels) in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `each' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:496:in `block in configure' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `call' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:523:in `enqueue_buffer' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:271:in `block in try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/1.9.1/monitor.rb:211:in `mon_synchronize' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:269:in `try_flush' 2012-08-20 15:12:25 +0900: /usr/lib64/fluent/ruby/lib/ruby/gems/1.9.1/gems/fluentd-0.10.25/lib/fluent/output.rb:120:in `run' 2012-08-20 15:12:25 +0900: process finished code=0 2012-08-20 15:12:25 +0900: starting fluentd-0.10.25 2012-08-20 15:12:25 +0900: reading config file path="/etc/td-agent/td-agent.conf" 2012-08-20 15:12:25 +0900: adding source type="forward" 2012-08-20 15:12:25 +0900: adding match pattern="service.**" type="forest" 2012-08-20 15:12:25 +0900: adding match pattern="plack.access.**" type="file_alternative" 2012-08-20 15:12:25 +0900: adding match pattern="plack.info.**" type="file_alternative" 2012-08-20 15:12:25 +0900: adding match pattern="slowfilter.**" type="copy" 2012-08-20 15:12:25 +0900: listening fluent socket on 0.0.0.0:24224 2012-08-20 15:12:26 +0900: out_forest plants new output: file for tag 'access.host02' 2012-08-20 15:12:26 +0900: out_forest plants new output: file for tag 'access.host01' 2012-08-20 15:12:27 +0900: out_forest plants new output: file for tag 'access.host03' 2012-08-20 15:13:09 +0900: out_forest plants new output: file for tag 'error.host01
Hello,
when using Fuentd, I have encountered that with AWS ELB, log data from server seemed to be dropped.
Unfortunately I do not have log data remaining and fluentd server did not generate its own log data.
I know there’s not much data you can refer to, but can you please let me know if there’s something to leave a comment about the architecture or case study from others?
my AWS ELB did not have any error log.
The past architecture :
server -> ELB -> fluentd server -> public network enhancer -> on-premise
it showed daily 5%~10% log data dropping
the new architecture :
server -> fluentd server -> public network enhancer -> on-premise
and/or
game server -> public network enhancer -> on-premise
it showed 0% log data dropping
thanks
Hi,
I recently installed the most recent version of Fluentd and the Forest plugin, and I got this in my logs:
emit transaction failed: error_class=NoMethodError error="undefined method `emit' for #<Fluent::Plugin::CopyOutput>" tag=".."
/var/lib/gems/2.3.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:171:in `emit'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/compat/output.rb:170:in `process'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/bare_output.rb:53:in `emit_sync'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/event_router.rb:90:in `emit_stream'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/event_router.rb:81:in `emit'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:205:in `on_message'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:295:in `call'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:295:in `block in on_read_msgpack'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:294:in `feed_each'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:294:in `on_read_msgpack'¬
/var/lib/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/io.rb:123:in `on_readable'¬
/var/lib/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/io.rb:186:in `on_readable'¬
/var/lib/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run_once'¬
/var/lib/gems/2.3.0/gems/cool.io-1.4.4/lib/cool.io/loop.rb:88:in `run'¬
/var/lib/gems/2.3.0/gems/fluentd-0.14.1/lib/fluent/plugin/in_forward.rb:117:in `run'¬
Maybe a change introduced in Fluent 0.14?
Thanks.
Firstly thank you for the amazing plugin.
Currently, I upgrade fluentd to v0.14 because I want to use nanosecond precision. And I found the plugin README says: NOTE: This plugin will not be updated: Use Fluentd v0.14 native API to handle tags.
As what I understood, the plugin is not required if I use v0.14? Where can I find the instruction about tag usage in v0.14?
Using: td-agent/ ruby 1.8 on Centos.
$ echo '{"a":5}' | fluent-cat notwork.out
$ tail /var/log/td-agent/td-agent.log
2013-07-27 20:53:00 +0000 [info]: out_forest plants new output: copy for tag 'notwork.out'
This does not create any directories or files:
<match notwork.*>
type forest
subtype copy
host XXX
port 50070
path /user/reports/logs/staging/${tag}/%Y-%m-%d/%Y-%m-%dT%H.json
username YYY
output_include_time false
output_include_tag false
host XXX
port 50070
path /user/reports/logs/staging/${tag}.copy/%Y-%m-%d/%Y-%m-%dT%H.json
flush_interval 10s
username YYY
output_include_time false
output_include_tag false
Thank you for your time, and all of your work!
Lance
I want to ask how to avoid the problem.
I'm not sure to be appropriate to issue the problem in this place, because this is involved with other system and plugins like buf_file, fluent-plugin-s3 and out_copy.
But it would be helpful if there is a workaround against this issue.
out_forest
logged error messages like below after the instance went into lb.
2015-05-08 19:30:00 +0900 [error]: plugin/out_forest.rb:145:rescue in plant: failed to configure/start sub output copy: Net::ReadTimeout
2015-05-08 19:30:00 +0900 [error]: plugin/out_forest.rb:146:rescue in plant: /opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:158:in `rescue in rbuf_fill'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:152:in `rbuf_fill'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:134:in `readuntil'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/protocol.rb:144:in `readline'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:39:in `read_status_line'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http/response.rb:28:in `read_new'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:29:in `block in new_transport_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:26:in `catch'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/aws-sdk-v1-1.63.0/lib/aws/core/http/patch.rb:26:in `new_transport_request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1378:in `request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1371:in `block in request'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:853:in `start'
/opt/td-agent/embedded/lib/ruby/2.1.0/net/http.rb:1369:in `request'
...
...
...
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:140:rescue in plant: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffe
r_path = /tmp/td-agent/s3-buffer/media_auth_log.ip-10-0-1-49
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:141:rescue in plant: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `c
onfigure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:164:in `on_message'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:238:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:238:in `block in on_read_msgpack'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:237:in `feed_each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:237:in `on_read_msgpack'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_forward.rb:91:in `run'
2015-05-08 20:16:51 +0900 [error]: plugin/out_forest.rb:142:rescue in plant: Cannot output messages with tag 'media_auth_log.ip-10-0-1-49'
...
# same errors (= failed to configure) were repeated until the instance went out from lb.
plant method
is invoked when sub plugin is not created and a message comes.
plant method
is also failed to create sub plugin.plant method
is invoked when a second message comes because sub plugin is not created yet.
td-agent.conf
$ td-agent -c td-agent.conf
2015-05-11 10:40:21 +0900 [info]: reading config file path="td-agent.conf"
2015-05-11 10:40:21 +0900 [info]: starting fluentd-0.12.7
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-mixin-config-placeholders' version '0.3.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-mixin-plaintextformatter' version '0.2.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-amplifier-filter' version '0.1.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-bigquery' version '0.2.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-buffer-lightening' version '0.0.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-cloudwatch' version '1.2.9'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-config-expander' version '0.1.5'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-datacounter' version '0.4.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-file-sprintf' version '0.0.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-filter' version '0.0.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-flowcounter' version '0.2.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-forest' version '0.3.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-format' version '0.0.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-grep' version '0.3.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-mackerel' version '0.0.8'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-map' version '0.0.4'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-mongo' version '0.7.8'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-norikra-patched-7' version '0.2.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-ping-message' version '0.1.2'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-record-modifier' version '0.2.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-retag' version '0.0.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite' version '0.0.12'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.5.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-rewrite-tag-filter' version '1.4.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-s3' version '0.5.7'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-sampling-filter' version '0.1.3'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-scribe' version '0.10.14'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-slack' version '0.2.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-suppress' version '0.0.6'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-td' version '0.10.26'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-td-monitoring' version '0.2.0'
2015-05-11 10:40:21 +0900 [info]: gem 'fluent-plugin-webhdfs' version '0.4.1'
2015-05-11 10:40:21 +0900 [info]: gem 'fluentd' version '0.12.7'
2015-05-11 10:40:21 +0900 [info]: gem 'fluentd' version '0.10.61'
2015-05-11 10:40:21 +0900 [info]: using configuration file: <ROOT>
<source>
type http
port 24225
</source>
<match ad.*>
type forest
subtype copy
<template>
<store>
type s3
aws_key_id AK***
aws_sec_key ***
s3_bucket log
path 20150511-fluent-plugin-forest/
buffer_type file
buffer_chunk_limit 8m
buffer_queue_limit 16
buffer_path /tmp/20150511-fluent-plugin-forest/s3-buffer-${tag}
flush_interval 1m
time_slice_format %Y%m%d/%H/%Y%m%d%H_${tag}
format json
include_time_key false
include_tag_key false
</store>
<store>
type file
path /tmp/20150511-fluent-plugin-forest/${tag_parts[1]}.log
time_slice_format %Y%m%d%H
time_slice_wait 1m
buffer_type file
buffer_path /tmp/20150511-fluent-plugin-forest/buffer-${tag}
buffer_chunk_limit 64m
compress gzip
</store>
</template>
</match>
</ROOT>
2015-05-11 10:40:21 +0900 [info]: adding match pattern="ad.*" type="forest"
2015-05-11 10:40:21 +0900 [info]: adding source type="http"
I reproduced the situation by stopping any outbound network connections.
$ ping google.jp
PING google.jp (173.194.126.216) 56(84) bytes of data.
^C
--- google.jp ping statistics ---
3 packets transmitted, 0 received, 100% packet loss, time 1999ms
I sent a log to td-agent.
$ curl -X POST -d 'json={"action":"login","user":2}' http://localhost:24225/ad.s3
And, out_copy was failed to send log to s3 via out_s3 due to open timeout.
2015-05-11 10:47:45 +0900 [error]: failed to configure/start sub output copy: can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration. error = #<
Net::OpenTimeout: execution expired>
2015-05-11 10:47:45 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:152:in `rescue in check_apikeys'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:148:in `check_apikeys'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:93:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:50:in `block in start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:49:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:49:in `start'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:133:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:47:45 +0900 [error]: Cannot output messages with tag 'ad.s3'
After that, I restored outbound network connection.
$ ping google.jp
PING google.jp (173.194.126.215) 56(84) bytes of data.
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=1 ttl=58 time=1.84 ms
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=2 ttl=58 time=1.95 ms
64 bytes from nrt04s07-in-f23.1e100.net (173.194.126.215): icmp_seq=3 ttl=58 time=1.86 ms
^C
--- google.jp ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2002ms
rtt min/avg/max/mdev = 1.846/1.889/1.959/0.061 ms
I sent a log to td-agent.
$ curl -X POST -d 'json={"action":"login","user":2}' http://localhost:24225/ad.s3
out_copy was failed to be configure.
2015-05-11 10:49:12 +0900 [error]: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/20150511-fluent-plugin-forest/s3-buffer-ad.s3
2015-05-11 10:49:12 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:49:12 +0900 [error]: Cannot output messages with tag 'ad.s3'
This error was repeated against every log.
2015-05-11 10:59:02 +0900 [error]: failed to configure sub output copy: Other 's3' plugin already use same buffer_path: type = s3, buffer_path = /tmp/20150511-fluent-plugin-forest/s3-buffer-ad.s3
2015-05-11 10:59:02 +0900 [error]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/buf_file.rb:97:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:197:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/output.rb:488:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-mixin-config-placeholders-0.3.0/lib/fluent/mixin/config_placeholders.rb:105:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-s3-0.5.7/lib/fluent/plugin/out_s3.rb:42:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:43:in `block in configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `each'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/out_copy.rb:34:in `configure'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:132:in `block in plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `synchronize'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:128:in `plant'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-forest-0.3.0/lib/fluent/plugin/out_forest.rb:168:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:88:in `emit_stream'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/event_router.rb:79:in `emit'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:167:in `on_request'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `call'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:334:in `on_message_complete'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `<<'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:236:in `on_read'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:123:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/io.rb:186:in `on_readable'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run_once'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.3.0/lib/cool.io/loop.rb:88:in `run'
/opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.7/lib/fluent/plugin/in_http.rb:113:in `run'
2015-05-11 10:59:02 +0900 [error]: Cannot output messages with tag 'ad.s3'
Thank you for reading a long post.
__TAG__
designator is not recognised with symlink_path.
Without __TAG__
everything works fine. With one - symlink is not created.
<store>
@type forest
subtype file
remove_prefix docker
<template>
format csv
force_quotes false
delimiter \t
include_time_key true
time_format %Y-%m-%d %H:%M:%S
path /var/log/td-agent/__TAG__
symlink_path /var/log/td-agent/__TAG__-latest.log
append true
time_slice_format %Y%m%d
time_slice_wait 10m
fields time, message
</template>
forestでファイルバッファを使用する際は
buffer_path /var/flunet/buffer/xxxxx.TAG
とTAGを指定しないとforestで生成されたインスタンスが同じバッファファイルを使用してしまうため
あるインスタンスがバッファファイルを処理すると他のインスタンスでファイル無しのエラーが発生します
Would it be possible to get a feature added so that I could use year/month/day as placeholders?
For example:
type file
path /data/logging/fluentd/${year}/${month}/${day_of_month}${hostname}.*.log
So that I would get files for today like:
/data/logging/fluentd/2014/03/05/some_host.timeslice.log
And tomorrow it would be in /data/logging/fluentd/2014/03/06/some_host.timeslice.log ?
I would love to use this with the file output plugin as well as the S3 output plugin, but we have hundreds of hosts and this would mean hundreds of files, multiplied by the timeslice, every single day, in the same directory.
This would make it easier for me to search logs and clean up logs later as well.
Forest does ALMOST exactly what I need it to otherwise!! Thanks!!
$ cat fluent.conf
<source>
type forward
</source>
<match nodelog.*>
type forest
subtype file
<template>
path /home/fluentd/test/nodelog.__TAG__
flush_interval 3s
</template>
</match>
$ fluentd -c fluent.conf --daemon pid.log -o debug.log -vv
$ echo '{"hello":"world"}' | fluent-cat nodelog.hogehoge
(...and the file buffer never gets created UPDATE minor edits to make the above commands correct)
It looks like it stalls here (not 100% sure)
The above configuration worked with ruby 2.2.0dev (2014-04-20 trunk 45641) [x86_64-linux]
im using td-agent 1.2.2.
i configed td-agent.conf like this.
<match a.*.**>
@type copy
<store>
@type forest
subtype file
<template>
path /${tag[0]}/${tag[1]}-${tag[2]}/${tag[1]}-${tag[2]}-%Y-%m-%d
<format>
@type json
</format>
<buffer tag, time>
@type file
chunk_limit_size 15k
path /a/b/${tag}.buffer
</buffer>
</template>
<case a.b.a>
symlink_path /a/b/a/a-b-a-current.log
</case>
<case a.b.b>
symlink_path /a/b/b/a-b-b-current.log
</case>
<case a.b.c>
symlink_path /a/b/c/a-b-c-current.log
</case>
</store>
</match>
i want to configure symlink_path with tags. (${tag[0]} or ${tag_parts[1]}, etc)
but fluentd don't support configuring symlink_path with tag-included path
so i had to define multiple statements like above conf.
but it causing error.
following is td-agent.log
2018-12-10 18:27:37 +0900 [warn]: #0 emit transaction failed: error_class=Errno::ENOENT error="No such file or directory @ rb_file_s_symlink - (/var/log/td-agent/buffer/file/a.b.b.buffer/buffer.b57ca793668322fdd83c701f8d4491ed2.log, /a/b/b/a-b-b-current.log)" location="/opt/td-agent/embedded/lib/ruby/2.4.0/fileutils.rb:321:in `symlink'" tag="a.b.b"
2018-12-10 18:27:37 +0900 [warn]: #0 suppressed same stacktrace
2018-12-10 18:27:37 +0900 [error]: #0 unexpected error on reading data host="111.111.111.111" port=19063 error_class=Errno::ENOENT error="No such file or directory @ rb_file_s_symlink - (/var/log/td-agent/buffer/file/a.b.b.buffer/buffer.b57ca793668322fdd83c701f8d4491ed2.log, /a/b/b/a-b-b-current.log)"
2018-12-10 18:27:37 +0900 [error]: #0 suppressed same stacktrace
can i using this conf?
I've think the statement in README:
NOTE: This plugin will not be updated: Use Fluentd v0.14 native API to handle tags
Should be explained somehow, how configure Fluentd to achieve functionality similar to fluent-plugin-forest.
hi. im using td-agent 1.2.2.
in my conf, i must use 2 sets with ,
for example,
<match *>
@type forest
subtype copy
<template>
<store>
@type file
# 'symlink_path' with ${tag[1]} doesn't working. so, i must seperate each tag's 'symlink_path' commands
<template>
</template>
<case tag1>
symlink_path /var/log/tag1/tag1-current.log
</case>
<case tag2>
symlink_path /var/log/tag2/tag2-current.log
</case>
</store>
</template>
<case tag1>
# do not send to elasticsearch
</case>
<case *>
# send to elasticsearch except for 'tag1'
</case>
</match>
the problem is, 'symlink_path' attribute in td-agent.conf doesn't working with ${tag[0]} like this
# doesn't working
symlink_path /var/log/${tag[0]}/${tag[0]}-current.log
so, i must use template-case sets.
of course, i can using symlink_path string like this, and this confing is working correctly.
# when using ${tag} only, it's working
# but i want to move the log's location to other path like /var/log/${tag[0]}/${tag[0]}-current.log.
symlink_path /var/log/${tag}-current.log
and there are some logs that must be sent to elasticsearch or not like this
</template>
<case tag1>
# do not send to elasticsearch
</case>
<case *>
# send to elasticsearch except for 'tag1'
</case>
so, i must use template-case sets.
but above conf doesn't working
how can i handling this problem??
Since fluentd 0.14 I can't make buffered outputs in a forest work anymore.
With a simple configuration like this:
<match **>
@type forest
subtype file
<template>
path /tmp/foo-__TAG__
buffer_type memory
flush_interval 1s
</template>
</match>
It seems buffers aren't flushed anymore.
(all works fine in a non-buffered configuration, and the buffered version works fine without forest)
I'm running the latest fluentd 0.14.6 and fluent-plugin-forest 0.3.1.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.