Comments (7)
daipom
commented on May 25, 2024
@pratikshavyas
Is there some reproduction condition missing?
I can't reproduce it.
When using your config, the following warnings occurred, but there was no duplication.
$ bundle add fluent-plugin-record-modifier
$ bundle exec fluentd -c test.conf
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"init supervisor logger path=nil rotate_age=nil rotate_size=nil"}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"parsing config file is succeeded path=\"/test/fluentd/config/fluent_log_label/1.conf\""}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"gem 'fluentd' version '1.16.2'"}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"gem 'fluent-plugin-record-modifier' version '2.2.0'"}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"using configuration file: <ROOT>\n <system>\n <log>\n format json\n time_format \"%Y-%m-%dT%H:%M:%S%z\"\n </log>\n </system>\n <label @FLUENT_LOG>\n <filter fluent.*>\n @type record_modifier\n enable_ruby true\n remove_keys dummy\n <record>\n log ${ { message: record[\"message\"] } }\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n time ${record.has_key?(\"time\") ? record[\"time\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\n </record>\n </filter>\n <match fluent.*>\n @type stdout\n </match>\n </label>\n</ROOT>"}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"starting fluentd-1.16.2 pid=838101 ruby=\"3.2.2\""}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"spawn command to main: cmdline=[\"/home/daipom/.rbenv/versions/3.2.2/bin/ruby\", \"-r/home/daipom/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/setup\", \"-Eascii-8bit:ascii-8bit\", \"/home/daipom/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bin/fluentd\", \"-c\", \"/test/fluentd/config/fluent_log_label/1.conf\", \"--under-supervisor\"]"}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"init worker0 logger path=nil rotate_age=nil rotate_size=nil","worker_id":0}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"adding filter in @FLUENT_LOG pattern=\"fluent.*\" type=\"record_modifier\""}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"adding match in @FLUENT_LOG pattern=\"fluent.*\" type=\"stdout\""}
{"time":"2023-12-20T15:34:06+0900","level":"warn","message":"parameter 'enable_ruby' in <filter fluent.*>\n @type record_modifier\n enable_ruby true\n remove_keys dummy\n <record>\n log ${ { message: record[\"message\"] } }\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n time ${record.has_key?(\"time\") ? record[\"time\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\n </record>\n</filter> is not used."}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"starting fluentd worker pid=838121 ppid=838101 worker=0","worker_id":0}
{"time":"2023-12-20T15:34:06+0900","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}
{"time":"2023-12-20T15:34:06+0900","level":"warn","message":"dump an error event: error_class=NoMethodError error=\"undefined method `strftime' for 1703054046:Integer\" location=\"/home/daipom/work/fluentd/fluentd/lib/fluent/time.rb:126:in `method_missing'\" tag=\"fluent.warn\" time=2023-12-20 15:34:06.794993563 +0900 record={\"message\"=>\"parameter 'enable_ruby' in <filter fluent.*>\\n @type record_modifier\\n enable_ruby true\\n remove_keys dummy\\n <record>\\n log ${ { message: record[\\\"message\\\"] } }\\n type log\\n level ${record.has_key?(\\\"level\\\") ? record[\\\"level\\\"]: \\\"unavailable\\\" }\\n time ${record.has_key?(\\\"time\\\") ? record[\\\"time\\\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\\n </record>\\n</filter> is not used.\", \"log\"=>{:message=>\"parameter 'enable_ruby' in <filter fluent.*>\\n @type record_modifier\\n enable_ruby true\\n remove_keys dummy\\n <record>\\n log ${ { message: record[\\\"message\\\"] } }\\n type log\\n level ${record.has_key?(\\\"level\\\") ? record[\\\"level\\\"]: \\\"unavailable\\\" }\\n time ${record.has_key?(\\\"time\\\") ? record[\\\"time\\\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\\n </record>\\n</filter> is not used.\"}, \"type\"=>\"log\", \"level\"=>\"unavailable\"}","worker_id":0}
2023-12-20 15:34:06.794993563 +0900 fluent.warn: {"message":"parameter 'enable_ruby' in <filter fluent.*>\n @type record_modifier\n enable_ruby true\n remove_keys dummy\n <record>\n log ${ { message: record[\"message\"] } }\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n time ${record.has_key?(\"time\") ? record[\"time\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\n </record>\n</filter> is not used.","log":{"message":"parameter 'enable_ruby' in <filter fluent.*>\n @type record_modifier\n enable_ruby true\n remove_keys dummy\n <record>\n log ${ { message: record[\"message\"] } }\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n time ${record.has_key?(\"time\") ? record[\"time\"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }\n </record>\n</filter> is not used."},"type":"log","level":"unavailable"}
{"time":"2023-12-20T15:34:06+0900","level":"warn","message":"dump an error event: error_class=NoMethodError error=\"undefined method `strftime' for 1703054046:Integer\" location=\"/home/daipom/work/fluentd/fluentd/lib/fluent/time.rb:126:in `method_missing'\" tag=\"fluent.info\" time=2023-12-20 15:34:06.795049000 +0900 record={\"pid\"=>838121, \"ppid\"=>838101, \"worker\"=>0, \"message\"=>\"starting fluentd worker pid=838121 ppid=838101 worker=0\", \"log\"=>{:message=>\"starting fluentd worker pid=838121 ppid=838101 worker=0\"}, \"type\"=>\"log\", \"level\"=>\"unavailable\"}","worker_id":0}
2023-12-20 15:34:06.795049000 +0900 fluent.info: {"pid":838121,"ppid":838101,"worker":0,"message":"starting fluentd worker pid=838121 ppid=838101 worker=0","log":{"message":"starting fluentd worker pid=838121 ppid=838101 worker=0"},"type":"log","level":"unavailable"}
{"time":"2023-12-20T15:34:06+0900","level":"warn","message":"dump an error event: error_class=NoMethodError error=\"undefined method `strftime' for 1703054046:Integer\" location=\"/home/daipom/work/fluentd/fluentd/lib/fluent/time.rb:126:in `method_missing'\" tag=\"fluent.info\" time=2023-12-20 15:34:06.795227360 +0900 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now running worker=0\", \"log\"=>{:message=>\"fluentd worker is now running worker=0\"}, \"type\"=>\"log\", \"level\"=>\"unavailable\"}","worker_id":0}
2023-12-20 15:34:06.795227360 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0","log":{"message":"fluentd worker is now running worker=0"},"type":"log","level":"unavailable"}
^C{"time":"2023-12-20T15:34:07+0900","level":"info","message":"Received graceful stop"}
{"time":"2023-12-20T15:34:08+0900","level":"info","message":"fluentd worker is now stopping worker=0","worker_id":0}
{"time":"2023-12-20T15:34:08+0900","level":"warn","message":"dump an error event: error_class=NoMethodError error=\"undefined method `strftime' for 1703054048:Integer\" location=\"/home/daipom/work/fluentd/fluentd/lib/fluent/time.rb:126:in `method_missing'\" tag=\"fluent.info\" time=2023-12-20 15:34:08.597267239 +0900 record={\"worker\"=>0, \"message\"=>\"fluentd worker is now stopping worker=0\", \"log\"=>{:message=>\"fluentd worker is now stopping worker=0\"}, \"type\"=>\"log\", \"level\"=>\"unavailable\"}","worker_id":0}
2023-12-20 15:34:08.597267239 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now stopping worker=0","log":{"message":"fluentd worker is now stopping worker=0"},"type":"log","level":"unavailable"}
{"time":"2023-12-20T15:34:08+0900","level":"info","message":"shutting down fluentd worker worker=0","worker_id":0}
{"time":"2023-12-20T15:34:08+0900","level":"info","message":"shutting down filter plugin type=:record_modifier plugin_id=\"object:be0\"","worker_id":0}
{"time":"2023-12-20T15:34:08+0900","level":"info","message":"shutting down output plugin type=:stdout plugin_id=\"object:bf4\"","worker_id":0}
{"time":"2023-12-20T15:34:08+0900","level":"info","message":"Worker 0 finished with status 0"}Then, I fixed your config as follows.
<system>
<log>
format json
time_format %Y-%m-%dT%H:%M:%S%z
</log>
</system>
<label @FLUENT_LOG>
<filter fluent.*>
@type record_modifier
- enable_ruby true
<record>
log ${ { message: record["message"] } }
type "log"
level ${record.has_key?("level") ? record["level"]: "unavailable" }
- time ${record.has_key?("time") ? record["time"]: time.strftime('%Y-%m-%dT%H:%M:%S%z') }
+ time ${record.has_key?("time") ? record["time"]: Time.now.strftime('%Y-%m-%dT%H:%M:%S%z') }
</record>
remove_keys dummy
</filter>
<match fluent.*>
@type stdout
</match>
</label>I don't know if this is the right config for you, but we won't get any more warnings.
And, there is no duplication.
$ bundle exec fluentd -c test.conf
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"init supervisor logger path=nil rotate_age=nil rotate_size=nil"}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"parsing config file is succeeded path=\"/test/fluentd/config/fluent_log_label/1.conf\""}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"gem 'fluentd' version '1.16.2'"}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"gem 'fluent-plugin-record-modifier' version '2.2.0'"}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"using configuration file: <ROOT>\n <system>\n <log>\n format json\n time_format \"%Y-%m-%dT%H:%M:%S%z\"\n </log>\n </system>\n <label @FLUENT_LOG>\n <filter fluent.*>\n @type record_modifier\n remove_keys dummy\n <record>\n log ${ { message: record[\"message\"] } }\n type log\n level ${record.has_key?(\"level\") ? record[\"level\"]: \"unavailable\" }\n time ${record.has_key?(\"time\") ? record[\"time\"]: Time.now.strftime('%Y-%m-%dT%H:%M:%S%z') }\n </record>\n </filter>\n <match fluent.*>\n @type stdout\n </match>\n </label>\n</ROOT>"}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"starting fluentd-1.16.2 pid=839277 ruby=\"3.2.2\""}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"spawn command to main: cmdline=[\"/home/daipom/.rbenv/versions/3.2.2/bin/ruby\", \"-r/home/daipom/.rbenv/versions/3.2.2/lib/ruby/site_ruby/3.2.0/bundler/setup\", \"-Eascii-8bit:ascii-8bit\", \"/home/daipom/.rbenv/versions/3.2.2/lib/ruby/gems/3.2.0/bin/fluentd\", \"-c\", \"/test/fluentd/config/fluent_log_label/1.conf\", \"--under-supervisor\"]"}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"init worker0 logger path=nil rotate_age=nil rotate_size=nil","worker_id":0}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"adding filter in @FLUENT_LOG pattern=\"fluent.*\" type=\"record_modifier\""}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"adding match in @FLUENT_LOG pattern=\"fluent.*\" type=\"stdout\""}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"starting fluentd worker pid=839297 ppid=839277 worker=0","worker_id":0}
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}
2023-12-20 15:42:40.909077337 +0900 fluent.info: {"pid":839297,"ppid":839277,"worker":0,"message":"starting fluentd worker pid=839297 ppid=839277 worker=0","log":{"message":"starting fluentd worker pid=839297 ppid=839277 worker=0"},"type":"log","level":"unavailable","time":"2023-12-20T15:42:40+0900"}
2023-12-20 15:42:40.909360120 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0","log":{"message":"fluentd worker is now running worker=0"},"type":"log","level":"unavailable","time":"2023-12-20T15:42:40+0900"}
^C{"time":"2023-12-20T15:43:07+0900","level":"info","message":"Received graceful stop"}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"fluentd worker is now stopping worker=0","worker_id":0}
2023-12-20 15:43:07.641909050 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now stopping worker=0","log":{"message":"fluentd worker is now stopping worker=0"},"type":"log","level":"unavailable","time":"2023-12-20T15:43:07+0900"}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"shutting down fluentd worker worker=0","worker_id":0}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"shutting down filter plugin type=:record_modifier plugin_id=\"object:be0\"","worker_id":0}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"shutting down output plugin type=:stdout plugin_id=\"object:bf4\"","worker_id":0}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"Worker 0 finished with status 0"}from fluentd.
pratikshavyas
commented on May 25, 2024
Hi @daipom
Thanks for the response. I see that there are duplicate logs coming. For Reference:
{"time":"2023-12-20T15:42:40+0900","level":"info","message":"fluentd worker is now running worker=0","worker_id":0}
2023-12-20 15:42:40.909360120 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now running worker=0","log":{"message":"fluentd worker is now running worker=0"},"type":"log","level":"unavailable","time":"2023-12-20T15:42:40+0900"}
{"time":"2023-12-20T15:43:07+0900","level":"info","message":"fluentd worker is now stopping worker=0","worker_id":0}
2023-12-20 15:43:07.641909050 +0900 fluent.info: {"worker":0,"message":"fluentd worker is now stopping worker=0","log":{"message":"fluentd worker is now stopping worker=0"},"type":"log","level":"unavailable","time":"2023-12-20T15:43:07+0900"}
from fluentd.
daipom
commented on May 25, 2024
@pratikshavyas
Oh! I see!
This is the expected behavior.
You can use the FLUENT_LOG label to send Fluentd's logs to other destinations.
And, the original logs are recorded as is.
So, if you want to customize the format of original Fluentd's logs, you need to consider ways not to use the FLUENT_LOG label.
from fluentd.
daipom
commented on May 25, 2024
So, if you want to customize the format of original Fluentd's logs, you need to consider ways not to use the FLUENT_LOG label.
Hmm, it would be difficult.
If you end up sending Fluentd's own logs to other destinations, you don't worry about this duplication.
You don't need to customize those original logs.
You can send only the customized logs in the FLUNET_LOG label.
<label @FLUENT_LOG>
<match fluent.*>
# You can set an output plugin to send other destinations.
...
</match>
</label>from fluentd.
pratikshavyas
commented on May 25, 2024
Thanks @daipom for the update.
from fluentd.
github-actions
commented on May 25, 2024
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days
from fluentd.
github-actions
commented on May 25, 2024
This issue was automatically closed because of stale in 7 days
from fluentd.
Related Issues (20)
- File watchers might not be handled properly causing gradual increase in CPU/Memory usage HOT 3
- in_tcp and in_syslog fail with LoadError unless oj gem is installed HOT 4
- Enable string interpolation for hash-type parameters
- Getting 503 error code in fluentd for retryable_response_codes HOT 1
- in_windows_eventlog documentation and module issues. HOT 3
- Td-agent windows fluentd connector log file is not getting rolled over (only td-agent.log file) HOT 2
- Out forward stuck establishing connection HOT 8
- Windows: td-agent 4.5.2 Too many open files HOT 1
- Buffer files remain after restart of fluentd HOT 5
- stdout P followed my infinite loop of "\\\\\\\\\" in cloudwatch log stream for fluentd
- Compression Support in out_http plugin
- fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) HOT 1
- DiskPressure due to 80 GB /home/fluentd/core. files
- Set log_path in system config HOT 2
- Duplicate logs getting created on Opensearch through fluentd HOT 4
- Startup error using docker startup to specify profile location
- Out_file short flush interval with timekey 1h in buffer HOT 5
- in_tail error after upgrading from 3.8.1 to td-agent-4.5.2 HOT 3
- Fluentd logs HOT 1
- Segmentation fault after updating from version 1.16.3 to 1.16.4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentd.