repeatedly / fluent-plugin-record-modifier Goto Github PK

View Code? Open in Web Editor NEW

118.0 6.0 41.0 119 KB

Fluentd filter plugin for modifying each event record

Ruby 100.00%

fluent-plugin-record-modifier's People

Stargazers

Watchers

Forkers

choplin sonots loveshell ixixi tagomoris ma2shita josephsu chickenlove lberruti kumarnmanoj r00tak cosmo0920 mshibuya frits-v builtbybrayne orihoch sebastiendarocha cajbecu nationalprostaff jimismith akashagrawal alexfouche nysalor smallcreep ferbar k-hal zp-alwayswin pdecat jvassev hellfosa shaikatz goashut daisukehayata ogidow kenhys znorris vparfonov marijus-ravickas robertleeuwis wolfi-chainguard-demo chainguard-wolfi-bites-back

fluent-plugin-record-modifier's Issues

Record Modifer on Windows Server doesn't execute Ruby code

I am running the Fluentd agent on Windows Server 2016.
I also installed the plug-in "fluent-plugin-grafana-loki" to allow me to output messages to loki.
I'm attempting to add the hostname as a label to my messages.
I found that when type=file or type=loki, the ruby code in the filter statements below don't work (don't even seem to run):

<match myfile>
  @type file
  path /var/log/td-agent/access
  <label>
    host
  </label>
</match>

<filter myfile>
  @type record_modifier
  <record>
    host "#{Socket.gethostname}"
  </record>
</filter>

<filter myfile>
  @type record_modifier
  prepare_value require 'socket'; @name = Socket.gethostname
  <record>
    host ${@name.downcase}
  </record>
</filter>

Any suggestions?

Fluentd crashes when plugin initialized on fluentd-0.12.42 (using v0.12-debian-onbuild docker image)

Hi,

I'm using fluent/fluentd:v0.12-debian-onbuild to build a custom image with extra plugins, for some reason this plugin fails to initialize when building with latest fluentd-v0.12.42

2017-12-16 19:00:10 +0000 [info]: reading config file path="/fluentd/etc/fluentd.conf"
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [warn]: '@' is the system reserved prefix. It works in the nested configuration for now but it will be rejected: @timestamp
2017-12-16 19:00:10 +0000 [info]: starting fluentd-0.12.42
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-add-uuid' version '0.1'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-anomalydetect' version '0.1.7'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-detect-exceptions' version '0.0.8'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-elasticsearch' version '1.13.0'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-forest' version '0.3.3'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-genhashvalue' version '0.04'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-kubernetes_metadata_filter' version '1.0.0'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-multiprocess' version '0.2.2'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-prometheus' version '0.4.0'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-record-modifier' version '1.0.0'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-record-reformer' version '0.9.1'
2017-12-16 19:00:10 +0000 [info]: gem 'fluent-plugin-systemd' version '0.0.9'
2017-12-16 19:00:10 +0000 [info]: gem 'fluentd' version '0.12.42'
2017-12-16 19:00:10 +0000 [info]: adding match pattern="fluent.**" type="null"
2017-12-16 19:00:10 +0000 [info]: adding filter pattern="elastic.buffers" type="record_modifier"
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- fluent/plugin/filter (LoadError)
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/lib/gems/2.3.0/gems/fluent-plugin-record-modifier-1.0.0/lib/fluent/plugin/filter_record_modifier.rb:1:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin.rb:173:in `block in try_load_plugin'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin.rb:170:in `each'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin.rb:170:in `try_load_plugin'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin.rb:130:in `new_impl'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/plugin.rb:63:in `new_filter'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/agent.rb:143:in `add_filter'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/agent.rb:62:in `block in configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/agent.rb:57:in `each'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/agent.rb:57:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/root_agent.rb:86:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/engine.rb:129:in `configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/engine.rb:103:in `run_configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:489:in `run_configure'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:160:in `block in start'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:366:in `main_process'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:339:in `block in supervise'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:338:in `fork'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:338:in `supervise'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/supervisor.rb:156:in `start'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/lib/fluent/command/fluentd.rb:173:in `<top (required)>'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
        from /var/lib/gems/2.3.0/gems/fluentd-0.12.42/bin/fluentd:8:in `<top (required)>'
        from /usr/local/bin/fluentd:22:in `load'
        from /usr/local/bin/fluentd:22:in `<main>'
2017-12-16 19:00:10 +0000 [info]: process finished code=256
2017-12-16 19:00:10 +0000 [warn]: process died within 1 second. exit.

This is my Dockerfile:

FROM fluent/fluentd:v0.12-debian-onbuild

RUN ulimit -n 65536

RUN buildDeps="sudo make gcc g++ libc-dev ruby-dev curl telnet tcpdump" \
 && apt-get update \
 && apt-get install -y --no-install-recommends $buildDeps \

 && gem install \
        fluent-plugin-elasticsearch:1.13.0 \
        fluent-plugin-record-reformer \
        fluent-plugin-anomalydetect \
        fluent-plugin-systemd:0.0.9 \
        fluent-plugin-kubernetes_metadata_filter \
        fluent-plugin-detect-exceptions \
        fluent-plugin-multiprocess \
        fluent-plugin-prometheus:0.4.0 \
        fluent-plugin-record-modifier \
        fluent-plugin-forest \
        fluent-plugin-genhashvalue:0.04 \
        fluent-plugin-add-uuid \
        strptime \

 && gem sources --clear-all \
 && SUDO_FORCE_REMOVE=yes \
    apt-get purge -y --auto-remove \
                  -o APT::AutoRemove::RecommendsImportant=false \
                  $buildDeps \
 && rm -rf /var/lib/apt/lists/* \
           /home/fluent/.gem/ruby/2.3.0/cache/*.gem

placeholders are not handled in nested fkeys

Given the below configuration,

<filter fluentd.**>
  @id fluentd_healthcheck
  @type record_modifier
  <record>
    host "#{ENV['K8S_NODE_NAME']}"
    kubernetes {
      "host": "#{ENV['K8S_NODE_NAME']}",
      "pod_name": "#{Socket.gethostname}"
    }
    origin fluentd_internal
  </record>
</filter>

Results in (formatted for readability):

fluentd-agent-v2-ggtkp fluentd-agent 2018-08-04 10:05:12 +0000 [warn]: #0 dump an error event:
error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError
error="400 - Rejected by Elasticsearch" location=nil tag="fluentd.pod.healthcheck" 
time=2018-08-04 10:05:07.480889198 +0000 
record={"log"=>"health check", "host"=>"ip-10-0-4-13.us-west-2.compute.internal", 
"kubernetes"=>"{\"host\":\"\#{ENV['K8S_NODE_NAME']}\",\"pod_name\":\"\#{Socket.gethostname}\"}", 
"origin"=>"fluentd_internal", "_hash"=>"MzA2ZTliZDQtYThlNy00ODM4LTg0MTI

encodings of char_encoding

Hi!
Could you please clarify which encodings char_encoding supports?

New fields added must be added to whitelist_keys (if it's used)

First, thanks for this plugin! :)

We've noticed that if whitelist_keys is used and new fields are added using <record>...</record>, these fields must also be added to whitelist_keys or we don't see them in emitted events, in a sense it make sense but would be nice if these fields would ne whitelisted by default.

Example:

<filter trans>
  @type record_modifier
  <record>
    project <something-here>
    docker_env <something-there>
  </record>
  whitelist_keys HOSTNAME,MESSAGE,CONTAINER_NAME,project,docker_env
</filter>

In the above example, if we don't include project and docker_env in whitelist_keys we don't get them.

replace is not woking

I use replace \k/\k in the configuration file.
but in kibana it still show \k\k

Label support seems to have a glitch

It looks to me like labels are not being followed.
I duplicate my log flow into two streams using labels,
and the second label (@EVENTLOG.LOGICMONITOR) does not receive keys which are removed in the first label (@EVENTLOG.SCALYR)
If I use "@type record_transformer" instead of "@type record_modifier" these two log flows are processed independently as expected.

<match eventlog.syslog.**>
  @type copy
  <store>
    @type relabel
    @label @EVENTLOG.SCALYR
  </store>
  <store>
    @type relabel
    @label @EVENTLOG.LOGICMONITOR
  </store>
</match>

<label @EVENTLOG.SCALYR>
  <filter eventlog.syslog.**>
    @type record_modifier
    remove_keys logline,pri,ident,severity
  </filter>
  <match eventlog.syslog.**>
    @type stdout
  </match>
</label>

<label @EVENTLOG.LOGICMONITOR>
  <filter eventlog.syslog.**>
    @type record_modifier
    <record>
      message ${record["node"]}:${record["ident"]}:${record["severity"]} ${record["logline"]}
    </record>
    remove_keys logline
  </filter>
  <match eventlog.syslog.**>
    @type stdout
  </match>
</label>

frozen strings from in_monitor_agent cause RuntimeError

It appears that the in_monitor_agent plugin uses frozen strings (for performance?). The filter_record_modifier plugin will attempt (in cases of broad wildcards, which I would like to use) to convert these keys, when the char_encoding parameter is set. This causes a runtime error.

I'd like to be able to set every event coming through my aggregators to be UTF-8. Is it possible to add a check for frozen strings, and duplicate them to be no longer frozen?

Record modifier is not adding new fields

Greetings,

I'm using this plugin to add new metadata to my logs, but it seems it fails to add new fields.
My td-agent.conf have a lot of different sources
Here is an example of one of them:

<source>
  @type tail
  path /path/to/file.log
#  pos_file /path/to/file.log.pos
  tag CUSTOMER.PRODUCT.SPRING-BOOT.DES
  <parse>
    @type none
  </parse>
</source>

and here are my record_modifier filters:

<filter *.**>
  @type record_modifier
  remove_keys ["worker", "facility", "level", "protocol"]
  char_encoding binary:utf-8
</filter>

<filter CUSTOMER.*>
  @type record_modifier
  <record>
    new_field customer
  </record>
</filter>

The char_encoding is working just fine, but there is no field added.

Here are some lines of my /var/log/td-agent/td-agent.log file

2018-04-05 09:47:48.178710183 -0300 CUSTOMER.PRODUCT.SPRING-BOOT.DES: {"message":"INFORMAÇÕES: Quantidade de solicitações com GR de vistoria paga e sem vistoria inicial: 0"}
2018-04-05 09:47:48.179287786 -0300 CUSTOMER.PRODUCT.SPRING-BOOT.DES: {"message":"IINFORMAÇÕES: >>>>>>> RESULTADO: "}
2018-04-05 09:47:48.179292605 -0300 CUSTOMER.PRODUCT.SPRING-BOOT.DES: {"message":"INFORMAÇÕES: Geração Automática de Vistorias finalizado com sucesso."}

How can we decode utf-8 to text

In the message i have \u0006 which is a tab. I can see char_encoding but unable to find something similar to decoding.

License missing from gemspec

RubyGems.org doesn't report a license for your gem. This is because it is not specified in the gemspec of your last release.

via e.g.

  spec.license = 'MIT'
  # or
  spec.licenses = ['MIT', 'GPL-2']

Including a license in your gemspec is an easy way for rubygems.org and other tools to check how your gem is licensed. As you can imagine, scanning your repository for a LICENSE file or parsing the README, and then attempting to identify the license or licenses is much more difficult and more error prone. So, even for projects that already specify a license, including a license in your gemspec is a good practice. See, for example, how rubygems.org uses the gemspec to display the rails gem license.

There is even a License Finder gem to help companies/individuals ensure all gems they use meet their licensing needs. This tool depends on license information being available in the gemspec. This is an important enough issue that even Bundler now generates gems with a default 'MIT' license.

I hope you'll consider specifying a license in your gemspec. If not, please just close the issue with a nice message. In either case, I'll follow up. Thanks for your time!

Appendix:

If you need help choosing a license (sorry, I haven't checked your readme or looked for a license file), GitHub has created a license picker tool. Code without a license specified defaults to 'All rights reserved'-- denying others all rights to use of the code.
Here's a list of the license names I've found and their frequencies

p.s. In case you're wondering how I found you and why I made this issue, it's because I'm collecting stats on gems (I was originally looking for download data) and decided to collect license metadata,too, and make issues for gemspecs not specifying a license as a public service :). See the previous link or my blog post about this project for more information.

How can I add a field current_time

Is there some thinks like current_time $(time) ?

Handling complex transformation

Hi wondering if the plugin can handle a transformation like

{
  "source": "192.168.29.133",
  "message": "SESSION,150,1,READ,SELECT,TABLE,public.ce_queue,\"select cq.uuid from ce_queue cq where \n\t cq.status='PENDING' limit $1 offset $2\",<not logged>"
}

to convert this to

{
  "source": "192.168.29.133",
  "operation": "READ",
  "name": "public.ce_queue",
  "query": "select cq.uuid from ce_queue cq where cq.status='PENDING' limit $1 offset $2",
  "parameters": "<not logged>"
}

Inaccuracies in README.md

For install fluent-plugin-record-modifier needed ruby-dev for native extantions
Error:

sudo gem install fluent-plugin-record-modifierFetching: msgpack-1.1.0.gem (100%)
Building native extensions.  This could take a while...
ERROR:  Error installing fluent-plugin-record-modifier:
	ERROR: Failed to build gem native extension.

    current directory: /var/lib/gems/2.4.0/gems/msgpack-1.1.0/ext/msgpack
/usr/bin/ruby2.4 -r ./siteconf20170930-10713-yrm82a.rb extconf.rb
mkmf.rb can't find header files for ruby at /usr/lib/ruby/include/ruby.h

extconf failed, exit code 1

Gem files will remain installed in /var/lib/gems/2.4.0/gems/msgpack-1.1.0 for inspection.
Results logged to /var/lib/gems/2.4.0/extensions/x86_64-linux/2.4.0/msgpack-1.1.0/gem_make.out

Add information about installing ruby

$ sudo apt-add-repository ppa:brightbox/ruby-ng
$ sudo apt-get update
Install the packages you need:
$ sudo apt-get install ruby2.4 ruby2.4-dev

Is if record.dig exists else null exclude even possible?

Loving the modifier, I'm trying to work on filtering records which are null adding an if statement, I’ve seen a few examples, but none match my use case. Has anyone come across this or can confirm its even possible?

Human readable action

partOf = 
if record.dig("kubernetes", "labels", "app.kubernetes.io/part-of").Value exists include record and value
else
remove_keys partOf

example dig

    <filter **>
      @type record_modifier
      <record>
        partOf ${record.dig("kubernetes", "labels", "app.kubernetes.io/part-of")}
      </record>
    </filter>

replace is adding two copies of string

fluentd:1.11.2
fluent-plugin-record-modifier:2.1.0

Config:

<filter foo-api.**>
  @type record_transformer
  <record>
    ecs_container_id "foo-api-#{Socket.gethostname}"
  </record>
</filter>

<filter foo-api.**>
  @type record_modifier
  <replace>
    key ecs_container_id
    expression /.*/
    replace foo-api-testing
  </replace>
</filter>

So I create a new field called "ecs_container_id" and set it to "foo-api-#{Socket.gethostname}" and this works.

As a test I then try to use the replace functionality (just to prove it works) to change that to "foo-api-testing". When doing this it writes two copies of the string in the new field so the value is:

foo-api-testingfoo-api-testing

I cannot use nested key in "replace"

I want to replace values in nested json
On my test, I changed sample data like this
{"data":{"key1":"hoge", "key2":"hoge", "key3":"bar"}}

and configuration, too.

<filter pattern>
  @type record_modifier
  <replace>
    # your key name
    **key record['data']['key1']  ## $.record.data.key1 and ${record["data"]["key1"] are all failed.** 
    # your regexp
    expression /^(?<start>.+).{2}(?<end>.+)$/
    # replace string
    replace \\k<start>ors\\k<end>
  </replace>
</filter>

Can anyone help me ?

Small numbers gets always converted to scientific notation

We are using the plugin to remove some unnecessary keys form a big json record, the original record looks like this

{
	"id": "0302d12661a5c1462f5d8ef3",
	"version": 1,
	"date": "2017-03-02 13:59:17",
	"share": 0,
	"earn": 1,
	"spent": 0.00001,
	"ip": "172.18.0.1",
	"referer": null,
	"userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/56.0.2924.87 Safari/537.36",
	"countryName": null,
	"region": null,
	"city": null,
	"latitude": null,
	"longitude": null,
	"isp": null,
	"connectionType": "wifi",
	"device": "Other",
	"deviceType": "desktop",
	"browser": "Chrome",
	"browserVersion": "56.0.2924",
	"os": "Mac OS X",
	"osVersion": "10.12.3",
	"param1": null,
	"param2": null,
	"param3": null,
	"param4": null,
	"param5": null,
	"param6": null,
	"level": "info",
	"message": ""
}

Applying this config

@type record_modifier
whitelist_keys earn,spent,date,id

It gets trasnformed to

{
	"id": "0302d12661a5c1462f5d8ef3",
	"date": "2017-03-02 13:59:17",
	"earn": 1,
	"spent": 1.0e-05
}

Spent went from 0,0001 to 1.0e-05.

It would be nice to have an option to enable/disabled this behavior.

Add a time object in the fluentd event

Hi,

I'd like to add time filed in the event, the following example appears to add a string type "formatted_time", but I need a time type field.

<filter pattern>
  @type record_modifier
  <record>
    formatted_time ${Time.at(time).to_s}
  </record>
</filter>

I try the following way

<filter pattern>
  @type record_modifier
  <record>
    formatted_time ${time}
  </record>
</filter>

But I got the following errors:

2021-11-10 06:24:28.085795373 +0000 fluent.warn: {"error":"#<NoMethodError: undefined method `to_msgpack' for 2021-11-10 06:24:28 +0000:Time>","location":"/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/event.rb:60:in `write'","tag":"iam.gu.cn.admin_api","message":"emit transaction failed: error_class=NoMethodError error=\"undefined method `to_msgpack' for 2021-11-10 06:24:28 +0000:Time\" location=\"/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.10.2/lib/fluent/event.rb:60:in `write'\" tag=\"iam.gu.cn.admin_api\""}

Please advise is it feasible to add a time type filed in the fluentd event? Thank you in adance.

How to add a nested field?

Given a log recod {"a": 1, "b":3 } I would like to transform it to

{"a": 1, "b":3, 'c': { 'd':4, 'e': 5} }

which is to add a field c with the nested value of { 'd':4, 'e': 5}?

rewrite tag

I'm trying to rewrite the tag, but it fail, the original tag remains.

So this is a example workflow.

# and add tag with_hostname and source syslog
<filter syslog.**>
  @type record_modifier
  <record>
    source syslog
    original_tag ${tag}
    # syslog source already has hostname
    tag with_hostname
  </record>
</filter>

# add a hostname to all messages that already have a with_source.
<filter with_source>
  @type record_modifier
  <record>
    host "#{Socket.gethostname}"
    tag with_hostname
  </record>
</filter>

<match with_hostname>

so original_tag is created with the "old" tag value. But tag doesn't get value with_hostname.

Why doesn't this rewrite the tag?

Removing Nested Keys

Hi, first off thank you for this plugin. It is great!

Is there any way to remove nested keys from a json string? Currently removing top level keys works great but I can find no way to remove nested keys.

Thanks

how to convert type of value in the new field?

hi
I use this plugin to create a new key (or a new field),the value is "123",how can I get a integer 123?as I output it to EFK stack for statistics

Docker caching using instance variable

Hi @repeatedly,

Thanks for the plugin!

I have a Syslog input and I'd like to add extra fields from Docker API:

In the current form, it won't be optimized as the call Docker calls will be make on each line.

Do you have idea on how to cache these call for at least some period of time?

Here's the current form:

<filter syslog.docker.containers.**>
  @type record_modifier
  prepare_value require 'docker'
  <record>
    docker_version ${Docker.version['Version'] rescue nil}
  </record>
  <record>
    docker_image ${sha_image = Docker::Container.get(record['ident']).info['Image']; Docker::Image.get(sha_image).info['RepoTags'].first.split(':').first}
  </record>
  <record>
    docker_tag ${sha_image = Docker::Container.get(record['ident']).info['Image']; Docker::Image.get(sha_image).info['RepoTags'].first.split(':').last}
  </record>
  <record>
    docker_restart_count ${Docker::Container.get(record['ident']).info['RestartCount']}
  </record>
  <record>
    docker_started_at ${Time.parse(Docker::Container.get(record['ident']).info['State']['StartedAt']).to_s}
  </record>
  <record>
    docker_finished_at ${Time.parse(Docker::Container.get(record['ident']).info['State']['FinishedAt']).to_s}
  </record>
  <record>
    docker_container ${Docker::Container.get(record['ident']).info['Name'][1..-1]}
  </record>
  <record>
    nomad_alloc_name ${Docker::Container.get(record['ident']).info['Config']['Env'].select { |e| e.include?('NOMAD_ALLOC_NAME') }.first.split('=').last rescue nil}
  </record>
  remove_keys extradata,msgid,container_id,pid
</filter>

Any help would be much appreciated 🙇

Lots of type errors in logs

My logs are getting flooded with these warnings

2022-01-11 03:52:34 +0000 [warn]: #0 dump an error event: error_class=TypeError error="no implicit conversion of String into Integer" location="/usr/lib/ruby/gems/2.7.0/gems/fluent-plugin-record-modifier-2.1.0/lib/fluent/plugin/filter_record_modifier.rb:174:in `[]'" tag="ssm.worker" time=2022-01-11 03:51:47.035123542 +0000 record=0

I'm not sure what the issue is. Here my relevant configs

<filter **>
  @type record_modifier
  @id rename_log_key
  <record>
    log_message ${record["log"]}
  </record>
  remove_keys log
</filter>
#######################

<filter **>
  @type record_modifier
  @id add_tag_to_field
  <record>
    fluentd_tag ${tag}
  </record>
</filter>

 fluent-gem list

*** LOCAL GEMS ***

async (1.30.1)
async-http (0.54.0)
async-io (1.32.2)
async-pool (0.3.9)
aws-eventstream (1.2.0)
aws-partitions (1.547.0)
aws-sdk-core (3.125.2)
aws-sigv4 (1.4.0)
benchmark (default: 0.1.0)
bigdecimal (1.4.4)
bundler (2.2.31)
cgi (default: 0.1.0.1)
concurrent-ruby (1.1.9)
console (1.13.1)
cool.io (1.7.1)
csv (default: 3.1.2)
date (default: 3.0.3)
dbm (default: 1.1.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
elasticsearch (7.13.3)
elasticsearch-api (7.13.3)
elasticsearch-transport (7.13.3)
etc (default: 1.1.0)
excon (0.89.0)
ext_monitor (0.1.2)
faraday (1.8.0)
faraday-em_http (1.0.0)
faraday-em_synchrony (1.0.0)
faraday-excon (1.1.0)
faraday-httpclient (1.0.1)
faraday-net_http (1.0.1)
faraday-net_http_persistent (1.2.0)
faraday-patron (1.0.0)
faraday-rack (1.0.0)
faraday_middleware-aws-sigv4 (0.3.0)
fcntl (default: 1.0.0)
fiber-local (1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-plugin-aws-elasticsearch-service (2.4.1)
fluent-plugin-concat (2.5.0)
fluent-plugin-elasticsearch (5.1.4)
fluent-plugin-grok-parser (2.6.2)
fluent-plugin-multi-format-parser (1.0.0)
fluent-plugin-prometheus (2.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.4.0)
fluentd (1.14.0)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
http_parser.rb (0.7.0)
ipaddr (default: 1.2.2)
irb (default: 1.2.6)
jmespath (1.5.0)
json (2.4.1)
logger (default: 1.4.2)
matrix (default: 0.2.0)
msgpack (1.4.2)
multi_json (1.15.0)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
nio4r (2.5.8)
observer (default: 0.1.0)
oj (3.10.18)
open3 (default: 0.1.0)
openssl (default: 2.1.3)
ostruct (default: 0.2.0)
prime (default: 0.1.1)
prometheus-client (2.1.0)
protocol-hpack (1.4.2)
protocol-http (0.21.0)
protocol-http1 (0.13.2)
protocol-http2 (0.14.2)
pstore (default: 0.1.0)
psych (default: 3.1.0)
racc (default: 1.4.16)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.5)
resolv (0.2.1)
rexml (default: 3.2.3.1)
rss (default: 0.2.8)
ruby2_keywords (0.0.5)
serverengine (2.2.4)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.5)
strscan (default: 1.0.3)
timeout (default: 0.1.0)
timers (4.3.3)
tracer (default: 0.1.0)
tzinfo (2.0.4)
tzinfo-data (1.2021.5)
uri (default: 0.10.0)
webrick (default: 1.6.1)
yajl-ruby (1.4.1)
yaml (default: 0.1.0)
zlib (default: 1.1.0)

Using same key for referencing record and `remove_keys` causes null

I've found a bit strange behavior about remove_keys.
This issue is occurred when key of record in <record> is also specified to remove_keys.

How to reproduce

<source>
  @type sample
  sample {"log": "[foo]"}
  rate 1
  tag sample
</source>

<filter sample>
  @type record_modifier
  <record>
    message ${record["log"]}
  </record>
  remove_keys log
</filter>

<match **> 
 @type stdout
</match>

Actual result

It shows:

2021-08-10 16:35:55.076556799 +0900 sample: {"foo":"bar","message":"[foo]"}
2021-08-10 16:35:56.077552818 +0900 sample: {"foo":"bar","message":null}
2021-08-10 16:35:57.078426650 +0900 sample: {"foo":"bar","message":null}

Expected result

I'm expected that message is not null.

2021-08-10 16:35:55.076556799 +0900 sample: {"foo":"bar","message":"[foo]"}
2021-08-10 16:35:56.077552818 +0900 sample: {"foo":"bar","message":"[foo]"}
2021-08-10 16:35:57.078426650 +0900 sample: {"foo":"bar","message":"[foo]"}

Additional Information

Fluentd 1.13.3
fluent-plugin-record-modifier (2.1.0)

fluent/fluent-plugin-rewrite-tag-filter#112

<match **> causes restart

i'm not sure record_modifier is to be blamed, but here is the log.

https://gist.github.com/trombik/1660e21f4868140626a9

if you replace ** with .foo or foo., events are processed without problem. tested on ruby 2.0.0 and 2.1.2, the same issue.

rubygem-fluent-mixin-config-placeholders-0.2.4
rubygem-fluent-plugin-record-modifier-0.1.3
rubygem-fluent-plugin-redis-0.2.0
rubygem-fluentd-0.10.48_1
rubygem-http_parser.rb-0.6.0
ruby-2.0.0.353_6,1
rubygem-yajl-ruby-1.2.0
rubygem-sigdump-0.2.2
rubygem-msgpack-0.5.8
rubygem-json-1.8.1
rubygem-cool.io-1.2.4
ruby20-gems-1.8.29

any idea?

incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)

Problem

I'm getting below error while shipping logs to ES via td-agent 1.11.1:

2020-11-01 17:11:42 +0530 [error]: #0 incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
  2020-11-01 17:11:42 +0530 [error]: #0 suppressed same stacktrace
2020-11-01 17:11:42 +0530 [error]: #0 incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/parser_regexp.rb:50:in `match'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/parser_regexp.rb:50:in `parse'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-grok-parser-2.6.1/lib/fluent/plugin/parser_multiline_grok.rb:21:in `block in parse'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-grok-parser-2.6.1/lib/fluent/plugin/parser_multiline_grok.rb:20:in `each'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluent-plugin-grok-parser-2.6.1/lib/fluent/plugin/parser_multiline_grok.rb:20:in `parse'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:546:in `block in parse_multilines'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:544:in `each'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:544:in `parse_multilines'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:469:in `call'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:469:in `receive_lines'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:845:in `block in handle_notify'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:877:in `with_io'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:825:in `handle_notify'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:808:in `block in on_notify'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:808:in `synchronize'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:808:in `on_notify'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:653:in `on_notify'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:325:in `block in setup_watcher'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin/in_tail.rb:596:in `on_timer'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run_once'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/cool.io-1.6.0/lib/cool.io/loop.rb:88:in `run'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin_helper/event_loop.rb:93:in `block in start'
  2020-11-01 17:11:42 +0530 [error]: #0 /opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-11-01 17:11:43 +0530 [error]: #0 incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
  2020-11-01 17:11:43 +0530 [error]: #0 suppressed same stacktrace

I've added the parameter suggested here 👍 https://github.com/repeatedly/fluent-plugin-record-modifier#char_encoding as it was recommended here https://docs.fluentd.org/quickstart/faq but the issue persists.

...

Steps to replicate

Provide example config and message

# encoding: utf-8
<source>
  @type tail
  path /var/log/messages
  pos_file /etc/td-agent/new_var_log_msg_grok.log.pos
  #time_format %Y-%m-%dT%H:%M:%S.%NZ
  time_format %b %dT%H:%M:%SZ
  tag var.msg
  <parse>
    @type multiline_grok
    <grok>
     pattern %{SYSLOGTIMESTAMP:time}%{SPACE}%{HOSTNAME:hostname}%{SPACE}%{GREEDYDATA:service_name}:%{GREEDYDATA:log_message}
    </grok>
  </parse>
</source>

<filter var.msg>
    @type record_modifier
     <record>
     hostname "#{Socket.gethostname}"
     formatted_time ${Time.at(time).iso8601(3)}
     char_encoding utf-8
     char_encoding utf-8:euc-jp
     </record>
</filter>

<match var.msg>
  @type elasticsearch
#  type_name "_doc"
  hosts redacted:9200
  scheme "https"
  ssl_version TLSv1_2
  ssl_verify false
  ca_file "/etc/td-agent/cert.crt"
  user redacted
  password redacted
  reload_connections false
  reconnect_on_error true
  reload_on_failure true
  log_es_400_reason false
  logstash_prefix messages_logs
  logstash_format true
  logstash_dateformat %V
  index_name "messages_logs"
  type_name "fluentd"
  include_timestamp true
  <buffer>
    @type file
    path /etc/td-agent/messages/buffers
    chunk_limit_size 1M
    flush_interval 5s
    retry_forever false
    retry_max_times 3
    retry_wait 10
    retry_max_interval 300
    flush_thread_count 8
  </buffer>
</match>

Expected Behavior or What you need to ask

The same config is working fine for most servers even without char_encoding parameter. Td-agent of same version should have same behaviour across servers with same configuration. The error should go after adding the encoding parameter.
...

Using Fluentd and ES plugin versions

OS version
Red Hat Enterprise Linux Server release 7.9 (Maipo)
Fluentd v0.12 or v0.14/v1.0

td-agent --version

td-agent 1.11.1
ES plugin 3.x.y/2.x.y or 1.x.y
- paste result of fluent-gem list, td-agent-gem list or your Gemfile.lock

  td-agent-gem list

*** LOCAL GEMS ***

addressable (2.7.0)
async (1.26.2)
async-http (0.52.4)
async-io (1.30.0)
async-pool (0.3.2)
aws-eventstream (1.1.0)
aws-partitions (1.337.0)
aws-sdk-core (3.102.1)
aws-sdk-kms (1.35.0)
aws-sdk-s3 (1.72.0)
aws-sdk-sqs (1.29.0)
aws-sigv4 (1.2.1)
benchmark (default: 0.1.0)
bigdecimal (default: 2.0.0)
bundler (2.1.4)
cgi (default: 0.1.0)
concurrent-ruby (1.1.6)
console (1.8.2)
cool.io (1.6.0)
csv (default: 3.1.2)
date (default: 3.0.0)
delegate (default: 0.1.0)
did_you_mean (default: 1.4.0)
digest-crc (0.6.1)
elasticsearch (7.8.0)
elasticsearch-api (7.8.0)
elasticsearch-transport (7.8.0)
elasticsearch-xpack (7.9.0)
etc (default: 1.1.0)
excon (0.75.0)
faraday (1.0.1)
fcntl (default: 1.0.0)
ffi (1.13.1)
fiddle (default: 1.0.0)
fileutils (default: 1.4.1)
fluent-config-regexp-type (1.0.0)
fluent-logger (0.8.2)
fluent-plugin-concat (2.4.0)
fluent-plugin-elasticsearch (4.1.1, 4.0.9)
fluent-plugin-grok-parser (2.6.1)
fluent-plugin-kafka (0.13.0)
fluent-plugin-prometheus (1.8.0)
fluent-plugin-prometheus_pushgateway (0.0.2)
fluent-plugin-record-modifier (2.1.0)
fluent-plugin-rewrite-tag-filter (2.3.0)
fluent-plugin-s3 (1.3.3)
fluent-plugin-systemd (1.0.2)
fluent-plugin-td (1.1.0)
fluent-plugin-td-monitoring (1.0.0)
fluent-plugin-webhdfs (1.2.5)
fluentd (1.11.1)
forwardable (default: 1.3.1)
getoptlong (default: 0.1.0)
hirb (0.7.3)
http_parser.rb (0.6.0)
httpclient (2.8.2.4)
io-console (default: 0.5.6)
ipaddr (default: 1.2.2)
ipaddress (0.8.3)
irb (default: 1.2.3)
jmespath (1.4.0)
json (default: 2.3.0)
logger (default: 1.4.2)
ltsv (0.1.2)
matrix (default: 0.2.0)
mini_portile2 (2.5.0)
minitest (5.13.0)
mixlib-cli (1.7.0)
mixlib-config (2.2.3)
mixlib-log (1.7.1)
mixlib-shellout (2.2.7)
msgpack (1.3.3)
multi_json (1.14.1)
multipart-post (2.1.1)
mutex_m (default: 0.1.0)
net-pop (default: 0.1.0)
net-smtp (default: 0.1.0)
net-telnet (0.2.0)
nio4r (2.5.2)
nokogiri (1.11.0.rc2)
observer (default: 0.1.0)
ohai (6.20.0)
oj (3.10.6)
open3 (default: 0.1.0)
openssl (default: 2.1.2)
ostruct (default: 0.2.0)
parallel (1.19.2)
power_assert (1.1.7)
prime (default: 0.1.1)
prometheus-client (0.9.0)
protocol-hpack (1.4.2)
protocol-http (0.20.0)
protocol-http1 (0.13.0)
protocol-http2 (0.14.0)
pstore (default: 0.1.0)
psych (default: 3.1.0)
public_suffix (4.0.5)
quantile (0.2.1)
racc (default: 1.4.16)
rake (13.0.1)
rdkafka (0.8.0)
rdoc (default: 6.2.1)
readline (default: 0.0.2)
readline-ext (default: 0.1.0)
reline (default: 0.1.3)
rexml (default: 3.2.3)
rss (default: 0.2.8)
ruby-kafka (1.1.0)
ruby-progressbar (1.10.1)
rubyzip (1.3.0)
sdbm (default: 1.0.0)
serverengine (2.2.1)
sigdump (0.2.4)
singleton (default: 0.1.0)
stringio (default: 0.1.0)
strptime (0.2.4)
strscan (default: 1.0.3)
systemd-journal (1.3.3)
systemu (2.5.2)
td (0.16.9)
td-client (1.0.7)
td-logger (0.3.27)
test-unit (3.3.4)
timeout (default: 0.1.0)
timers (4.3.0)
tracer (default: 0.1.0)
tzinfo (2.0.2)
tzinfo-data (1.2020.1)
uri (default: 0.10.0)
webhdfs (0.9.0)
webrick (default: 1.6.0)
xmlrpc (0.3.0)
yajl-ruby (1.4.1)
yaml (default: 0.1.0)
zip-zip (0.3)
zlib (default: 1.1.0)

ES version (optional)
7.5.1

Whitelist nested json

Hi!

Is there any way to whitelist nested keys from a json string? Currently whitelisting top level keys works great but I can not find a way to whitelist nested keys.

Thanks in advance

Case insensitive replace

Hi,
Is it possible to perform a case insensitive replace via and expression statement?

how to use .match(/pattern/) when using record_modifier

I want to add an api field to record_modifier, but it doesn't seem to work when I search with Kibana, it reports：

This field is present in your Elasticsearch mapping but not in the 500 documents shown in the doc table. You may still be able to visualize or search on it.

config:

    <filter {{ .Values.sourceTag }}>
      @type record_modifier
      <record>
        api ${record['log'].match(/\/api\/.*? /)}
        level
      </record>
    </filter>

record['log'] examples（one per line）:

2022/09/26 16:34:30 DBG abc/xyz.go:46 > Register Route - DELETE, /api/role/v2
2022/09/26 16:34:30 DBG cmdbase/parser.go:46 > Register Route - GET, /api/role/v2/sys_feat
2022/09/26 16:34:30 Serving gRPC-Gateway on http://0.0.0.0:8070
2022/09/26 16:34:30 INF gwserver/gateway_server.go:152 > Start rest server in 0.0.0.0:7847
2022/09/26 16:35:36 DBG middleware/api_entry.go:97 > <- GET /api/ping to_xyz=infra-abc 4d65822107fcfd52
2022/09/26 16:35:36 DBG middleware/perm.go:93 > perm check, userID: 0, partyID:  4d65822107fcfd52
2022/09/26 16:35:36 /app/paas_go/cofu/models/party/party.go:311 record not found
2022/09/26 16:35:36 WRN monitor/instrument.go:75 > record metrics failed, Err: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: address $[MONITOR_GRPCSERVICEURL]: missing port in address" 4d65822107fcfd52
2022/09/26 16:35:53 DBG middleware/api_entry.go:97 > <- GET /api/ping to_xyz=infra-yao 78629a0f5f3f164f
2022/09/26 16:35:53 DBG middleware/perm.go:93 > perm check, userID: 0, partyID:

When I change the configuration to the following, it can get data what doesn't match what I need exactly

    <filter {{ .Values.sourceTag }}>
      @type record_modifier
      <record>
        api ${record['log'].split(' ')[7])}
        level
      </record>
    </filter>

how to edit the config?

version 0.6.1 can't dup FixNums

There is an issue with gem version 0.6.1, FixNum types cannot be dup'd. I believe these are probably coming from either monitor_agent or flowcounter, possibly both.

  2017-12-18 07:51:13 -0700 [error]: suppressed same stacktrace
2017-12-18 07:51:32 -0700 [warn]: emit transaction failed: error_class=TypeError error="can't dup Fixnum" tag="denver.non
prod.app.uel.forwarder.fluentd.flowcount"
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:121:in `dup'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:121:in `block in set_encoding'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:119:in `each_pair'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:119:in `set_encoding'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:107:in `block in filter_stream'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:1
49:in `feed_each'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event.rb:1
49:in `each'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluent-plugin-record-modifier-0.6.1/l
ib/fluent/plugin/filter_record_modifier.rb:90:in `filter_stream'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_rout
er.rb:152:in `block in emit'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_rout
er.rb:151:in `each'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_rout
er.rb:151:in `emit'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/event_router.rb:90:in `emit_stream'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:178:in `on_message'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:338:in `call'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:338:in `block in on_read_msgpack'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:337:in `feed_each'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:337:in `on_read_msgpack'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.6/lib/cool.io/io.rb:123:in `on_readable'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.6/lib/cool.io/io.rb:186:in `on_readable'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run_once'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/cool.io-1.4.6/lib/cool.io/loop.rb:88:in `run'
  2017-12-18 07:51:32 -0700 [warn]: /opt/td-agent/embedded/lib/ruby/gems/2.1.0/gems/fluentd-0.12.40/lib/fluent/plugin/in_forward.rb:120:in `run'

Fluentd can't forward logs bacause of:

#0 dump an error event: error_class=NoMethodError error="undefined method has_key?' for nil:NilClass" location="/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluent-plugin-record-modifier-2.0.1/lib/fluent/plugin/filter_record_modifier.rb:173:in expand'"

Question: replace whole record with single field from it

I want to use this plugin to catch failed records from Elasticsearch output (e.g. "rejected by Elasticsearch") to put them into "dead-letter" output. Those failed records are emitted as fluent.warn records, with original message stored in "record" key of whole log message. What I want is to take this "record" key and move it down to fluentd pipeline as whole message itself with new tag. I did not find any documentation describing this neither for record_transformer plugin neither this one.
Example of that is wanted:
Original record:

2020-05-21 10:34:35.497925679 +0000 fluent.warn: 
{
    "error": "#<Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError: 400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'object mapping for [sample] tried to parse field [sample] as object, but found a concrete value'>",
    "location": null,
    "tag": "test.log",
    "time": 1589206011,
    "record": {
        "foo": "bar",
        "key": "value",
        "sample": [],
        "blah-blah": 133163771
    },
    "message": "dump an error event: error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch [error type]: mapper_parsing_exception [reason]: 'object mapping for [sample] tried to parse field [sample] as object, but found a concrete value'" location=nil tag="test.log" time=1589206011 record={\"foo\"=>\"bar\", \"key\"=>\"value\", \"sample\"=>[], \"blah-blah\"=>133163771}"
}

Modified record:

2020-05-21 10:34:36.497925679 +0000 dead.log: 
{
    "foo": "bar",
    "key": "value",
    "sample": [],
    "blah-blah": 133163771
}

error when using whitelist

    <source>
      @type tail
      <parse>
        @type json
      </parse>
      path /var/log/containers/kube-dns*
      path_key path
      pos_file /var/log/es-kubernetes.pos
      tag k8s
    </source>

    <filter k8s>
      @type parser
      key_name path
      reserve_data true
      remove_key_name_field true
      <parse>
        @type regexp
        expression /^.*kube-(?<service_type>.+)-(master|slave).*/
      </parse>
    </filter>

    <filter k8s>
      @type record_modifier
      <record>
        Payload ${record['log']}
      </record>
      whitelist_keys Payload,service_type
    </filter>

and the error is :
ArgumentError error="path does not exist
There are no errors when I delete the whitelist

Replace Nest Key

Hey @repeatedly , I want to redact my value, the key is nested.
However, just some record has both 2 level of nested keys.
{
"properties": {
"messageTracingId": 12345,
"connectionId": "ABCDE",
"traceId": "akhsb:01",
"hub": "chatsamplehub",
"userId": "eyJAdlhopWkXY.eyJuKXE4gp4QP.hlGrujNIvT"
},
"resourceId": "balabal",
"time": "2023-07-10T10:45:29Z"
}

I want to replace value of the [properties][userId], if it's pattern match. But not every record has properties and not each properties has userId.
How should I check before I do the following?

filter mdsdlog.asrs.customer>
@type record_modifier

expression /(?-i)eyJ(?i)[a-z0-9\-_%]+\.(?-i)eyJ(?i)[a-z0-9\-_%]+\.[a-z0-9\-_%]+/ key **$.properties.userId** replace "[REDACTED]"

Unable to create nested key

I am attempting to parse a Tomcat access log. I am currently using a regex parser to do so. The parser works well and all the expected fields are created.

From there, I'd like to be able to take some of the the records created by the regex parser and structure them in key-dictionary pair,. For example, I'd like

record {
  httpVersion:  "HTTP/1.1"   
  remoteIp:  "10.0.23.251"   
  requestMethod:  "GET"   
  responseSize:  "2"   
  status:  "200"   
  textPayload:  "/app/api/1.1/healthcheck/status"   
  timestamp:  "30/Nov/2018:19:01:01 -0600"   
 }

to become

record {
  httpRequest: {
    remoteIp:  "10.0.23.251"
    httpVersion:  "HTTP/1.1"
    requestMethod:  "GET"
    responseSize:  "2"
    status:  "200"
  }
  textPayload:  "/forms/api/1.1/healthcheck/status"
  timestamp:  "30/Nov/2018:19:01:01 -0600"
}

I have tried using

<filter **.localhost-access-log>
  @type record_transformer
  <record>
    ${record["httpRequest"]["remoteIp"]} ${record["remoteIp"]}
  </record>
</filter>

<filter **.localhost-access-log>
  @type record_transformer
  <record>
    httpRequest.remoteIp ${record["remoteIp"]}
  </record>
</filter>

and even

<filter **.localhost-access-log>
  @type record_transformer
  <record>
    $.httpRequest.remoteIp ${record["remoteIp"]}
  </record>
</filter>

So far nothing has worked.

Does record transformation even support doing this?

I would try to take another approach to the problem but I have no control over the server that I'm importing this the logs into and changing the log format in Tomcat isn't very practical as it would require changes to many different applications owned by different teams.

Support multiline ruby code

Is it possible to support multiline code?

  <filter **>      
    @type record_modifier      
    <record>      
      somekey  ${if !record['field1'].nil?;       
                           record['field1'];       
                 elsif record['field2'] != "";       
                    record['field2'];       
                 else       
                    record['somekey'];       
                 end;}      
    </record>      
  </filter>

Currently i get error in fluentd (1.7.2)

2021-05-21 11:59:16 +0000 [error]: /usr/local/share/gems/gems/fluent-plugin-record-modifier-2.1.0/lib/fluent/plugin/filter_record_modifier.rb:175: unterminated string meets end of file  
            end  
               ^ 
               ^ error_class=SyntaxError error="/usr/local/share/gems/gems/fluent-plugin-record-modifier-2.1.0/lib/fluent/plugin/filter_record_modifier.rb:175: unterminated string meets end of file\n            end\n               ^\n/usr/local/share/gems/gems/fluent-plugin-record-modifier-2.1.0/lib/fluent/plugin/filter_record_modifier.rb:175: syntax error, unexpected end-of-input, expecting keyword_end\n            end\n               ^"

Single Backslag not able to print into field

#TargetUserName and TargetDomainName
<filter log.raw>
  @type record_modifier
  remove_keys _dummy_
  <record>
    _dummy_ ${if record['TargetUserName'] and record['TargetDomainName']; record['TargetDomainUserName'] = "${record['TargetUserName']}\${record['TargetDomainName']}"; end; nil}
  </record>
</filter>

Shows --> "TargetDomainUserName":"Hello#{record['TargetDomainName']}"

"${record['TargetUserName']}\\${record['TargetDomainName']}"

Shows -> "TargetDomainUserName":"Hello\\World"

So there is no way to put a single backslash on the record_modifier

support nested keys via the helper

A revival of #15 .
The record_accessor helper plugin is available for some time, perhaps this can be implemented
cleanly now?

Thanks!

#<filter **>
#  @type record_modifier
#  <record>
#    log ${record['log'].tr("\n", "\t")}
#  </record>
#  remove_keys _hash, $.kubernetes, $.docker
#</filter>

<filter **>
  @type record_transformer
  auto_typecast false
  enable_ruby true
  <record>
    # use double quotes to avoid literal strings!
    log ${record['log'].tr("\n", "\t")}
  </record>
  remove_keys _hash, $.kubernetes, $.docker
</filter>

JSON formatted output

Hi there,

how to create json formatted output like this:

{
  "data": {
    "body": {
      "message": {
        "body": "Hello, world!"
      }
    }
  }
}

Thanks,
Andrey

record-modifier doesn't work

Fluent-Bit v0.11.15
ruby 2.1.10p492 (2016-04-01 revision 54464) [x86_64-linux]
CentOS Linux release 7.3.1611 (Core)

td-agent-bit.conf

[INPUT]                                                                                                                                                         
    Name cpu                                                                                                                                                    
    Tag  cpu.local                                                                                                                                                                                                                                                                                               
[OUTPUT]                                                                                                                                                        
    Name  stdout                                                                                                                                                
    Match *                                                                                                                                                                                                                                                                                                                                                                                                       
[FILTER]                                                                                                                                                        
    Name record_modifier                                                                                                                                        
    Match *                                                                                                                                                     
    record host aaaa

I find td-agent-bit can't work, at the same times, i have some questions for you,

fluent-bit current release is v0.11.15 (about 17 days ago); and why fluentd >= v0.14.0 ?

empty

replace $record with key

I want to
remove_keys log

and use ${record} = ${record['log']}

is this possible?

BUG: use router.emit_stream instead of Engine.emit_stream with Fluentd 0.14.4

It appears that fluent 0.14.4 wants plugins to update, specifically in fluent-plugin-record-modifier-0.5.0/lib/fluent/plugin/out_record_modifier.rb:79 to use router.emit_stream instead of Engine.emit_stream

The relevant line in fluentd is here
The relevant line in out_record_modifier.rb is here

I created a gist to reproduce the error here.

I am running latest Docker (17.03.1-ce-mac5 (16048)) on Mac OS X El Capitan 10.11.6 (15G1421).
I am pulling the latest fluentd (0.14.4) using the latest fluent docker image (fluent/fluentd:v0.14).
I am using fluent-plugin-flowcounter v0.4.2