Optionally Disable `Timeout Error Event` about fluent-plugin-concat HOT 5 CLOSED

If dispatch the message when timeout occurred, we may get broken incomplete message.
It is very hard to process them properly, I think.

You can use timeout_label to handle timeout events same as normal events.

<filter docker.*>
  @type concat
  ...
  timeout_label @TIMEOUT
</filter>

<label @TIMEOUT>
  <match docker.*>
    @type stdout
  </match>
</label>

<match docker.*>
  @type stdout
</match>

I cannot detect that next chain is filter plugin or output plugin in timeout context.

from fluent-plugin-concat.

Hmmmm... true I'll try to use that

After concat I have this config

<match **>
    type copy
    <store>
        type elasticsearch
        host elasticsearch
        port 9200
        logstash_format true
    </store>
    <store>
        type grepcounter
        count_interval 30
        input_key log
        regexp ERROR
        threshold 1
        delimiter \n\n
        add_tag_prefix notification.error
    </store>
</match>

I tried to use multiline_end_regexp but most the times the end of a multline log is the begging of a new one... for exemple: setting "flush_interval 5" and "disable_timeout true" the user would have the assumption no other log came, so keep on (even knowing the risk of broken log)... This option should also disable the warning in fluentd log

I'm not sure (to be honest it's my first time using fluentd)... but using timeout_label the config above should be copied in a ? I mean... or is there anyway to reference instead of Ctrl C then V?

from fluent-plugin-concat.

You can omit multiline_end_regexp if you don't know that the end line of logs.
You must specify multiline_start_regexp only.

For example,

<fileter docker.*>
  @type concat
  key message
  multi_line_start_regxp /^Start/
  flush_interval 6000 # very very long flush interval
</filter>

This works as followings:

1. Start starts multiline logs
2. Concatenate any lines to buffer until next Start
3. Get next Start, flush buffer before this Start
4. Start new buffer and concatenate any lines to buffer until next Start

📝 If you cannot get next Start, this plugin cannot flush buffer forever (until flush_interval).

I'm not sure (to be honest it's my first time using fluentd)... but using timeout_label the config above should be copied in a ? I mean... or is there anyway to reference instead of Ctrl C then V?

Yes. You need to copy and paste, if you want same behavior for normal and timeout.
I know this is annoying to maintain fluentd configuration.

from fluent-plugin-concat.

Sure... I got the idea :D

Gonna close this issue

from fluent-plugin-concat.

@okkez The copy paste can be avoided by using the relabel plugin, as follows:

<filter **>
@type concat
key "message"
multiline_start_regexp "/^Start/"
flush_interval 5
timeout_label "@LASTLINE"
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
<label @LASTLINE>
<match **>
@type relabel
@label @NORMAL
</match>
</label>
<label @NORMAL>
<match **>
@type stdout
</match>
</label>

from fluent-plugin-concat.