Feature request: new configuration: 'timeout_is_error' about fluent-plugin-concat HOT 20 CLOSED

Is there really no way to simply close a multiline capture when timeout is reached and process like normal? A lot of situations (java logs & stacktrace) have no clear multiline end.

The other plugins from fluentd, like the memory buffer ones seems to be able to simply apply a flush interval to their processing, at which point all the collected stuff simply goes on to be processed normally.

from fluent-plugin-concat.

A workaround this would be to use the relabel plugin to label in both cases i.e. in case when concat gives timeout and in case when it does not and then, finally putting all your output plugins under that label. Here is an example:

<filter **>
@type concat
key "message"
multiline_start_regexp "/^Start/"
flush_interval 5
timeout_label "@NORMAL"
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
<label @NORMAL>
<match **>
@type stdout
</match>
</label>

This will solve the duplicacy issue.

from fluent-plugin-concat.

I will consider this issue in next week.
(I will add new option or ...)

from fluent-plugin-concat.

This does not solve my use case. I do not want to wait forever for the next line. In some cases, it could be hours or days before the next line occurs. If 60s go by with no new lines, that's probably it - I want to output what is in the buffer without generating an error.

from fluent-plugin-concat.

Use timeout_label configuration.

from fluent-plugin-concat.

That's what I do currently, but it adds a lot of pointless complexity to my config files and generates spurious timeout warning messages. All I want is to output my timeouts the same way other lines are output.

from fluent-plugin-concat.

+1

Some time ago I requested the same in #7

from fluent-plugin-concat.

Hmm...

• Use timeout_label
  • 👍 Reduce warnings
  • 👎 Complex duplicated configuration
• Not using timeout_label
  • 👎 warnings
  • 👎 Complex duplicated configuration

You want:

• No warnings when timeout occurs
• Need timeout and flush buffer and emit normal event
• No complex duplicated configuration

Right?

from fluent-plugin-concat.

Yes. When the timeout happens, just emit whatever's in the buffer, as if a startline had come in.

from fluent-plugin-concat.

It is impossible to implement such a thing.
It is the limitation of Fluentd.

from fluent-plugin-concat.

Er? You already emit the buffer content, directed to timeout_label. Is it really not possible to simply do the same, but direct it to the default label instead?

And it's definitely possible to not emit an error message - just make line 188 conditional.

from fluent-plugin-concat.

It is impossible.
If just call router.emit, it skips remained filters defined after filter_concat.

You can set log_level per plugin.

<filter>
  @type concat
  log_level warn
  # ... snip
</filter>

from fluent-plugin-concat.

I have a way to work around this issue, but still not perfect:
multiline_start_regexp /[\d-]+ [\d:.]+ *ERROR/

so, only error stack trace is printed, the multiline plugin is activated. of course, this config suppose you print stack trace only when error happened.

but this is not perfect as well, because when an error printed, if there is no more preceding logs, this error may not be flushed

from fluent-plugin-concat.

Something like this from fluentd 0.12.20+:

multiline_flush_interval

The interval of flushing the buffer for multiline format. The default is disabled.

If you set multiline_flush_interval 5s, in_tail flushes buffered event after 5 seconds from last emit. This option is useful when you use format_firstline option. Since v0.12.20 or later.

from fluent-plugin-concat.

Is there really no way to simply close a multiline capture when timeout is reached and process like normal?

There is no way.
This plugin is filter, so we cannot emit normal event. In fact, filter cannot emit normal event. It is difference from output plugin.

The other plugins from fluentd, like the memory buffer ones seems to be able to simply apply a flush interval to their processing, at which point all the collected stuff simply goes on to be processed normally.

Those plugins are output plugin. But this is filter plugin.
We don't use Fluentd's buffer mechanism in this plugin.

from fluent-plugin-concat.

Got it. An unfortunate fluentd limitation.

from fluent-plugin-concat.

I've got a solution for this issue, but it is not perfect and not implemented.

• Add new configuration soft_flush_interval, this counts timeout in main thread
• if Fluentd does not send events to this filter plugin:
  • soft_flush_interval: does not flush buffer
  • flush_interval: flush buffer and timeout error occur
• if Fluentd sends events but not the end of multline:
  • soft_flush_interval: flush buffer as normal events
  • flush_interval: flush buffer and timeout error occur (if soft_flush_interval > flush_interval: this means soft_flush_interval is disabled)
• This solution introduces complexity to this pluigin

Is this enough or not?

from fluent-plugin-concat.

@okkez this would not really help my case as when output happens, it always starts with a start-line followed by some arbitrary number of lines. Then there can be no output for prolonged period of time and new event that starts with a start-line.

from fluent-plugin-concat.

Thanks you for comments.
I will add this to README.md.

from fluent-plugin-concat.

@mdamir 's workaround works nicely! Thanks :)

from fluent-plugin-concat.