Timeout flush [dump an error event] about fluent-plugin-concat HOT 3 CLOSED

In this configuration, fluent-plugin-concat flushes all buffer each 5sec (flush_interval) if events does not match multiline_start_regexp.

For details:

A event occurs for each second.

timeout

1. event {message: "MATCH multiline_start_regexp"
2. event {message: "NOT MATCH multiline_start_regexp"
3. event {message: "NOT MATCH multiline_start_regexp"
4. event {message: "NOT MATCH multiline_start_regexp"
5. event {message: "NOT MATCH multiline_start_regexp"
   • 5 seconds elapsed: flush fluent-plugin-concat internal buffer(5lines), emit error event and log timeout error

not timeout

1. event {message: "MATCH multiline_start_regexp"
2. event {message: "NOT MATCH multiline_start_regexp"
3. event {message: "NOT MATCH multiline_start_regexp"
4. event {message: "NOT MATCH multiline_start_regexp"
5. event {message: "MATCH multiline_start_regexp"
   • flush fluent-plugin-concat internal buffer (4lines) and reset interval

from fluent-plugin-concat.

I my case it seems that some similar log messages are actually concatenated correctly but some not. But these log messages are almost similar.

Example: wich is ok

Caused by: java.net.UnknownHostException: dummy-db
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.postgresql.core.PGStream.<init>(PGStream.java:61)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121)
    ... 97 common frames omitted

Example: on which i get an error

Caused by: java.net.UnknownHostException: dummy-db
    at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
    at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
    at java.net.Socket.connect(Socket.java:589)
    at org.postgresql.core.PGStream.<init>(PGStream.java:61)
    at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:121)
    ... 89 common frames omitted

These are offcourse part of a bigger log-stream but i am lost in why i get an error.
2016-05-30 18:34:24 +0000 [warn]: dump an error event: error_class=Fluent::ConcatFilter::TimeoutError error="Timeout flush: ...

Any clue or hint on how i can get to know the reason why these messages are not concatenated ok?

from fluent-plugin-concat.

For example. 1st Java exception is raised at 00:00:00, then 2nd Java exception is raised at 00:00:06 (after flush_interval).
You get concatenated 1st Java exception's logs.
2nd Java exception's logs are not concatenated and occured timeout error because 3rd Java exception has not been raised.

You cannot get concatenated log for each Java exception when you use multiline_start_regexp only.
You can get concatenated log for each Java exception when use both multiline_start_regexp and multiline_end_regexp.

from fluent-plugin-concat.