What does Timeout flush exactly mean ? about fluent-plugin-concat HOT 5 CLOSED

What does Timeout flush excatly mean ?

1. multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2}/ matches to 2021-06-03 15:45:25 no 32 hello world
2. the following logs are not matched by multiline_start_regexp.
3. within flush_interval 3s, log entry is not matched at all

Thus, above configutation only works if something like 2021-06-03 15:45:25 no 32 hello world is logged periodically (every within 3 seconds)

What does fluent-plugin-concat flush ? to where ? and why timeout ?

In the above configuration, flush from 2021-06-03 15:45:25 no 32 hello world to ..., as a record then processed <match > section.

If you don't care about timeout event, you can throw it away like this:

<filter> 
  @type concat
 ...
  timeout_label @TIMEOUT
</filter>

<label @TIMEOUT>
  <match **>
    @type null
  </match>
</label>

from fluent-plugin-concat.

Thank you for your reply . But I still do not understand it .

What I thought is that :

1. multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2}/ matches to 2021-06-03 15:45:25 no 32 hello world
   fluent-plugin-concat think that it is the start part of a mutiline , and then send it to buffer
2. the following logs are not matched by multiline_start_regexp.
   fluent-plugin-concat think that they are continuous parts of the mutiline , these logs are also appended to the buffer
3. within 3 seconds (flush_interval 3s) , if fluent-plugin-concat see another new /^\d{4}-\d{1,2}-\d{1,2}/ , it will flush the buffer to <match> setion .
4. after 3 seconds (flush_interval 3s) , if fluent-plugin-concat still does not see another new /^\d{4}-\d{1,2}-\d{1,2}/ , it will flush the buffer to <match> setion .

Anyway , the main problem is that I can not figure out how to capture the last record in the logfile error.log-schedule-server.log.

This is the flunetd.conf, but the last log record will not be captured and sent to ES.

<system>
  log_level debug
</system>


<source>
  @type tail
  path /fluentd/etc/source-logs/schedule/error.log-schedule-server.log
  pos_file /fluentd/etc/source-logs/schedule/error.log-schedule-server.log.pos
  read_from_head true
  tag btcs.schedule.error
  <parse>
    @type none
  </parse>
</source>


<filter btcs.**>
  @type concat
  key message
  multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2}/
  flush_interval 3s
  timeout_label @NORMAL
</filter>


<match **>
  @type relabel
  @label @NORMAL
</match>

<label @NORMAL>
<match btcs.**>
  @type elasticsearch
  host elasticsearch
  index_name BTCS-all-in-one
  flush_interval 1s
</match>
</label>

from fluent-plugin-concat.

Same question for me here. My logs stopped to be processed and the only error log that I found was this timeout issue, happening immediatelly after the last processed log.

2021-06-15 20:30:08 +0000 [error]: #0 failed to flush timeout buffer error_class=ArgumentError error="@MULTILINE_LOGS label not found"

My settings:

# ...

<filter multiline.**>
   @type concat
   key message
   multiline_start_regexp /^(E|F), \[[^ ]+ \#\d+\]( )+[^ ]+ -- :/
   continuous_line_regexp /^(?!(^., \[[^ ]+ \#\d+\]( )+[^ ]+ -- :)).+/
   flush_interval 60
   timeout_label @MULTILINE_LOGS
</filter>

<filter multiline.**>
  @type concat
  key message
  stream_identity_key request_id
  multiline_start_regexp /^I, \[.+\]  INFO -- : \[.+\] Started.+for.+at.+/
  multiline_end_regexp /^I, \[.+\]  INFO -- : \[.+\] Completed.+/
  flush_interval 60
  timeout_label @MULTILINE_LOGS
</filter>

# ...

<label @SYSTEM>
  <match *.**>
    @type copy

    <store>
      @type elasticsearch
      host ...
      port ...
      logstash_format true
      logstash_prefix fluentd
      logstash_dateformat %Y%m%d
      include_tag_key true
      type_name access_log
      tag_key @log_name
      flush_interval 1s
      reload_connections false
      reconnect_on_error true
      reload_on_failure true
      @log_level debug
      <buffer>
         @type memory
         flush_mode interval
         retry_type exponential_backoff
         flush_thread_count 2
         flush_interval 5s
         retry_forever true
         retry_max_interval 15
         chunk_limit_size 2M
         queue_limit_length 32
         overflow_action block
      </buffer>
    </store>

    <store>
      @type stdout
    </store>
  </match>
</label>

from fluent-plugin-concat.

Finally, I switch to use "mutiline" parser of fluentd .

<source>
  @type tail
  path /logs/schedule/error.log-schedule-server.log
  pos_file /logs/schedule/error.log-schedule-server.log.pos
  read_from_head true
  tag btcs.schedule.error
  multiline_flush_interval 5s
  <parse>
    @type multiline
    format_firstline /\d{4}-\d{1,2}-\d{1,2}/
    format1 /^(?<message>.*)/
  </parse>
</source>

The key point is multiline_flush_interval 5s.
when multiline meet the first format_firstline line in the log file , it will wait 5s ,
after 5s , if it does not meet the second format_firstline line ,
all the log lines in the buffer will be considered as one multiline.
This is the key point for capture the last one multiline in the log file.

from fluent-plugin-concat.

I'll close that #108 (comment) solves the issue.

from fluent-plugin-concat.