Comments (4)
ctretyak
commented on June 25, 2024
1
I have the same problem. Example from README doesn't work.
<filter **>
@type concat
key message
multiline_start_regexp /^Start/
flush_interval 5
timeout_label @NORMAL
</filter>
<match **>
@type relabel
@label @NORMAL
</match>
<label @NORMAL>
<match **>
@type stdout
</match>
</label>
It's suppress warn about timeout, but flushed logs don't come to Elastic
from fluent-plugin-concat.
niltonvasques
commented on June 25, 2024
Same problem here... It happens after some days, and then suddenly the fluentd is not sending logs anymore to my elastic. Here is my settings:
My logs stopps to be processed and the only error log that I found was this timeout issue, happening immediatelly after the last processed log.
2021-06-15 20:30:08 +0000 [error]: #0 failed to flush timeout buffer error_class=ArgumentError error="@MULTILINE_LOGS label not found"
My settings:
# ...
<filter multiline.**>
@type concat
key message
multiline_start_regexp /^(E|F), \[[^ ]+ \#\d+\]( )+[^ ]+ -- :/
continuous_line_regexp /^(?!(^., \[[^ ]+ \#\d+\]( )+[^ ]+ -- :)).+/
flush_interval 60
timeout_label @MULTILINE_LOGS
</filter>
<filter multiline.**>
@type concat
key message
stream_identity_key request_id
multiline_start_regexp /^I, \[.+\] INFO -- : \[.+\] Started.+for.+at.+/
multiline_end_regexp /^I, \[.+\] INFO -- : \[.+\] Completed.+/
flush_interval 60
timeout_label @MULTILINE_LOGS
</filter>
# ...
<label @SYSTEM>
<match *.**>
@type copy
<store>
@type elasticsearch
host ...
port ...
logstash_format true
logstash_prefix fluentd
logstash_dateformat %Y%m%d
include_tag_key true
type_name access_log
tag_key @log_name
flush_interval 1s
reload_connections false
reconnect_on_error true
reload_on_failure true
@log_level debug
<buffer>
@type memory
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever true
retry_max_interval 15
chunk_limit_size 2M
queue_limit_length 32
overflow_action block
</buffer>
</store>
<store>
@type stdout
</store>
</match>
</label>
from fluent-plugin-concat.
niltonvasques
commented on June 25, 2024
I performed some changes in order to capture the timeout log, but after 5 days the same issue happened again. These was the changes:
last log
2021-06-22 18:09:36 +0000 [info]: #0 Timeout flush: multiline.:d41c1185-cbba-40c7-9d77-12449587539a
Changes
# ...
<filter multiline.**>
@type concat
# ....
timeout_label @SYSTEM
</filter>
<filter multiline.**>
@type concat
# ....
timeout_label @SYSTEM
</filter>
# ...
<label @SYSTEM>
<match *.**>
@type copy
<store>
# elastic ...
</store>
<store>
@type stdout
</store>
</match>
</label>
from fluent-plugin-concat.
satrushn
commented on June 25, 2024
I also tried to use labels and I've got errors in fluentd log like:
[error]: #0 failed to flush timeout buffer error_class=ArgumentError error="@Normal label not found"
And there is no last log line from the multiline batch of one event!
What to do?
from fluent-plugin-concat.
Related Issues (20)
- Concat containerd/docker output in the same config HOT 3
- What does Timeout flush exactly mean ? HOT 5
- concat lose last log line when it is a single line HOT 8
- Concatenating cri logs configError: Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required HOT 1
- How to concat messages handled by timeout_label
- error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kernel:default" location=nil tag="kernel" HOT 1
- Concatenate various fields
- Unknown filter plugin 'concat' [fluentd1.3.3 concat2.5.0] HOT 2
- Allow ruby expressions in plugin fields
- using the concat plugin with CRI logs ends up deleting the stream key from the record HOT 1
- process_partial_cri error="undefined method `split' for nil:NilClass" HOT 2
- Java multi-line logs merge into one line HOT 1
- separator not working for pretty json HOT 2
- Encoding Compatibility Error
- Concat problem on kubernetes with message with \n HOT 3
- emit transaction failed: error_class=NoMethodError error="undefined method `key?' for \"\":String" HOT 1
- Got "invalid byte sequence in UTF-8" error when use concat plugin HOT 6
- disable filter chain optimization because [Fluent::Plugin::ConcatFilter] uses filter_stream method HOT 1
- Can this handle multiline and non-multiline logs? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-concat.