Comments (8)
Do you mean that you want to concatenate lines which has same timestamp?
Your logs like following?
Error [2018-10-01 15:12:31,323] (something goes here)
INFO [2018-10-01 15:12:31,323] (something gose here)
Maybe, you can concatenate such logs using filter_parser and this plugin like following:
<source>
@type tail
path /path/to/log
tag test.log
<parse>
@type none
</parse>
</source>
<filter>
@type parser
key_name message
<parse>
@type regexp
expression /^(?<level>(?:INFO|Error)) \[(?<timestamp>\d{4}\d{2}-\d{2}- \d{2}:\d{2}:\d{2})\](?<rest>.*)/
</parse>
</filter>
<filter>
@type concat
key message
stream_identity_key timestamp
multiline_start_regexp ... # I can't imagine your full logs...
</filter>
<match **>
@type stdout
</match>
But I did not test above configuration at all.
from fluent-plugin-concat.
and in kibana this !at org are different logs, i want to have them in the ERROR log
from fluent-plugin-concat.
You can use continuous_line_regexp
like following.
<filter>
@type concat
key message
multiline_start_regexp /^ERROR/
continuous_line_regexp /^!/
</filter>
from fluent-plugin-concat.
but in the difference type of log i got another type of string in the beggining thats why i want to make it by the date format can i just do something like
<filter kubernetes.**>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /^\[(?<timestamp>\d{4}\d{2}-\d{2}- \d{2}:\d{2}:\d{2})\](?<rest>.*)/
flush_interval 60
</filter>
from fluent-plugin-concat.
How about following?
<filter>
@type concat
key message
multiline_start_regexp /^[A-Z]+ \[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\].*/
continuous_line_regexp /^!/
</filter>
from fluent-plugin-concat.
i cant hardcode continous lines like that because not always they start with ! that's why i want to just start new log when he gets [\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}].*/
from fluent-plugin-concat.
multiline_start_regexp /\[\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\].*/ this works thanks
from fluent-plugin-concat.
Thank you!
from fluent-plugin-concat.
Related Issues (20)
- Concat containerd/docker output in the same config HOT 3
- What does Timeout flush exactly mean ? HOT 5
- concat lose last log line when it is a single line HOT 8
- Concatenating cri logs configError: Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required HOT 1
- How to concat messages handled by timeout_label
- error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kernel:default" location=nil tag="kernel" HOT 1
- Concatenate various fields
- Unknown filter plugin 'concat' [fluentd1.3.3 concat2.5.0] HOT 2
- Allow ruby expressions in plugin fields
- using the concat plugin with CRI logs ends up deleting the stream key from the record HOT 1
- process_partial_cri error="undefined method `split' for nil:NilClass" HOT 2
- Java multi-line logs merge into one line HOT 1
- separator not working for pretty json HOT 2
- Encoding Compatibility Error
- Concat problem on kubernetes with message with \n HOT 3
- emit transaction failed: error_class=NoMethodError error="undefined method `key?' for \"\":String" HOT 1
- Got "invalid byte sequence in UTF-8" error when use concat plugin HOT 6
- timeout flush and loss log HOT 4
- disable filter chain optimization because [Fluent::Plugin::ConcatFilter] uses filter_stream method HOT 1
- Can this handle multiline and non-multiline logs? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-concat.