Comments (5)
Akhisar
commented on May 27, 2024
pasting the config again
<source>
@id fluentd-containers-flink.log
@type tail
path /var/log/containers/**flink**.log
pos_file /var/log/fluentd-containers-flink.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag raw.kubernetes.*
format json
@label @CONCAT_FLINK
</source>
<label @CONCAT_FLINK>
<filter **flink**>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /\d{4}-\d{1,2}-\d{1,2}/
continuous_line_regexp /Caused\sby:|\s+at\s.*|^java|\s+ \.\.\. (\d)+ more/
#multiline_end_regexp /\s+.*more$/
flush_interval 10
timeout_label @OUTPUT
</filter>
<filter **flink**>
@type parser
key_name log
reserve_data true
<parse>
@type regexp
expression /^(?<log_date>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s(?<log_level>\w+)\s+(?<log_logger>.*?\s)\s+-(?<log_message>.*)/m
</parse>
</filter>
<match>
@type relabel
@label @OUTPUT
</match>
</label>from fluent-plugin-concat.
okkez
commented on May 27, 2024
What is your expected result?
If you want to remove an extra newline \n, you can use the separator parameter.
I've tested with the following configuration:
<source>
@type dummy
dummy [
{"log":"2019-02-08 10:03:50,885 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:35000] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:03:50.885661556Z"},
{"log":"2019-02-08 10:03:51,670 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n","stream":"stdout","time":"2019-02-08T10:03:51.671149184Z"},
{"log":"java.io.IOException: Unknown operation 71\n","stream":"stdout","time":"2019-02-08T10:03:51.671201184Z"},
{"log":"\u0009at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:03:51.671218384Z"},
{"log":"2019-02-08 10:04:05,884 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:35602] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:04:05.884977614Z"},
{"log":"2019-02-08 10:04:06,670 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n","stream":"stdout","time":"2019-02-08T10:04:06.671248435Z"},
{"log":"java.io.IOException: Unknown operation 71\n","stream":"stdout","time":"2019-02-08T10:04:06.671281135Z"},
{"log":"\u0009at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:04:06.671293435Z"},
{"log":"2019-02-08 10:04:20,885 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:36216] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:04:20.886034327Z"},
{"log":"2019-02-08 10:04:21,671 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n","stream":"stdout","time":"2019-02-08T10:04:21.672123047Z"},
{"log":"java.io.IOException: Unknown operation 71\n","stream":"stdout","time":"2019-02-08T10:04:21.672146847Z"},
{"log":"\u0009at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:04:21.672151847Z"}
]
tag dummy
</source>
<filter dummy>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /\d{4}-\d{1,2}-\d{1,2}/
continuous_line_regexp /Caused\sby:|\s+at\s.*|^java|\s+ \.\.\. (\d)+ more/
#multiline_end_regexp /\s+.*more$/
flush_interval 10
timeout_label @OUTPUT
</filter>
<match dummy>
@type stdout
</match>The result:
2019-02-12 11:07:10.093425404 +0900 dummy: {"log":"2019-02-08 10:03:50,885 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:35000] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:03:50.885661556Z"}
2019-02-12 11:07:13.098477408 +0900 dummy: {"log":"2019-02-08 10:03:51,670 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n\njava.io.IOException: Unknown operation 71\n\n\tat org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:03:51.671149184Z"}
2019-02-12 11:07:14.000064485 +0900 dummy: {"log":"2019-02-08 10:04:05,884 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:35602] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:04:05.884977614Z"}
2019-02-12 11:07:17.004081321 +0900 dummy: {"log":"2019-02-08 10:04:06,670 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n\njava.io.IOException: Unknown operation 71\n\n\tat org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:04:06.671248435Z"}
2019-02-12 11:07:18.005495485 +0900 dummy: {"log":"2019-02-08 10:04:20,885 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.21.0.106:36216] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n","stream":"stdout","time":"2019-02-08T10:04:20.886034327Z"}
2019-02-12 11:07:21.010161527 +0900 dummy: {"log":"2019-02-08 10:04:21,671 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n\njava.io.IOException: Unknown operation 71\n\n\tat org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","time":"2019-02-08T10:04:21.672123047Z"}
from fluent-plugin-concat.
Akhisar
commented on May 27, 2024
Hi,
Thanks for the reply.
My issue is that in Kibana both the log lines are concatenated together which is strange.
from fluent-plugin-concat.
okkez
commented on May 27, 2024
Hmm, it's strange...
You can debug your configuration using out_stdout plugin and filter_stdout plugin like following:
<source>
@id fluentd-containers-flink.log
@type tail
path /var/log/containers/**flink**.log
pos_file /var/log/fluentd-containers-flink.log.pos
time_format %Y-%m-%dT%H:%M:%S.%NZ
tag raw.kubernetes.*
format json
@label @CONCAT_FLINK
</source>
<label @CONCAT_FLINK>
<filter **flink**>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /\d{4}-\d{1,2}-\d{1,2}/
continuous_line_regexp /Caused\sby:|\s+at\s.*|^java|\s+ \.\.\. (\d)+ more/
#multiline_end_regexp /\s+.*more$/
flush_interval 10
timeout_label @OUTPUT
</filter>
<filter **flink**>
@type stdout
</filter>
<filter **flink**>
@type parser
key_name log
reserve_data true
<parse>
@type regexp
expression /^(?<log_date>\d{4}-\d{2}-\d{2}\s\d{2}:\d{2}:\d{2},\d{3})\s(?<log_level>\w+)\s+(?<log_logger>.*?\s)\s+-(?<log_message>.*)/m
</parse>
</filter>
<match>
@type relabel
@label @OUTPUT
</match>
</label>
<label @OUTPUT>
<match>
@type stdout
</match>
</label>
from fluent-plugin-concat.
Akhisar
commented on May 27, 2024
I checked with stdout on output side.
With
<filter **flink**>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2},\d{1,3} .*/
multiline_end_regexp /\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2},\d{1,3}/
flush_interval 10
timeout_label @OUTPUT
</filter>2019-03-05 07:05:33.348407995 +0000 kubernetes.var.log.containers.scenario-service-flink-jobmanager-547fcc8f48-6j7sq_nightly_scenario-service-flink-jobmanager-7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6.log: {"log":"2019-03-05 07:05:33,144 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n","stream":"stdout","log_date":"2019-03-05 07:05:33,144","log_level":"ERROR","log_logger":"org.apache.flink.runtime.blob.BlobServerConnection ","log_message":" Error while executing BLOB connection.\n","docker":{"container_id":"7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6"},"kubernetes":{"container_name":"scenario-service-flink-jobmanager","namespace_name":"nightly","pod_name":"scenario-service-flink-jobmanager-547fcc8f48-6j7sq","pod_id":"e096ba68-3bab-11e9-b301-92e9bbf34b8a","labels":{"app":"scenario-service-flink-jobmanager","pod-template-hash":"1039774904","release":"nightly-scenario-service-flink-jobmanager"},"host":"aks-nightly-49443436-2","master_url":"https://nightly-4f2ceb09.hcp.westeurope.azmk8s.io:443/api","namespace_id":"ccf92e65-3bab-11e9-b301-92e9bbf34b8a"},"cluster":"KubernetesNightlyFRP"}
2019-03-05 07:05:38 +0000 [info]: #0 stats - namespace_cache_size: 1, pod_cache_size: 4, namespace_cache_api_updates: 4, pod_cache_api_updates: 4, id_cache_miss: 4
2019-03-05 07:05:33.145215330 +0000 kubernetes.var.log.containers.scenario-service-flink-jobmanager-547fcc8f48-6j7sq_nightly_scenario-service-flink-jobmanager-7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6.log: {"log":"java.io.IOException: Unknown operation 71\n\tat org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","docker":{"container_id":"7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6"},"kubernetes":{"container_name":"scenario-service-flink-jobmanager","namespace_name":"nightly","pod_name":"scenario-service-flink-jobmanager-547fcc8f48-6j7sq","pod_id":"e096ba68-3bab-11e9-b301-92e9bbf34b8a","labels":{"app":"scenario-service-flink-jobmanager","pod-template-hash":"1039774904","release":"nightly-scenario-service-flink-jobmanager"},"host":"aks-nightly-49443436-2","master_url":"https://nightly-4f2ceb09.hcp.westeurope.azmk8s.io:443/api","namespace_id":"ccf92e65-3bab-11e9-b301-92e9bbf34b8a"},"cluster":"KubernetesNightlyFRP"}
Here I get three log entries:
First:
WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.23.0.106:54728] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded
Second:
ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.
Third:
java.io.IOException: Unknown operation 71
at org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)
With
<filter **flink**>
@type concat
key log
stream_identity_key container_id
multiline_start_regexp /^\d{4}-\d{1,2}-\d{1,2} \d{2}:\d{2}:\d{2},\d{1,3} .*/
flush_interval 10
timeout_label @OUTPUT
</filter>2019-03-05 06:59:47.428020218 +0000 kubernetes.var.log.containers.scenario-service-flink-jobmanager-547fcc8f48-6j7sq_nightly_scenario-service-flink-jobmanager-7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6.log: {"log":"2019-03-05 06:59:37,701 WARN akka.remote.transport.netty.NettyTransport - Remote connection to [/172.23.0.106:43358] failed with org.apache.flink.shaded.akka.org.jboss.netty.handler.codec.frame.TooLongFrameException: Adjusted frame length exceeds 40485760: 1195725860 - discarded\n2019-03-05 06:59:48,146 ERROR org.apache.flink.runtime.blob.BlobServerConnection - Error while executing BLOB connection.\n\njava.io.IOException: Unknown operation 71\n\n\tat org.apache.flink.runtime.blob.BlobServerConnection.run(BlobServerConnection.java:120)\n","stream":"stdout","docker":{"container_id":"7306a89cc3da3f55ae6c37c18228e658388f70168796311110459ec83b8fb4b6"},"kubernetes":{"container_name":"scenario-service-flink-jobmanager","namespace_name":"nightly","pod_name":"scenario-service-flink-jobmanager-547fcc8f48-6j7sq","pod_id":"e096ba68-3bab-11e9-b301-92e9bbf34b8a","labels":{"app":"scenario-service-flink-jobmanager","pod-template-hash":"1039774904","release":"nightly-scenario-service-flink-jobmanager"},"host":"aks-nightly-49443436-2","master_url":"https://nightly-4f2ceb09.hcp.westeurope.azmk8s.io:443/api","namespace_id":"ccf92e65-3bab-11e9-b301-92e9bbf34b8a"},"cluster":"KubernetesNightlyFRP"}
with this all three logs are clubbed together. But I want first log as one event and second and third combined as the next event.
from fluent-plugin-concat.
Related Issues (20)
- Concat containerd/docker output in the same config HOT 3
- What does Timeout flush exactly mean ? HOT 5
- concat lose last log line when it is a single line HOT 8
- Concatenating cri logs configError: Either n_lines, multiline_start_regexp, multiline_end_regexp, partial_key or use_partial_metadata is required HOT 1
- How to concat messages handled by timeout_label
- error_class=Fluent::Plugin::ConcatFilter::TimeoutError error="Timeout flush: kernel:default" location=nil tag="kernel" HOT 1
- Concatenate various fields
- Unknown filter plugin 'concat' [fluentd1.3.3 concat2.5.0] HOT 2
- Allow ruby expressions in plugin fields
- using the concat plugin with CRI logs ends up deleting the stream key from the record HOT 1
- process_partial_cri error="undefined method `split' for nil:NilClass" HOT 2
- Java multi-line logs merge into one line HOT 1
- separator not working for pretty json HOT 2
- Encoding Compatibility Error
- Concat problem on kubernetes with message with \n HOT 3
- emit transaction failed: error_class=NoMethodError error="undefined method `key?' for \"\":String" HOT 1
- Got "invalid byte sequence in UTF-8" error when use concat plugin HOT 6
- timeout flush and loss log HOT 4
- disable filter chain optimization because [Fluent::Plugin::ConcatFilter] uses filter_stream method HOT 1
- Can this handle multiline and non-multiline logs? HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluent-plugin-concat.