Comments (9)
Can you double check that
framing = "nul"
is present in your [input]
section?
from flowgger.
yes, the framing is "nul"
from flowgger.
As a temporary workaround, 4fe397f downgrades serde_json
to the 0.7
branch.
from flowgger.
Thank you very much for the quick fix!
from flowgger.
Actually, I'm not sure this is a regression. The behavior of serde_json
0.8 appears to be correct, and actually fixes a lack of proper validation that was present in the 0.7 series.
According to the JSON specification, characters in unicode range U+0000
to U+001F
, which includes\n
, are control characters. These must be escaped.
This is consistent with Javascript parsers that refuse straight newlines in JSON values:
JSON.parse('{"t":"x\ny"}')
throws a SyntaxError: Unexpected token in JSON at position 7
exception.
from flowgger.
Yes, I know this does not pass most of json validator with '\n'.
But in gelf specification:
http://docs.graylog.org/en/2.0/pages/gelf.html
Character '\n' is allowed in example payload.
from flowgger.
I think there is a misunderstanding of the GELF specification, more specifically of the example they provide.
{ "full_message": "Backtrace here\nmore stuff" }
is the final content of a single JSON string. No replacements will be applied to the content within quotes. \n
is and will remain parsed as the character \
followed by the character n
.
This is not equivalent to
{ "full_message": "Backtrace here
more stuff" }
which is a different string (and not a valid JSON string).
JSON strings should be considered as raw strings, not as higher-level strings as present in several programming languages, that will eventually be converted to raw strings.
As a result, the GELF specification does not allow straight 0x0a
characters within strings, and clients depending on this behavior are broken.
from flowgger.
A note was added to the Wiki in order to clarify this.
Producing conformant GELF messages should never be an issue with client libraries using JSON encoders.
What GELF client libraries producing invalid JSON strings did you use?
from flowgger.
I tested Logstash and other gelf library, all of them parse json directly.
For my simple test given previously, I do mistake from the payload, I must protect the "\n" in echo command to be valid JSON.
I have some client which I do not know their GELF clients (mostly apache logs), that generates "new line special character" in their JSON in their value. It breaks like 10-20% of their logs sent to flowgger when I upgraded to 0.2.0. This behaviour was new, so I was worried at first.
Issue like Graylog2/graylog2-server#2048
Has described in graylog's issue, GELF must be a valid JSON to be decode.
Valid test with echo command of GELF/TCP+TLS:
echo -e '{"version":"1.1", "host": "example.org", "short_message": "A short GELF message that helps you identify what is going on", "full_message": "Backtrace here\\n\\nmore stuff", "timestamp": 1470749969, "level": 1, "_user_id": 9001, "_some_info": "foo", "some_metric_num": 42.0}\0' | openssl s_client -quiet -no_ign_eof -connect localhost:12202
from flowgger.
Related Issues (20)
- Publish to crates.io HOT 10
- Update dependencies version HOT 2
- Add file size/time based rotation
- Add log pass-through feature HOT 1
- Increase unit test coverage and add performance testing
- GELF decoder doesn't respect output.gelf_extra add or replace fields
- LTSV decoder doesn't respect output.ltsv_extra add or replace fields
- coio is not build in cargo.io migrate to tokyo the tls module HOT 6
- Lua Parsing HOT 1
- Timezone ? HOT 1
- Comparision to Vector (https://vector.dev/) HOT 5
- Decoder RFC 5424 does not correctly parse structured data HOT 2
- Log file naming hardcode the time and extensions separators
- Enhancement request: NATS Output HOT 1
- Release a new version HOT 2
- Build fail with redis feature HOT 1
- Send and receive utf-8 messages HOT 1
- Add timestamps HOT 2
- Can I create a simple effective filter using config file flowgger.toml?
- Cross build workflow fails again HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flowgger.