Add structured logging next to developer logging about rails HOT 3 OPEN

Huge +1 to having something out-of-the-box for this; we've managed to grow several different implementations of structured logging at Zendesk and I would very much like there to be a clear gold standard here to switch to!

One thing I would like to see addressed in a built-in structured logging solution is what to do about multiple independent processes writing to stdout.

It's very common, in my experience, for apps in container environments to be expected to write their logs to stdout, where it gets collected by the container engine and then forwarded to a centralised logging solution. It's also very common to run Rails apps in a multiprocess app server like Unicorn or Puma's cluster mode, so there might be several independent Rails processes running in the one container. The problem is, by default, all of those processes will share the single stdout stream, leading to issues where structured log messages might be interleaved and thus their structure broken.

I've been sketching out a way to solve this for us at Zendesk (https://github.com/zendesk/multiproc_log_device). Essentially, it runs a parent process above Unicorn which exposes a unix socket, and reopens stdout in each process to connect to that socket. Then, the parent process can merge the logs together in a structure aware way.

I would be very interested in hearing how other Rails users have dealt with this issue, and very very interested in trying to hammer out a way this problem can be addressed in a built-in Rails structured logging solution.

from rails.

we should add structured json logging on a per-request basis

Good initiative!

Maybe logfmt (https://brandur.org/logfmt) could be supported as an output alternative to JSON?

Assuming it's all strings and only one level deep (that's how JSON log lines are usually structured), logfmt is 1:1 with JSON log lines, so it's only a matter of output.

from rails.

Assuming it's all strings and only one level deep (that's how JSON log lines are usually structured), logfmt is 1:1 with JSON log lines, so it's only a matter of output.

It is not uncommon for JSON log lines to have multiple nested levels when using observability tools, so if logfmt is supported, I suggest making JSON the primary focus and gracefully degrading to logfmt.

Semantic Logger has a nice approach of allowing the user to choose among a list of formatters for each appender that receives the logs. In fact, if you take a look at their documentation for appenders (a concept that I think would be fantastic to have as part of this work), you'll find an example payload that shows a nested payload structure. 😁

from rails.