ECS formatter for zerlog

pinging @axw & @karmi

The Setup guide documents with Filebeat but not with the now recommended Elastic Agent

provide ECS formatter for Serilog.

pinging @Mpdreamz, @russcam & @codebrain

Implement an ECS layout for winston.

logback.xml like this:

    <file>${LOG_FILE}</file>
    <rollingPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedRollingPolicy">
        <fileNamePattern>${LOG_FILE}.%d{yyyy-MM-dd}.%i.gz</fileNamePattern>
        <maxFileSize>${LOG_FILE_MAX_SIZE:-10MB}</maxFileSize>
        <maxHistory>${LOG_FILE_MAX_HISTORY:-0}</maxHistory>
    </rollingPolicy>
</appender>

logInfo:

{"@timestamp":"2021-04-29T02:41:57.625Z", "log.level": "INFO", "message":[id: 0x546ff99b] REGISTERED", "ecs.version": "1.2.0","service.name":"xxx","event.dataset":"xxx.log","process.thread.name":"duplex-server-boss-4-1","log.logger":"io.netty.handler.logging.LoggingHandler","log":{"origin":{"file":{"name":"AbstractInternalLogger.java","line":148},"function":"log"}}}

filebeat resolve:

nothing log.origin field

"message": {
"event.dataset": "spring-boot-application.log",
"process.thread.name": "duplex-server-boss-4-1",
"log.logger": "io.netty.handler.logging.LoggingHandler",
"@timestamp": "2021-04-29T03:12:51.943Z",
"log.level": "INFO",
"message": "[id: 0x546ff99b] REGISTERED",
"ecs.version": "1.2.0",
"service.name": "spring-boot-application"
}

source code here ：

co.elastic.logging.EcsJsonSerializer

public static void serializeOrigin(StringBuilder builder, String fileName, String methodName, int lineNumber) {
builder.append(""log":{");
builder.append(""origin":{");
builder.append(""file":{");
builder.append(""name":"");
JsonUtils.quoteAsString(fileName, builder);
builder.append('"');
if (lineNumber >= 0) {
builder.append(',');
builder.append(""line":");
builder.append(lineNumber);
}
builder.append("},");
builder.append(""function":"");
JsonUtils.quoteAsString(methodName, builder);
builder.append('"');
builder.append("}");
builder.append("},");
}

Similar to elastic/apm#231, let's also sync the spec to the individual ECS loggers.

As an MVP, we can start with the Java logger and then add others gradually.

loggers:
  - REPO: "ecs-logging-java"
    SPEC_PATH: "ecs-logging-core/src/test/resources/spec"

Add a formatter for Zend Log.

Add a formatter for Monolog.

Add a formatter for Rails' ActiveSupport::Logger

The ECS Logging Java Setup recommends the Filebeat log input that is deprecated since 7.16 in favor of the filestream input

provide a formatter for Microsoft.Extensions.Logging.Abstractions.

pinging @Mpdreamz, @russcam & @codebrain

Implement an ECS layout for structlog.

pinging @beniwohli & @basepi

Implement a formatter for logrus.

pinging @axw & @karmi

Just wanted to drop you a message to let you know that I've created a Swift logging implementation for logging in ECS log format:

rwbutler/swift-log-ecs

It's intended to be used with Apple's SwiftLog.

I created it to use myself with server-side Swift (Vapor) but hopefully it will be of use to others as well. It can be combined with other log handlers allowing log messages to be sent to different logging backends.

We've had multiple strems of discussions about how to properly version ECS libraries. There are three main options:

1. Version loggers independently, starting with 1.0.0 and using semver
   Loggers just depend on a small subset of ECS fields that are available since ECS 1.2. We don't expect many breaking changes on these fields, even in new major versions of ECS.
2. Version them in alignment with ECS
   The .NET loggers does that as it generates types based on the ECS schema definition. For ECS, there's a desire to align with the stack release versions as of 8.0.
3. Version loggers in alignment with the stack release
   Makes it easier for users to grasp which versions work well with each other, especially when it comes to breaking changes.

Currently, loggers are using a mix of 1. and 2.

We actually have a breaking change lined up for 8.0 regarding event.dataset vs data_stream.dataset (#38) where it would help if loggers would be aligned with the stack release.

Let's discuss which versioning scheme we want to follow for the library's GA and whether to change things up when 8.0 comes along.

I personally think that we should aim to align with the stack versions as of 8.0.

I'm not so sure if we should jump into that right away for the GA of the ECS loggers. My main concern would be the management overhead associated with it. I anticipate that most loggers are fairly stable with infrequent code changes after GA. When we align with stack releases, we'd still need to release minor and bugfix versions in sync with the stack release even if there are no changes in a logger. With automation in place, we can hopefully bring that overhead down to a minimum.

Open questions when aligning with the stack release

• Can we automatically create stack release branches to minimize the management overhead?
• How would the integration into unified release work?
• We shouldn't require users to update their Elastic stack and their log library versions at the same time. How should we define the support range of a ECS logger with the stack version? I guess at the least, a ECS logger with version 8.x should be compatible with all minors within 8.x and 8.last (or even 8.x?) should be compatible with 9.x. But should 8.x loggers also be compatible with 7.last or even 7.x versions of the stack?

We have started to put the documentation for the individual loggers to the elastic.co website.

Similar to the overview docs in APM, I think it makes sense that we have an overview page for ECS loggers that describe at a high level what ECS loggers are, what benefits they have, and link to the individual language loggers.

This can serve as a canonical page when linking from other parts of our docs, such as from the Filebeat docs.

I guess we could use some of the things that are written in the overview page of the ecs-logging-java docs. The question is if we want to duplicate some of the information, link to the generic overview docs, or have the docs for the individual loggers be sub-sections of the global docs.

@bmorelli25 do you have an opinion on that?

Implement an ECS compliant formatter/layout for ruby lang's Logger.

• Create validator of the spec
• Fill OSS checklist
• Create docs

Add ECS formatter support to bunyan.

The two kinds I see are the following (the naming here is up for discussion):

ECS Libraries

These libraries can ultimately be used to create or consume ECS events. Part of their goal is to represent the schema's key names and data types in the target language with fidelity, using the language's idioms (OOP, using the language's type closest to ECS', etc.)

Delivering these requires a lot of work to map ECS & Elasticsearch semantics to the target language. Each ECS release will require a release of this type of library, since it's meant to map all of the ECS field definitions.

In this category:

• The Go library, currently in the ECS repo
• The .Net library

ECS Logging Formatters

I see these as "drop-in" log formatters. By "drop-in" I mean as a baseline, developers need not care about ECS and all its fields per se. They should be useable by developers who just want to provide a log level, a message and perhaps a few structured keys. In doing so, these log formatters should then populate the correct ECS fields based on no changes to the application (other than changing the log formatter).

I like the Java library's list of fields that can be populated, based on this getting started experience.

These libraries should also eventually allow users that care about ECS and want to fill specific additional ECS fields to do so.

Once we reach a baseline functionality for these libraries, I think they will require less adjustments and changes over time. They will mostly map to a stable subset of ECS fields (see like to Java lib above), so an ECS release won't require a release of the library. When users want to fill more ECS fields, this can be done by using strings and datatypes that fit well enough so that the JSON representation will successfully populate an ECS index. They don't need the full type safety and support of the language, they just need the JSON output to match.

In this category:

• The Java Logging library
• The PHP Logging library

I think both of these types of libraries can play together if we want: pass an "ECS object" to a log formatter, and it populates appropriate all ECS fields. But I don't see this as necessary. I still see them as distinct types of libraries.

Next steps

• Converge on correct naming conventions for each.
• Perhaps we need a spec for each kind? I see this new spec as a good starting point for the logging formatters.
• Do we want to deliver one kind before the other?
• Feel free to bring up any additional thing that needs to be hashed out between the two.

Implement a logger for lograge.

create an ECS formatter for python's logging.

pinging @beniwohli & @basepi

When making the ECS logger and formatter libraries GA, most of them will be versioned independently of the stack or ECS, therefore library versions will be starting with v1.0.0.

The loggers set the ecs.version according to the ecs-logging spec. The current behavior for most loggers is to set the ecs.version to the minimal supported ECS version supported by this logger. This has the advantage that the logger libraries do not need to be updated and released on every ECS release. It also seems reasonable to use the minimal supported ECS version when querying log data.

@elastic/ecs does setting the ecs.version to the minimal supported version sound reasonable from an ECS perspective?

Hey @kuisathaverat,

could you or someone from the automation team help to set up a CI Server for these projects?
The idea would be to have one or more repos for Java, PHP, .NET, Ruby, JavaScript, Python and go.

Do you think it makes sense to have a separate CI server for that or would you prefer to re-use the observability/apm-ci for that?

https://www.app-metrics.io/

Incredibly popular .NET metrics library.

Potential overlap here: elastic/ecs#474

As discussed during our last sync, we need to add a proper beta disclaimer to all repositories to state that the layouts, enrichers, formatters provided by Elastic are still in a beta state.

Beta Disclaimer & Tasks

Text

Please note that this is library is in a beta version and backwards-incompatible changes might be introduced in future releases.

Tasks

• Add the disclaimer note as md to this repo #25

Repositories

• ecs-logging-java
• ecs-logging-js
• ecs-dotnet
• ecs-logging-php
• ecs-logging-python

Add an ECS formatter for pino similar to this.

Formatter for zap

pinging @axw & @karmi

https://www.elastic.co/guide/en/ecs-logging/overview/master/intro.html#_why_ecs_logging says:

Decently human-readable JSON structure

The first three fields are always @timestamp, log.level and message. ...

The following logging frameworks do not currently support controlling the order of log record fields, so cannot reasonably support this as a strong requirement:

• Go's zap (https://github.com/elastic/ecs-logging-go-zap)
• Node.js's pino (https://github.com/pinojs/pino, https://github.com/elastic/ecs-logging-js/tree/master/loggers/pino)

This issue is to collect notes and concerns with relaxing the spec and language around these fields always having to be first.

• Missing ECS loggers
  • Ruby
  • Go: logrus and zerolog
• Ensure all loggers comply with spec
  • Validate against the spec (example: elastic/ecs-logging-java#111)
  • Ensure loggers integrate with APM by adding trace.id and transaction.id identifiers (see also #29)
• Dotted field names: dot_expand Filebeat processor to deal with dotted field names like log.level elastic/beats#20489 -> elastic/beats#22849
• Review external licenses & OSS compliance
• Docs
  • Add documentation for each logger on elastic.co (example: elastic/ecs-logging-java#112, live docs)
  • Cross-links from Filebeat and APM log correlation documentation

Implementation Status