Is your feature request related to a problem? Please describe.
The ECS description for the Client.IP field seems to suggest that it supports multiple values: "Can be one or multiple IPv4 or IPv6 addresses." and is currently implemented as a string. When pulling data from HttpContext you end up with a value that contains both IPv4 and IPv6 addresses as a single string. It would be really useful to have these as separate values because with the current implementation it becomes harder to filter for a given IP-address when searching the logs.

Describe the solution you'd like
Change the Elastic.CommonSchema.Client.Ip datatype from string to string[] and change the two implementations of Elastic.CommonSchema.Serilog.HttpAdapter to get the IPv4 and IPv6 addresses separately. Maybe the MapToIPv4() and MapToIPv6() methiods are useful?

var ip4 = _httpContextAccessor.HttpContext.Features.Get<IHttpConnectionFeature>()?.RemoteIpAddress.MapToIPv4();
var ip6 = _httpContextAccessor.HttpContext.Features.Get<IHttpConnectionFeature>()?.RemoteIpAddress.MapToIPv6();

Describe alternatives you've considered
Perhaps a config option in Elastic.CommonSchema.Serilog.IEcsTextFormatterConfiguration where you can choose if you want to store the IPv4, IPv6 og both addresses in the Client.Ip field? Then you wouldn't have to change the datatype.

Additional context
This is what is currently logged for Client when writing a log event using Elastic.CommonSchema.Serilog.EcsTextFormatter:

    "client": {
        "user": {
            "id": ["de52e1bd-7486-45f6-bed0-3734f655f563"],
            "name": "administrator"
        },
        "address": "::ffff:172.19.0.1",
        "ip": "::ffff:172.19.0.1"
    }

It would be nize to instead get something like this:

    "client": {
        "user": {
            "id": ["de52e1bd-7486-45f6-bed0-3734f655f563"],
            "name": "administrator"
        },
        "address": "::ffff:172.19.0.1",
        "ip": ["::ffff","172.19.0.1"]
    }

ECS version: latest as of writing

Elasticsearch version (if applicable): not applicable

.NET framework / OS: .NET Core 3.1, Windows and Linux

Description of the problem, including expected versus actual behavior:
When using Serilog and the WithElasticApmCorrelationId method, I get a StackOverflowException. Without that call, I don't get the exception.

Here's a stack trace (which clearly shows there's a lot of recursion).

System.Private.CoreLib.dll!System.Globalization.CompareInfo.FindString(uint dwFindNLSStringFlags, string lpStringSource, int startSource, int cchSource, string lpStringValue, int startValue, int cchValue, int* pcchFound) Unknown
System.Private.CoreLib.dll!System.Globalization.CompareInfo.StartsWith(string source, string prefix, System.Globalization.CompareOptions options) Unknown
System.Private.CoreLib.dll!System.Globalization.CompareInfo.IsPrefix(string source, string prefix, System.Globalization.CompareOptions options) Unknown
System.Private.CoreLib.dll!string.StartsWith(string value, System.StringComparison comparisonType) Unknown
System.Private.CoreLib.dll!string.StartsWith(string value) Unknown
Serilog.Extensions.Logging.dll!Serilog.Extensions.Logging.SerilogLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception exception, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.AspNetCore.dll!Elastic.Apm.AspNetCore.AspNetCoreLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Elastic.Apm.Logging.LogLevel level, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception e, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.ScopedLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Elastic.Apm.Logging.LogLevel level, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception e, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.LoggingExtensions.DoLog(Elastic.Apm.Logging.IApmLogger logger, Elastic.Apm.Logging.LogLevel level, string message, System.Exception e, object[] args) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.LoggingExtensions.MaybeLogger.Log(string message, object[] args) Unknown
Elastic.Apm.dll!Elastic.Apm.CurrentExecutionSegmentsContainer.ContextLocalHolder<Elastic.Apm.Model.Transaction>.Value.get() Unknown
Elastic.Apm.dll!Elastic.Apm.CurrentExecutionSegmentsContainer.CurrentTransaction.get() Unknown
Elastic.Apm.dll!Elastic.Apm.Api.Tracer.CurrentTransaction.get() Unknown
Elastic.Apm.SerilogEnricher.dll!Elastic.Apm.SerilogEnricher.ElasticApmEnricher.Enrich(Serilog.Events.LogEvent logEvent, Serilog.Core.ILogEventPropertyFactory propertyFactory) Unknown
Serilog.dll!Serilog.Core.Enrichers.SafeAggregateEnricher.Enrich(Serilog.Events.LogEvent logEvent, Serilog.Core.ILogEventPropertyFactory propertyFactory) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Write(Serilog.Events.LogEvent logEvent) Unknown
Serilog.Extensions.Logging.dll!Serilog.Extensions.Logging.SerilogLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Microsoft.Extensions.Logging.LogLevel logLevel, Microsoft.Extensions.Logging.EventId eventId, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception exception, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.AspNetCore.dll!Elastic.Apm.AspNetCore.AspNetCoreLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Elastic.Apm.Logging.LogLevel level, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception e, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.ScopedLogger.Log<Elastic.Apm.Logging.LogValuesFormatter.LogValues>(Elastic.Apm.Logging.LogLevel level, Elastic.Apm.Logging.LogValuesFormatter.LogValues state, System.Exception e, System.Func<Elastic.Apm.Logging.LogValuesFormatter.LogValues, System.Exception, string> formatter) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.LoggingExtensions.DoLog(Elastic.Apm.Logging.IApmLogger logger, Elastic.Apm.Logging.LogLevel level, string message, System.Exception e, object[] args) Unknown
Elastic.Apm.dll!Elastic.Apm.Logging.LoggingExtensions.MaybeLogger.Log(string message, object[] args) Unknown
Elastic.Apm.dll!Elastic.Apm.CurrentExecutionSegmentsContainer.ContextLocalHolder<Elastic.Apm.Model.Transaction>.Value.get() Unknown
Elastic.Apm.dll!Elastic.Apm.CurrentExecutionSegmentsContainer.CurrentTransaction.get() Unknown
Elastic.Apm.dll!Elastic.Apm.Api.Tracer.CurrentTransaction.get() Unknown
Elastic.Apm.SerilogEnricher.dll!Elastic.Apm.SerilogEnricher.ElasticApmEnricher.Enrich(Serilog.Events.LogEvent logEvent, Serilog.Core.ILogEventPropertyFactory propertyFactory) Unknown
Serilog.dll!Serilog.Core.Enrichers.SafeAggregateEnricher.Enrich(Serilog.Events.LogEvent logEvent, Serilog.Core.ILogEventPropertyFactory propertyFactory) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Serilog.Core.ILogEventSink.Emit(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Dispatch(Serilog.Events.LogEvent logEvent) Unknown
Serilog.dll!Serilog.Core.Logger.Write(Serilog.Events.LogEvent logEvent) Unknown

I have cut some of the stack trace down a bit, since it is insanely long and just repeats itself for obvious reasons.

Here's a screenshot of my Call Stack window at the time of the exception occuring, if that makes it easier to read.

ECS schema version (e.g. 1.4.0): 1.5

ECS .NET assembly version (e.g. 1.4.2): 1.5.1

Elasticsearch version (if applicable):

.NET framework / OS: .NET Core 2.1 / Windows

Description of the problem, including expected versus actual behavior: The library seems to cause HttpContext.Request.Method to be empty in some random occurrences. This is most likely caused by HttpContext being accessed between the requests.

Additional resources:

• dotnet/aspnetcore#3627
• aspnet/KestrelHttpServer#2591

Steps to reproduce:

1. Configure the ".MapHttpContext(ctx.Configuration.Get())"
2. Set up a script that will send 100+ requests to the service. Some of them should be 404 since the HttpContext.Request.Method isn't set.
3. Remove the ".MapHttpContext(ctx.Configuration.Get())"
4. Set up a script that will send 100+ requests to the service. All requests should be fine.

Since it is not possible to serialize directly into the StringBuilder, then you can simplify this method:

		protected override void RenderFormattedMessage(LogEventInfo logEventInfo, StringBuilder target)
                {
                        // blah blah
			var output = ecsEvent.Serialize();
                        target.Append(output);
                }

Into this:

		protected override string GetFormattedMessage(LogEventInfo logEvent)
		{
                        // blah blah
			var output = ecsEvent.Serialize();
                        return output;
		}

And remove the existing GetFormattedMessage

Tracking desires for GA release:

• Blog post for ECS .NET (to raise awareness + show usage) (in staging)
• Update README.md in root repository as ".NET integrations repository" (look at sourcelink README for inspiration).
• Determine how to handle pre-release package references - #38
• Decide how to publish/version the Elastic.CommonSchemaNEST assembly - not including in GA release
• NLog Formatter - moving to an issue ticket and removing from this GA meta ticket
• Reevaluate if we want to release prior versions as GA - opening as new ticket
• Lock versioning to MAJOR.MINOR.0 instead of MAJOR.0.0 in the assembly identity - #34
• Lock versioning to MAJOR.MINOR.0 instead of MAJOR.0.0 in nuget package dependency ranges - #38

Hi,

currently, we play around with the usage of the schema. We have trouble with the metadata field.

Is there any reason why the Metadata field is named _metadata?
Because also Elasticsearch mentioned here: "Field names beginning with _ are not supported"

There is another feature request to allow structured data on the root level. This would also be a solution. But we need a way to extend the logging with our own fields for business requirements. The schema does not fit here for all cases (e.g. we want to have all headers of the request logged)

Is your feature request related to a problem? Please describe.
I was testing out an Alpha release of the NLog Elastic target (markmcdowell/NLog.Targets.ElasticSearch#125) and I found a few items I was curious about.

1. Why is log.level a top level field instead of being under the log object?
2. I don't see any use of ${callsite} or ${callsite-linenumber} in the output. I'd prefer to see those under log.origin.function and log.origin.file.line respectively assuming they are available at runtime. Looking at the Elastic.CommonSchema.Log object definition, this should be doable. Perhaps having a boolean flag on the layout so people could configure to send it or not?
3. As part of the "process" information, I'd like to be able to output the assembly version information so I can track errors that may have happened due to an upgrade. Before I found this PR, I was planning on putting it under the existing package.version field since the "process" object didn't seem to have version info. Can that be configured? @snakefoot mentioned that the version number is available under Service. I also saw it available under Package. I think Service makes more sense since the Package fields are probably meant more for tracking installations instead of run logs.
4. For some reason the log.logger shows as "Elastic.CommonSchema.NLog" instead of my logger name (i.e. "my.namespace.MyClass"). This seems unexpected. I don't really care what the logging system was that sent the log to Elastic, I care about the system generating the log. fixed in #67

Describe the solution you'd like
Some more configuration options for logging items.

• Flag to log the CallSite information under the Log.Origin structure
• If there really isn't a good place to store the process version, placing that in a tag would be nice.
• Don't place the log.level as a top level field and instead use the field on the log object.
• Upgrade the documentation for NLog configuration to include configuring <metadata> items that can be sent in addition to the standard logs.

This c# code...

class Program
{
    static void Main(string[] args)
    {
        Log.Logger = new LoggerConfiguration()
            .MinimumLevel.Debug()
            .WriteTo.Console(new EcsTextFormatter())
            .CreateLogger();
        var user = new Elastic.CommonSchema.User
        {
            Id = "111",
            Name = "test"
        };
        Log.Information("{@user}", user);
    }
}

...puts the user object under '_metadata'. I would have thought that because I used an Elastic.CommonSchema.User object, it would have come out at the top level. What am I missing?

Configuration...
Elastic.CommonSchema 1.5.1
Elastic.CommonSchema.Serilog 1.5.1
Serilog 2.10.0
Serilog.Sinks.Console 3.1.1
.net core 3.1 console application.

Relates: elastic/ecs-logging#27

There is an effort to make ECS logging generally available across the language ecosystems of APM agents. As part of this, the ECS projects for .NET should be checked:

• Validation against the spec
• Integrate with APM by adding trace.id and transaction.id
• Review of dependency licenses
• Documentation for the elastic.co website

Is your feature request related to a problem? Please describe.

Bug reports that say "X isn't working", but don't specify which library / logging platform they are using, are confusing. If the context is not clear, you need to first ask which library they are using.

Describe the solution you'd like

Update the bug template to include a question for which library/Nuget package the issue is related to (Extensions, Serilog, NLog, etc).

You could also add something similar in the feature request section, if relevant.

Describe alternatives you've considered

Having to either guess or ask in each submitted issue what it relates to.

Additional context

Could also add tags to make it easy to identify on the issue list. e.g. Core, Extensions, NLog, Serilog.

Is your feature request related to a problem? Please describe.
In ECS there is no top level "log.level" property. Generally in Elastic, this doesn't seem to pose an issue as keying works the same when using "log.level" or "log": { "level": ...}. However, this does not match the ECS specification and when pulling the documents themselves, the "log.level" is not in the "right" place.

Describe the solution you'd like
Possibly post the log level in both fields like so:

{
    "@timestamp": "2020...",
    "message": "my log message",
    "log.level": "ERROR",
    "log": {
        "level": "ERROR",
...
}

Describe alternatives you've considered
Don't post the top level "log.level" key at all, and just use the object key

Additional context
see #77 for past discussion.

ECS integration/library project(s) (e.g. Elastic.CommonSchema.Serilog): Elastic.CommonSchema.NLog + Elastic.Apm.NLog

ECS schema version (e.g. 1.4.0):

ECS .NET assembly version (e.g. 1.4.2): 1.5.1

Elasticsearch version (if applicable):

.NET framework / OS: .NET-Framework 4.7.2, Win 10

Description of the problem, including expected versus actual behavior:
If I log a Warning with an exception, the logentry isn't written to the file when i use the EcsLayout

Steps to reproduce:

1. add Extensions to nlog.config (also Install the according NuGet-Packages):

 	<extensions>
		<add assembly="NLog.Targets.ElasticSearch"/>
		<add assembly="Elastic.Apm.NLog"/>
		<add assembly="Elastic.CommonSchema.NLog"/>	
	</extensions>

2. Add a target that uses the EcsLayout

 <target xsi:type="File" name="logEcs" fileName="./logs/ecs.json">
	  <layout xsi:type="EcsLayout"/>
</target>

3. try to write a log-entry with an exception:

        private static readonly NLog.Logger Logger = NLog.LogManager.GetCurrentClassLogger();

        static void Main(string[] args)
        {
            Logger.Info("Hello world");
            Logger.Warn(new KeyNotFoundException("Key"), "Error!!!");
        }

The Output-File only has 1 logmessage written, the one with the exception is missing_

{"@timestamp":"2021-03-31T13:53:32.6042159+02:00","log.level":"Info","message":"Hello world","ecs":{"version":"1.5.0"},"event":{"severity":6,"timezone":"Mitteleurop\u00E4ische Zeit","created":"2021-03-31T13:53:32.6042159+02:00"},"host":{"name":"*********","ip":["******"]},"log":{"logger":"Logging.Elastic.Demo.NLogTest.Program","original":"Hello world"},"process":{"thread":{"id":*******},"pid":*****,"name":"Logging.Elastic.Demo.NLogTest","executable":"C:\\Source\\Logging.Elastic.Demo\\Logging.Elastic.Demo.NLogTest\\bin\\Debug\\Logging.Elastic.Demo.NLogTest.exe","title":"C:\\Source\\Logging.Elastic.Demo\\Logging.Elastic.Demo.NLogTest\\bin\\Debug\\Fronius.Logging.Elastic.Demo.NLogTest.exe"},"server":{"user":{"name":"********"}}}

The NLOG-Logfile shows the following Error/Exception:

2021-03-31 13:53:32.8682 Error File Target[logEcs]: Error has been raised. Exception: System.NullReferenceException: Der Objektverweis wurde nicht auf eine Objektinstanz festgelegt.
   bei Elastic.CommonSchema.NLog.EcsLayout.CatchError(Exception error) in /home/mpdreamz/projects/elastic/ecs-dotnet/src/Elastic.CommonSchema.NLog/EcsLayout.cs:Zeile 191.
   bei Elastic.CommonSchema.NLog.EcsLayout.GetError(Exception exception) in /home/mpdreamz/projects/elastic/ecs-dotnet/src/Elastic.CommonSchema.NLog/EcsLayout.cs:Zeile 168.
   bei Elastic.CommonSchema.NLog.EcsLayout.RenderFormattedMessage(LogEventInfo logEvent, StringBuilder target) in /home/mpdreamz/projects/elastic/ecs-dotnet/src/Elastic.CommonSchema.NLog/EcsLayout.cs:Zeile 122.
   bei NLog.Layouts.Layout.RenderAppendBuilder(LogEventInfo logEvent, StringBuilder target, Boolean cacheLayoutResult)
   bei NLog.Targets.FileTarget.RenderFormattedMessage(LogEventInfo logEvent, StringBuilder target)
   bei NLog.Targets.FileTarget.RenderFormattedMessageToStream(LogEventInfo logEvent, StringBuilder formatBuilder, Char[] transformBuffer, MemoryStream streamTarget)
   bei NLog.Targets.FileTarget.Write(LogEventInfo logEvent)
   bei NLog.Targets.Target.Write(AsyncLogEventInfo logEvent)

Is your feature request related to a problem? Please describe.
Expose an option to chose the casing of the custom fields. Currently, all custom fields that are injected into the logline are in the snake case. [DataMember(Name="")] attribute gets ignored if I for example define the name in camel case.

Describe the solution you'd like
In the configuration to configure the desired casing for custom fields.

Describe alternatives you've considered
I'm not sure if this can be currently done with middleware. If yes, please describe how.

ECS schema version (e.g. 1.4.0):
1.5.0

ECS .NET assembly version (e.g. 1.4.2):
1.5.0

Elasticsearch version (if applicable):
7.2+

.NET framework / OS:
windows dotnetcore 3.1

Description of the problem, including expected versus actual behavior:
When NLog starts up using a default configured EcsLayout, an exception is thrown that ElasticApmTraceId can not be found. This is because ElasticApmTraceId and ElasticApmTransactionId are defined in Package Elastic.Apm.NLog, however Elastic.Apm.NLog is not a required dependency of Elastic.CommonSchema.NLog through NuGet

Steps to reproduce:

1. Add NLog.Targets.ElasticSearch (alpha version, see above) and Elastic.CommonSchema.NLog 1.5.0 to your project.
2. Configure the project using the minimal options:

<extensions>
    <add assembly="NLog.Targets.ElasticSearch"/>
    <add assembly="Elastic.CommonSchema.NLog"/>
</extensions>
<targets>
   <target xsi:type="ElasticSearch" enableJsonLayout="true">
       <layout xsi:type="EcsLayout" />
   </target>
</targets>

3. Try to write a log message that is routed to the elastic target

exception is thrown:

System.TypeInitializationException
  HResult=0x80131534
  Message=The type initializer for 'Program' threw an exception.
  Source=Program
  StackTrace:
   at Program.<Main>d__7.MoveNext() in ....\Program.cs:line 34

  This exception was originally thrown at this call stack:
    [External Code]

Inner Exception 1:
NLogConfigurationException: Exception when parsing ....\bin\Debug\netcoreapp3.1\NLog.config. 

Inner Exception 2:
TargetInvocationException: Exception has been thrown by the target of an invocation.

Inner Exception 3:
ArgumentException: LayoutRenderer cannot be found: 'ElasticApmTraceId'

Expected Behavior:
Either the NuGet package should list Elastic.Apm.NLog as a dependency, or the Layout should be configured that it only uses items out of the Elastic.Apm.NLog package when it's available (possibly through reflection or a configuration option)

See: elastic/ecs#813

Hi,

I'm using the NLog ECS Layout (Elastic.CommonSchema.NLog) and want to serialize key-value pairs from the MDLC as top-level fields into my JSON log messages.

I'm running into an issue where the fields are serialized into my log messageswith _metadata prepended to the field name, making them unindexable. (because Elasticsearch does not allow indexing of fields that start with an _)

I would very much appreciate if an optioncould be add to the Elastic.CommonSchema.NLog Layout so that dynamic fields serialized into the log messages from the MDLC could be added without _metadata prepended to the field name.

This feature seems to be somewhat related to #104

Core minimum for release 1.8.0

• Autogenerate from the schema definition.
• Fix up any compatibility issues with clients/usage, e.g. different loggers.

Depending on the extent of changes, for anything new can either be added as part of the main release, or added as subsequent point releases.

• Review schema changes/additions for anything new that is worth adding to Extensions
• Review schema changes/additions for anything new that is worth adding to NLog
• Review schema changes/additions for anything new that is worth adding to Serilog
• Review schema changes/additions for anything new that is worth adding to other libraries

These don't all have to be done at once, or as a block, i.e. can incrementally add to different libraries as time permits.

Serilog stack trace formatting for exceptions in LogEventConverter.cs can cause a NullReferenceException when attempting to access a null stack trace frame of an inner exception. This causes message formatting and sinking to fail entirely here.

ECS .NET assembly version (e.g. 1.4.2):1.5.1
.NET framework / OS: net472, win10, but applies to any compatible Fx or OS version.

Description of the problem, including expected versus actual behavior:
A NullReferenceExeption is internally thrown and swallowed, but is accessible through Serilog's SelfLog debugging tool. Expected: the log event to be formatted with exception and stack trace.
Actual: nothing was logged. Serilog debugger captures the null ref.

Steps to reproduce:

1. Run this code in a simple console app. Uses Serilog console sink.

class Program
    {
        static void Main(string[] args)
        {
            Serilog.Debugging.SelfLog.Enable(msg => Console.WriteLine(msg));

            var logger = new LoggerConfiguration()
                .WriteTo.Console(new EcsTextFormatter())
                .CreateLogger();

            try
            {
                throw new Exception("Error", new Exception());  // This 'inner exception' has no stack frame.
            }
            catch (Exception ex)
            {
                logger.Error(ex, "An error occurred.");
            }

            Console.ReadLine();
        }
    }

Output:

2020-11-04T17:09:55.0003908Z Caught exception while emitting to sink Serilog.Sinks.SystemConsole.ConsoleSink: System.NullReferenceException: Object reference not set to an instance of an object.
   at Elastic.CommonSchema.Serilog.LogEventConverter.CatchErrors(IReadOnlyCollection`1 errors)
   at Elastic.CommonSchema.Serilog.LogEventConverter.GetError(IReadOnlyList`1 exceptions)
   at Elastic.CommonSchema.Serilog.LogEventConverter.ConvertToEcs(LogEvent logEvent, IEcsTextFormatterConfiguration configuration)
   at Elastic.CommonSchema.Serilog.EcsTextFormatter.Format(LogEvent logEvent, TextWriter output)
   at Serilog.Sinks.SystemConsole.ConsoleSink.Emit(LogEvent logEvent)
   at Serilog.Core.Sinks.SafeAggregateSink.Emit(LogEvent logEvent)

Admittedly it doesn't make sense why an inner exception would be initialized in this manner, but I don't think ECS.Serilog should fail to format an ECS message because of it. I discovered this issue due to capturing and logging an error from a 3rd party NuGet package.

Proposal: better null handling in stack trace formatting.

ECS schema version (e.g. 1.4.0):
1.5.0

ECS .NET assembly version (e.g. 1.4.2):
1.5.1

Elasticsearch version (if applicable):

.NET framework / OS:
.NET Core 3.1 / Windows

Description of the problem, including expected versus actual behavior:
When using Elastic.CommonSchema.NLog the logged field process.thread.id contains process id instead of thread id.

Steps to reproduce:

1. Create new project with Elastic.CommonSchema.NLog
2. Log data
3. process.thread.id == process.pid

Consider moving "common" code used by both Serilog and NLog into the Elastic.CommonSchema project, so that they are in sync. It will be interesting to see if this is inherently possible in all cases as the Elastic.CommonSchema project should not contain references to any logging libraries.

For example Serilog CatchErrors includes StackTrace for inner-exceptions But NLog CatchError does not (At the same time you can optimize Serilog to not allocate unnecessary Exception-List)

Elastic.CommonSchema.NLog

How to enrich / output custom ECS properties such as EventId, EventAction, etc. while using NLog?

Unfortunately, in our environment, we usually just use the nlog.config so that we can manipulate the rules and targets per environment.

I really want to place a custom properties / traceable GUID,etc.) in the Event (Hopefully EventId) in order to have it used in Elasticsearch as the DocumentId. We also want to route these properties in a hub.

How can I do that?

Might just be a point of ignorance but...
I noticed that in Elastic.CommonSchema.Serilog, if I remove the project reference to Elastic.CommonSchema and instead reference the nuget package Elastic.CommonSchema 1.2.0.-alpha1, it doesn't compile (the serialize method is missing). Do we know where the discrepancy came from?

Is your feature request related to a problem? Please describe.
Adding a field when using Serilog is difficult.
From @Mpdreamz You can extend the types to inject new properties but I don't think we expose this in a nice way through serilog yet

Describe the solution you'd like
An easy way to add:

• custom fields
• default fields

Describe alternatives you've considered

Additional context
Sometimes I want to add an extra field to a log that should nest under the existing ECS top level fields. For example, if I want to add event.dataset

However, in same cases I would like to configure this on the Logger so it always logs it by default.

ECS version: 1.4.0

Elasticsearch version (if applicable): N/A

.NET framework / OS: netcoreapp3.0 MacOS

Description of the problem, including expected versus actual behavior:

When adding the EcsTextFormatter for dotnet we have noticed it doesn't add newlines after each entry. This causes a few issues. First it is hard to read manually to determine when one log entry starts and another ends. Another issue if there is a few console outputs that isn't part of the logging such as the hosting provider startup that get appended to the end of the json output and not the next line since the ecs doesn't add it.

Looks like the fix would be to change outout.Write to output.WriteLine here: https://github.com/elastic/ecs-dotnet/blob/master/src/Elastic.CommonSchema.Serilog/EcsTextFormatter.cs#L32
Possibly adding a configuration to not break existing.

Steps to reproduce:

1. Follow steps to add ECS Serilog formatter

var logger = new LoggerConfiguration()
                .WriteTo.Console(new EcsTextFormatter())
                .CreateLogger();

Logs are all combined:

{"@timestamp":"2020-02-10T11:28:36.010901-06:00","log.level":"Information","message":"Starting web host","ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:36.010901-06:00"},"log":{"logger":"Elastic.CommonSchema.Serilog","original":null},"process":{"thread":{"id":1},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}WARNING: Only use the Vault LDAP Authentication Method when developing locally.
MyConnectionStringValue
{"@timestamp":"2020-02-10T11:28:37.292461-06:00","log.level":"Information","message":"User profile is available. Using '\"/Users/mitch.maler/.aspnet/DataProtection-Keys\"' as key repository; keys will not be encrypted at rest.","_metadata":{"full_name":"/Users/mitch.maler/.aspnet/DataProtection-Keys"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:37.292461-06:00"},"log":{"logger":"Microsoft.AspNetCore.DataProtection.KeyManagement.XmlKeyManager","original":null},"process":{"thread":{"id":1},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}Hosting environment: Development
Content root path: /Users/mitch.maler/repos/sre-kubernetes-project-examples/aspnetcore-service-sample/Demo.Sample.WebApi
Now listening on: http://localhost:5000
Application started. Press Ctrl+C to shut down.
{"@timestamp":"2020-02-10T11:28:37.554143-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MR:00000001","request_path":"/","span_id":"|fddaab66-424572f7a43c6ff4.","trace_id":"fddaab66-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MR"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:37.554143-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":8},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:37.621892-06:00","log.level":"Information","message":"Request finished in 70.4575ms 404 ","_metadata":{"elapsed_milliseconds":70.4575,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 70.4575ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MR:00000001","request_path":"/","span_id":"|fddaab66-424572f7a43c6ff4.","trace_id":"fddaab66-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MR"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:37.621892-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":8},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:37.926152-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MS:00000001","request_path":"/","span_id":"|fddaab67-424572f7a43c6ff4.","trace_id":"fddaab67-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MS"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:37.926152-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:37.928062-06:00","log.level":"Information","message":"Request finished in 1.9781ms 404 ","_metadata":{"elapsed_milliseconds":1.9781,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 1.9781ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MS:00000001","request_path":"/","span_id":"|fddaab67-424572f7a43c6ff4.","trace_id":"fddaab67-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MS"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:37.928062-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.297684-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MT:00000001","request_path":"/","span_id":"|fddaab68-424572f7a43c6ff4.","trace_id":"fddaab68-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MT"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.297684-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":10},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.301349-06:00","log.level":"Information","message":"Request finished in 3.2831ms 404 ","_metadata":{"elapsed_milliseconds":3.2831,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 3.2831ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MT:00000001","request_path":"/","span_id":"|fddaab68-424572f7a43c6ff4.","trace_id":"fddaab68-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MT"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.301349-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":10},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.664058-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MU:00000001","request_path":"/","span_id":"|fddaab69-424572f7a43c6ff4.","trace_id":"fddaab69-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MU"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.664058-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":11},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.665668-06:00","log.level":"Information","message":"Request finished in 1.295ms 404 ","_metadata":{"elapsed_milliseconds":1.295,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 1.295ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MU:00000001","request_path":"/","span_id":"|fddaab69-424572f7a43c6ff4.","trace_id":"fddaab69-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MU"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.665668-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":11},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.985182-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MV:00000001","request_path":"/","span_id":"|fddaab6a-424572f7a43c6ff4.","trace_id":"fddaab6a-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MV"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.985182-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:38.986888-06:00","log.level":"Information","message":"Request finished in 1.427ms 404 ","_metadata":{"elapsed_milliseconds":1.427,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 1.427ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0MV:00000001","request_path":"/","span_id":"|fddaab6a-424572f7a43c6ff4.","trace_id":"fddaab6a-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0MV"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:38.986888-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:39.342789-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N0:00000001","request_path":"/","span_id":"|fddaab6b-424572f7a43c6ff4.","trace_id":"fddaab6b-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N0"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:39.342789-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":11},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:39.344394-06:00","log.level":"Information","message":"Request finished in 1.365ms 404 ","_metadata":{"elapsed_milliseconds":1.365,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 1.365ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N0:00000001","request_path":"/","span_id":"|fddaab6b-424572f7a43c6ff4.","trace_id":"fddaab6b-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N0"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:39.344394-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":11},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:39.710219-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N1:00000001","request_path":"/","span_id":"|fddaab6c-424572f7a43c6ff4.","trace_id":"fddaab6c-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N1"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:39.710219-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":8},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:39.711554-06:00","log.level":"Information","message":"Request finished in 0.9139ms 404 ","_metadata":{"elapsed_milliseconds":0.9139,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 0.9139ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N1:00000001","request_path":"/","span_id":"|fddaab6c-424572f7a43c6ff4.","trace_id":"fddaab6c-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N1"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:39.711554-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":8},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:40.001856-06:00","log.level":"Information","message":"Request starting HTTP/1.1 GET http://localhost:5000/  ","_metadata":{"protocol":"HTTP/1.1","method":"GET","content_type":null,"content_length":null,"scheme":"http","path_base":"","path":"/","query_string":"","hosting_request_starting_log":"Request starting HTTP/1.1 GET http://localhost:5000/  ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N2:00000001","request_path":"/","span_id":"|fddaab6d-424572f7a43c6ff4.","trace_id":"fddaab6d-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N2"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:40.001856-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}{"@timestamp":"2020-02-10T11:28:40.002983-06:00","log.level":"Information","message":"Request finished in 0.8796ms 404 ","_metadata":{"elapsed_milliseconds":0.8796,"status_code":404,"content_type":null,"hosting_request_finished_log":"Request finished in 0.8796ms 404 ","event_id":{"properties":[{"name":"Id","value":{}}]},"request_id":"0HLTE9QKSD0N2:00000001","request_path":"/","span_id":"|fddaab6d-424572f7a43c6ff4.","trace_id":"fddaab6d-424572f7a43c6ff4","parent_id":"","connection_id":"0HLTE9QKSD0N2"},"ecs":{"version":"1.4.0"},"event":{"severity":2,"timezone":"Central Standard Time","created":"2020-02-10T11:28:40.002983-06:00"},"log":{"logger":"Microsoft.AspNetCore.Hosting.Diagnostics","original":null},"process":{"thread":{"id":6},"pid":79132,"name":"Demo.Sample.W","executable":"Demo.Sample.W","title":""}}

In order to fix #58 I opened elastic/apm-agent-dotnet#753 in the .NET APM Agent repo.

That'll fix the issue, but we also have to make sure that Elastic.Apm.SerilogEnricher forces users to use the latest agent, otherwise they'll run into the same bug.

So, once the next APM .NET Agent version is out which contains elastic/apm-agent-dotnet#753, we need to update Elastic.Apm.SerilogEnricher.csproj and make sure Elastic.Apm.SerilogEnricher depends on the version with the fix.

ECS version: 1.4.0 (assembly version: 1.4.0-beta1)

.NET framework / OS: .Net Core 2.2 / Windows

Description of the problem, including expected versus actual behavior:

Actual:

	"log": {
		"logger": "\u0022MyAssembly.MyLogger\u0022",
		"original": "\u0022MyAssembly.MyLogger\u0022"
	},

Steps to reproduce:

var logger = new LoggerConfiguration()
  .WriteTo.RollingFile(new EcsTextFormatter(), "Logs/{Date}.log")
  .CreateLogger();

Seems like bug in LogJsonConverter.Write:

writer.WriteString("logger", value.Logger);
writer.WriteString("original", value.Logger);

When writing the formatted json to a Serilog Durable Http Sink (Serilog.Sinks.Http), the log is corrupted as a BOM is written after the first log rather than as the first bytes of the file. The Http Sink in turn uses a file sink to store the buffer before shipping to http endpoint (Logstash in my case). Logstash cannot process the logs due to an embedded BOM.

The cause of the issue comes down to the EcsTextFormatter writing directly to the base stream rather than the buffer. When the Serilog File Sink flushes the buffer for the first time, the BOM is written after the first log rather than being the first bytes.

Please see Issue raised on Serilog project where this was discussed, prior to coming to conclusion that EcsTextFormatter should be modified.

To prove this, changing the code to below results in the correct sequence of writes:
output.Write(ecsEvent.Serialize());

Hi 👋 ,

Just wanted to try out what you have been cooking lately - I've created sample console app and tried to use ElasticsearchBenchmarkExporter exporter with sample Md5VsSha256 benchmark. After running the benchmark I got following exception

/usr/local/share/dotnet/dotnet /Users/robertlyson/projects/BenchmarkDotNetExporter/Exporter/bin/Debug/netcoreapp3.0/Exporter.dll
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'. The system cannot find the file specified.

File name: 'Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'
   at Elastic.CommonSchema.BenchmarkDotNetExporter.ElasticsearchBenchmarkExporter..ctor(ElasticsearchBenchmarkExporterOptions options)
   at Exporter.Program.Main(String[] args) in /Users/robertlyson/projects/BenchmarkDotNetExporter/Exporter/Program.cs:line 27



Process finished with exit code 6.

Installing Elasticsearch.Net 7.x into the test project resolves the issue. Looks like Elastic.CommonSchema.BenchmarkDotNetExporter package is missing dependency to Elasticsearch.Net?

ECS version: 1.4.0

Elasticsearch version (if applicable): 7.5.2

.NET framework / OS: .net core 3.0 / macOS

Description of the problem, including expected versus actual behavior:

Steps to reproduce:

1. Create new .net core console project
2. Install Elastic.CommonSchema.BenchmarkDotNetExporter version 1.4.0
3. Try to run this benchmark

var options = new ElasticsearchBenchmarkExporterOptions("http://localhost:9200")
{
    GitBranch = "externally-provided-branch",
    GitCommitMessage = "externally provided git commit message",
    GitRepositoryIdentifier = "repository"
};
var exporter = new ElasticsearchBenchmarkExporter(options);

BenchmarkRunner.Run(typeof(Md5VsSha256), DefaultConfig.Instance.With(exporter));

Fails with

/usr/local/share/dotnet/dotnet /Users/robertlyson/projects/BenchmarkDotNetExporter/Exporter/bin/Debug/netcoreapp3.0/Exporter.dll
Unhandled exception. System.IO.FileNotFoundException: Could not load file or assembly 'Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'. The system cannot find the file specified.

File name: 'Elasticsearch.Net, Version=7.0.0.0, Culture=neutral, PublicKeyToken=96c599bbe3e70f5d'
   at Elastic.CommonSchema.BenchmarkDotNetExporter.ElasticsearchBenchmarkExporter..ctor(ElasticsearchBenchmarkExporterOptions options)
   at Exporter.Program.Main(String[] args) in /Users/robertlyson/projects/BenchmarkDotNetExporter/Exporter/Program.cs:line 27



Process finished with exit code 6.

Is your feature request related to a problem? Please describe.
I could not find any documentation as to why Serilog and NLog integration requires Http.Abstractions 2.2.0. The latest Serilog (2.9.0) still supports .NET Core 2.1 and seems unusual that the ECS requires 3.0.

Describe the solution you'd like
Target Microsoft.AspNetCore.Http.Abstractions 2.1.1 or greater instead of 2.2.0.

Describe alternatives you've considered
I have back-ported the Serilog package to use Http.Abstractions 2.1.1 with no issues from my initial testing.

Additional context
N/A

ECS schema version (e.g. 1.4.0):
1.5.0

ECS .NET assembly version (e.g. 1.4.2):
1.5.0

Elasticsearch version (if applicable):

.NET framework / OS:
Core 3.1 / Linux (or any)

Description of the problem, including expected versus actual behavior:
The Labels property is defined as IDictionary<string, object>, but should probably be IDictionary<string, string>

In the specification / YAML the type of labels is 'object', but I think that refers to the labels item itself, i.e. the dictionary is the object, not the contents.

The text of the specification defines the values as type 'keyword' (i.e string), and specifically states they should not be nested objects. The yaml also has a different property, object_type: keyword, that I think defines the type of the contents.

Allowing objects is obviously cooler, because there is semantic value in distinguishing between "5" and the number 5, etc, but I think the spec is that the values are supposed to be strings.

Steps to reproduce:

1. Add a labels key/value with a scalar, e.g. int, or nested objects:
   logEvent.Labels.Add("Foo", 5);
   logEvent.Labels.Add("bar", new User() { Name = "waz" });

2. See the JSON in Elastic / Kibana
   "labels": {
   "foo": 6,
   "bar": {
   "name": "waz"
   }
   },

The formatted log contains data within a hard-coded set of fields, which are very useful data points, but leaves a lot of the schema unused, and instead puts log event data into "_metadata". There appears to be no way to move data from the log event to somewhere in the schema. There should be a hook into the formatter such that more properties of the log event can be out of "_metadata" and into the right place in the schema.

Request:
Given the logging user logs properties (e.g. from context), and the user specifies a mapping from the log event to the schema, the formatted text should have the additional schema fields populated.

Pull from Tracing root object

In CatchErrors method of LogConverter for Serilog you are trying to get first frame from the stacktrace of each exception (and also for all inner exceptions).
The problem is that the frame is not tested for null and it will throw an exception when trying to obtain the fileName, method etc.

To reproduce is very simple. In user code (in a controller for instance) just add this line:
throw new ApplicationException("lorem...", new ApplicationException("ipsum"));

Obs.: For the inner exception the stack trace will have no frames as it was issued in the same context like the outer exception and this will cause the LogConverter to fail on CatchErrors methon when treating the inner exceptions

https://github.com/serilog/serilog-settings-configuration

First time post, excuse me if I am not conforming to a format.

I want to use ECSTextFormatter, however I have am loading configuration of Serilog through the appettings.json. Is there any chance of this happening or a way to do this that I am just not seeing/understanding in the documentation?

Thank you in advance.

ECS version: 1.4.0

Elasticsearch version (if applicable): 7.6.0

.NET framework / OS: .NET Core 3.1 / Windows 10 and CentOS (Docker)

Description of the problem, including expected versus actual behavior:
When used in a ASP.NET Core 3.1 MVC/API-project the Elastic.CommonSchema.Serilog.EcsTextFormatter is unable to pickup any info from HttpContext so this data is not added to the JSON-serialized log output. Debugging LogEventConverter indicates that configuration.MapHttpAdapter is null, so it does not try to get any data from HttpContext.

Steps to reproduce:

1. Using Visual Studio 2019 -> File -> New Project -> ASP.NET Core Web Application.
2. Select API template.
3. Dependencises -> Manage NuGet Packages -> Add reference to NuGet packages Serilog.AspNetCore, Serilog.Sinks.Console, Serilog.Settings.Configuration and Elastic.CommonSchema.Serilog
4. Change Startup.cs and Program.cs according to Serilog docs https://github.com/serilog/serilog/wiki/Getting-Started
5. Configure EcsTextFormatter in Program.cs as described in ecs-dotnet docs https://github.com/elastic/ecs-dotnet/tree/master/src/Elastic.CommonSchema.Serilog
6. Run/debug solution using Kestrel (Select WebApplication1, not IIS Express).
7. Confirm that the console output does not contain any of the HttpContext data such as Server, url, UserAgent etc.

Program.cs:

using System;
using System.IO;
using Elastic.CommonSchema.Serilog;
using Microsoft.AspNetCore;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Serilog;

namespace WebApplication1
{
	public class Program
	{
		public static IConfiguration Configuration { get; } = new ConfigurationBuilder()
			.SetBasePath(Directory.GetCurrentDirectory())
			.AddJsonFile("appsettings.json", optional: true, reloadOnChange: true)
			.AddEnvironmentVariables()
			.Build(); 

		public static IWebHost BuildWebHost(string[] args) =>
			WebHost.CreateDefaultBuilder(args)
				.UseStartup<Startup>()
				.UseSerilog((ctx, config) =>
				{
					config.ReadFrom.Configuration(ctx.Configuration);
					config.WriteTo.Console(new EcsTextFormatter());
				})
				.UseKestrel()
				.Build();

		public static void Main(string[] args)
		{
			Log.Logger = new LoggerConfiguration()
				.ReadFrom.Configuration(Configuration)
				.Enrich.WithProperty("App Name", "Serilog Web App Sample")
				.CreateLogger();
			try
			{
				BuildWebHost(args).Run();
				return;
			}
			catch (Exception ex)
			{
				Log.Fatal(ex, "Host terminated unexpectedly");
				return;
			}
			finally
			{
				Log.CloseAndFlush();
			}
		}
	}
}

Startup.cs:

using Microsoft.AspNetCore.Builder;
using Microsoft.AspNetCore.Hosting;
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using Microsoft.Extensions.Hosting;

namespace WebApplication1
{
	public class Startup
	{
		public Startup(IConfiguration configuration)
		{
			Configuration = configuration;
		}

		public IConfiguration Configuration { get; }

		// This method gets called by the runtime. Use this method to add services to the container.
		public void ConfigureServices(IServiceCollection services)
		{
			services.AddControllers();
		}

		// This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
		public void Configure(IApplicationBuilder app, IWebHostEnvironment env)
		{
			if (env.IsDevelopment())
			{
				app.UseDeveloperExceptionPage();
			}

			app.UseRouting();

			app.UseAuthorization();

			app.UseEndpoints(endpoints =>
			{
				endpoints.MapControllers();
			});
		}
	}
}

Console output:

{
	"@timestamp": "2020-03-11T12:31:42.7001953+01:00",
	"log.level": "Information",
	"message": "Request starting HTTP/1.1 GET http://localhost:5000/weatherforecast  ",
	"_metadata": {
		"message_template": "{HostingRequestStartingLog:l}",
		"protocol": "HTTP/1.1",
		"method": "GET",
		"content_type": null,
		"content_length": null,
		"scheme": "http",
		"path_base": "",
		"path": "/weatherforecast",
		"query_string": "",
		"hosting_request_starting_log": "Request starting HTTP/1.1 GET http://localhost:5000/weatherforecast  ",
		"event_id": {
			"id": 1
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7001953+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Hosting.Diagnostics",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7114543+01:00",
	"log.level": "Information",
	"message": "Executing endpoint '\"WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)\"'",
	"_metadata": {
		"message_template": "Executing endpoint '{EndpointName}'",
		"endpoint_name": "WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)",
		"event_id": {
			"name": "ExecutingEndpoint"
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7114543+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Routing.EndpointMiddleware",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7244166+01:00",
	"log.level": "Information",
	"message": "Route matched with \"{action = \\\"Get\\\", controller = \\\"WeatherForecast\\\"}\". Executing controller action with signature \"System.Collections.Generic.IEnumerable`1[WebApplication3.WeatherForecast] Get()\" on controller \"WebApplication3.Controllers.WeatherForecastController\" (\"WebApplication3\").",
	"_metadata": {
		"message_template": "Route matched with {RouteData}. Executing controller action with signature {MethodInfo} on controller {Controller} ({AssemblyName}).",
		"route_data": "{action = \"Get\", controller = \"WeatherForecast\"}",
		"method_info": "System.Collections.Generic.IEnumerable`1[WebApplication3.WeatherForecast] Get()",
		"controller": "WebApplication3.Controllers.WeatherForecastController",
		"assembly_name": "WebApplication3",
		"event_id": {
			"id": 3,
			"name": "ControllerActionExecuting"
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"id": "d756316f-e9e8-45f3-bbea-9b2fe6986f5a",
		"action": "WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)",
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7244166+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7361828+01:00",
	"log.level": "Information",
	"message": "Executing ObjectResult, writing value of type '\"WebApplication3.WeatherForecast[]\"'.",
	"_metadata": {
		"message_template": "Executing ObjectResult, writing value of type '{Type}'.",
		"type": "WebApplication3.WeatherForecast[]",
		"event_id": {
			"id": 1,
			"name": "ObjectResultExecuting"
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"id": "d756316f-e9e8-45f3-bbea-9b2fe6986f5a",
		"action": "WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)",
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7361828+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7486952+01:00",
	"log.level": "Information",
	"message": "Executed action \"WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)\" in 13.0702ms",
	"_metadata": {
		"message_template": "Executed action {ActionName} in {ElapsedMilliseconds}ms",
		"elapsed_milliseconds": 13.0702,
		"event_id": {
			"id": 2,
			"name": "ActionExecuted"
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"id": "d756316f-e9e8-45f3-bbea-9b2fe6986f5a",
		"action": "WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)",
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7486952+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7585408+01:00",
	"log.level": "Information",
	"message": "Executed endpoint '\"WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)\"'",
	"_metadata": {
		"message_template": "Executed endpoint '{EndpointName}'",
		"endpoint_name": "WebApplication3.Controllers.WeatherForecastController.Get (WebApplication3)",
		"event_id": {
			"id": 1,
			"name": "ExecutedEndpoint"
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7585408+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Routing.EndpointMiddleware",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}
{
	"@timestamp": "2020-03-11T12:31:42.7686387+01:00",
	"log.level": "Information",
	"message": "Request finished in 68.629ms 200 application/json; charset=utf-8",
	"_metadata": {
		"message_template": "{HostingRequestFinishedLog:l}",
		"elapsed_milliseconds": 68.629,
		"status_code": 200,
		"content_type": "application/json; charset=utf-8",
		"hosting_request_finished_log": "Request finished in 68.629ms 200 application/json; charset=utf-8",
		"event_id": {
			"id": 2
		},
		"request_id": "0HLU5LUMBRSDR:00000003",
		"request_path": "/weatherforecast",
		"span_id": "|a1999d5b-40d6b8dab8d21e2c.",
		"trace_id": "a1999d5b-40d6b8dab8d21e2c",
		"parent_id": "",
		"connection_id": "0HLU5LUMBRSDR"
	},
	"ecs": {
		"version": "1.4.0"
	},
	"event": {
		"severity": 2,
		"timezone": "W. Europe Standard Time",
		"created": "2020-03-11T12:31:42.7686387+01:00"
	},
	"log": {
		"logger": "Microsoft.AspNetCore.Hosting.Diagnostics",
		"original": null
	},
	"process": {
		"thread": {
			"id": 13
		},
		"pid": 15588,
		"name": "WebApplication3",
		"executable": "WebApplication3",
		"title": "WebApplication3"
	}
}

So, I'm confused as to how this is supposed to work. One possible issue I see is that my .csproj file has netcoreapp3.1 so the conditional logic for getting HttpContext/HttpContextAccessor using #if NETSTANDARD does not kick in, but even when I rewrite the code to only support ".NET Core mode" it is still not able to get a reference to IHttpContextAccessor.

I have custom Serilog enrichers which use IHttpContextAccessor to get data from HttpContext, and they work fine, but I guess they use DI which does not seem to work for EcsTextFormatterConfiguration. Am I supposed to manually get IHttpContextAccessor from my DI container, and then pass it to MapHttpContext on the configuration objext? If so, then I guess configuring this with appsettings.json is out of the question? Maybe I'm just not understanding how this is supposed to work...

Core minimum for release 1.6.0

• Autogenerate from the schema definition.
• Fix up any compatibility issues with clients/usage, e.g. different loggers.

Depending on the extent of changes, for anything new can either be added as part of the main release, or added as subsequent point releases.

• Review schema changes/additions for anything new that is worth adding to Extensions
• Review schema changes/additions for anything new that is worth adding to NLog
• Review schema changes/additions for anything new that is worth adding to Serilog
• Review schema changes/additions for anything new that is worth adding to other libraries

These don't all have to be done at once, or as a block, i.e. can incrementally add to different libraries as time permits.

Note that a lot of new fields may not be relevant for libraries, e.g. the new x509.* fields may have no directly relevant default values (like many other fields).

Call Stack:

   at System.Object.GetType()
   at Elastic.CommonSchema.Serialization.MetaDataDictionaryConverter.Write(Utf8JsonWriter writer, IDictionary`2 value, JsonSerializerOptions options)
   at Elastic.CommonSchema.Serialization.BaseJsonConverter.Write(Utf8JsonWriter writer, Base value, JsonSerializerOptions options)
   at System.Text.Json.JsonPropertyInfoNotNullable`4.OnWrite(WriteStackFrame& current, Utf8JsonWriter writer)
   at System.Text.Json.JsonPropertyInfo.Write(WriteStack& state, Utf8JsonWriter writer)
   at System.Text.Json.JsonSerializer.Write(Utf8JsonWriter writer, Int32 originalWriterDepth, Int32 flushThreshold, JsonSerializerOptions options, WriteStack& state)
   at System.Text.Json.JsonSerializer.WriteCore(Utf8JsonWriter writer, Object value, Type type, JsonSerializerOptions options)
   at Elastic.CommonSchema.Base.Serialize(Stream s)
   at Serilog.Sinks.File.FileSink.Emit(LogEvent logEvent)
   at Serilog.Sinks.RollingFile.RollingFileSink.Emit(LogEvent logEvent)
   at Serilog.Core.Sinks.SafeAggregateSink.Emit(LogEvent logEvent)

LogEvent:

Output:

...
	"_metadata": {
		"protocol": "HTTP/1.1",
		"method": "GET",
		"content_type": 

ECS version: 1.4.0 (assembly version: 1.4.0-beta1)

.NET framework / OS: .Net Core 2.2 / Windows

Description of the problem, including expected versus actual behavior:
Non ASCII symbols got escaped, users can't read logs.

Actual (as WriteTo.File):

...
	"message": "\u042F\u0437\u044B\u043A: { \u0417\u043D\u0430\u0447\u0435\u043D\u0438\u0435: \u0022\u0420\u0443\u0441\u0441\u043A\u0438\u0439\u0022 }",
	"_metadata": {
		"data": {
			"properties": [{
				"name": "\u0417\u043D\u0430\u0447\u0435\u043D\u0438\u0435",
				"value": {
					
				}
			}]
		}
	},
...
	"event": {
		"severity": 2,
		"timezone": "RTZ 3 (\u0437\u0438\u043C\u0430)",
		"created": "2020-01-14T10:42:16.8076199+04:00"
	},
...

Expected (as WriteTo.Console):

...
	"message": "Язык: { Значение: \"Русский\" }",
	"_metadata": {
		"data": {
			"properties": [{
				"name": "Значение",
				"value": {
					
				}
			}]
		}
	},
...
	"event": {
		"severity": 2,
		"timezone": "RTZ 3 (зима)",
		"created": "2020-01-14T10:44:23.8543233+04:00"
	},
...

Steps to reproduce:

      Log.Logger = new LoggerConfiguration()
        .WriteTo.Console(new EcsTextFormatter())
        .WriteTo.File(new EcsTextFormatter(), "d:\\log.log")
        .CreateLogger();

      Log.Logger.Information("Язык: {@Data}", new { Значение = "Русский" });

Cause
Seems like Base.Serialize should create Utf8JsonWriter with new JsonWriterOptions {Encoder = JavaScriptEncoder.UnsafeRelaxedJsonEscaping }.
Now JavaScriptEncoder.UnsafeRelaxedJsonEscaping only used in JsonSerializer.Serialize.

You should change this method:

		private static Log GetLog(LogEventInfo logEventInfo)
		{
			var log = new Log
			{
				Level = logEventInfo.Level.ToString(),
				Logger = SpecialKeys.DefaultLogger,
				Original = logEventInfo.Message
			};

			return log;
		}

Into this (Use logEventInfo.Logger instead of SpecialKeys.DefaultLogger):

		private static Log GetLog(LogEventInfo logEventInfo)
		{
			var log = new Log
			{
				Level = logEventInfo.Level.ToString(),
				Logger = logEventInfo.Logger,
				Original = logEventInfo.Message
			};

			return log;
		}

This will match the use of sourceContext for Serilog.

I think the generated classes and properties to store trace and transaction id do not match ecs.

What we generate here are:

• tracing.trace.id
• tracing.transaction.id

But they should be:

• trace.id
• transaction.id

See in docs - I'm a little bit confused about this syntax, but e.g. the generated go code does not seem to have the additional nesting we have here (tracing).

I tested with passing EcsTextFormatter to serilog-sinks-elasticsearch.

The test was done with #12.

All works fine, logs show up in Kibana.

Problem is that the trace id is stored in tracing.trace.id, so the "View in APM" button does not work. - I assume it needs a trace.id field.

Another reason I think the additional nesting is not needed is because Kibana searches for trace.id and not transaction.trace.id when you jump from a trace to logs - the query itself works, because of the OR you see in the statement, so that makes the "APM -> Log" jumping work, but for the other way I guess we need trace.id.

ECS schema version (e.g. 1.4.0): 1.5.0

ECS .NET assembly version (e.g. 1.4.2): 1.5.1 (Elastic.CommonSchema.Serilog)

Elasticsearch version (if applicable): 7.4.0

.NET framework / OS: .NET Core 3.1 / macOS

Description of the problem, including expected versus actual behavior:

When a request event contains a host name, it is used to override the server IP. The field server.ip is then invalid according to the ECS schema and the event cannot be indexed into Elasticsearch ("failed to parse field [server.ip] of type [ip]"). Since the field contained a valid IP before being overridden (at least with .NET Core) the expected behavior would be not to override it.

There might be some use cases I am not seeing here but at least using ASP.NET Core this cannot seem to work. The IP is overridden here: https://github.com/elastic/ecs-dotnet/blob/master/src/Elastic.CommonSchema.Serilog/LogEventConverter.cs#L195 with a value that comes from here: https://github.com/dotnet/aspnetcore/blob/5b2f3fb5f7f24ac3e91c5150a55cc411b2b36b76/src/Hosting/Hosting/src/Internal/HostingRequestStartingLog.cs#L33. For .NET Core using a HttpContextAcessor this IP is already set here and the value seems correct: https://github.com/elastic/ecs-dotnet/blob/master/src/Elastic.CommonSchema.Serilog/Http/HttpContextAccessorAdapter.cs#L133. So this might be a fix made for another framework. I don't have visibility on that.

Steps to reproduce:

1. It works on a simple ASP.NET project so do a dotnet new web then add Elastic.CommonSchema.Serilog
2. Setup a EcsTextFormatterConfiguration with MapHttpContext
3. dotnet run
4. curl localhost:5000 => The log line from Microsoft.AspNetCore.Hosting.Diagnostics should contain a server.ip = localhost:5000 instead of an IP address

ECS integration/library project(s) (e.g. Elastic.CommonSchema.Serilog): Elastic.CommonSchema.BenchmarkDotNetExporter
ECS schema version (e.g. 1.4.0): 1.5.1
ECS .NET assembly version (e.g. 1.4.2): 1.5.1
Elasticsearch version (if applicable): ES Cloud 7
.NET framework / OS: netcore 3.1

Description of the problem, including expected versus actual behavior:
Serialization exception after benchmarks have been constructed just before posting to ES

Specifically on this line:

System.TypeInitializationException
HResult=0x80131534
Message=The type initializer for 'Elastic.CommonSchema.Serialization.JsonConfiguration' threw an exception.

Inner Exception 1:
FileNotFoundException: Could not load file or assembly 'System.Text.Encodings.Web, Version=5.0.0.0, Culture=neutral, PublicKeyToken=cc7b13ffcd2ddd51'. The system cannot find the file specified.

Steps to reproduce:

1. Created a simple benchmark and configured ES Options.
2. Include the nuget reference to 1.5.1 BenchmarkDotNetExporter
3. Failure of benchmarks complete and upload to ES attempted

I managed to fix this by cloning the repo and updating all System.Text.Json reference to 5.0.2 and the problem went away. I believe it is just the fact that I referenced from Nuget rather than the version.

The ecs-logging spec defines a service.name property that can be populated with the APM agent service name. ECS .NET APM integrations should populate this value.

Possible Implementation

The APM .NET agent does not currently expose the Service name of the agent instance on the Agent static class. First, we should decide what should be exposed, and how to expose it. For example, it may not make sense to expose the whole Service, but just the name.

I really like Elasticsearch.Extensions.Logging. It is a piece of cake to implement.
However, not all of my apps are logging to Elastic. Some are logging to Splunk. ECS is a superior schema. I would like to utilize it as the common logging output. Splunk should have no problem with it.

How can I just log from .NET logging to a file with ECS format? Can you provide me some guidance here? Most are using NLog.

Could you adjust the ECS version names a little bit? There's two things I see that could be confusing about them, in the readme.

• ECS always explicitly includes all 3 numbers in the version number, including .0 patch. So instructing people to construct the URL with the "1.2" tag will give them a 404. They'll figure it out, but it's an annoyance we can avoid :-)
• Also the actual tag in Git includes a leading v.

So the full 1.2 tag actually looks like v1.2.0 :-)

Loving the readme otherwise, good work!

This ticket is a continuation of the following PR: #85

• Rely on Elasticsearch to generate the id
• Move to a simple buffering approach and _bulk
• Add a hook to be notified of failures
• state always an IEnumerable<KeyValuePair<string, object>>?
• Make sure trace.id flows into Elasticsearch so that Elastic APM can corrolate logs. (done #96 )
• Downgrade Microsoft.Extensions.* packages to 2.1.0.
• Should we default to logstash-[current_date] like the serilog sink does?
• Remove .Cast<char>
• Remove unused _w3CFormat regular expression
• ExtractW3cSpanIdFromActivityId has a possible detection edgecase that needs to be adressed. (#96 eliminated it)
• Introduce integration tests for this project.
• Add support for CloudConnectionPool (#98) ty @btastic !
• Make sure CreateConnectionPool throws instead of returning null

When all that is done we can:

• Remove <IsPackable>false</IsPackable> from the project

That should be enough to create and publish the package when we tag the next version.

cc @russcam @gregkalapos @sgryphon feel free to take an item and submit PR's. If it's a bigger chunk of work leave a note so we don't duplicate efforts.

Logger:

var logger = new LoggerConfiguration()
  .Enrich.WithMachineName()
  .Enrich.WithElasticApmCorrelationInfo()
  .WriteTo.RollingFile(new EcsTextFormatter(), "Logs/{Date}.log")
  .CreateLogger();

Output:

...
	"host": {
		"name": "\u0022mycomputername\u0022"
	},
	"log": {
		"logger": "\u0022Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor\u0022",
		"original": "\u0022Microsoft.AspNetCore.Mvc.Internal.ObjectResultExecutor\u0022"
	},
...

ECS version: 1.4.0

Elasticsearch version (if applicable): 7.5.2

.NET framework / OS: .net core 3.0 / macOS

Description of the problem, including expected versus actual behavior:

Running this sample console app with test benchmark

using System;
using System.Security.Cryptography;
using BenchmarkDotNet.Attributes;
using BenchmarkDotNet.Configs;
using BenchmarkDotNet.Running;
using Elastic.CommonSchema.BenchmarkDotNetExporter;

namespace Exporter
{
    public class Program
    {
        static void Main(string[] args)
        {
            var options = new ElasticsearchBenchmarkExporterOptions("http://localhost:9200")
            {
                GitBranch = "externally-provided-branch",
                GitCommitMessage = "externally provided git commit message",
                GitRepositoryIdentifier = "repository"
            };
            var exporter = new ElasticsearchBenchmarkExporter(options);

            BenchmarkRunner.Run(typeof(Md5VsSha256), DefaultConfig.Instance.With(exporter));
        }

        public class Md5VsSha256
        {
            private SHA256 sha256 = SHA256.Create();
            private MD5 md5 = MD5.Create();
            private byte[] data;

            [Params(1000, 10000)]
            public int N;

            [GlobalSetup]
            public void Setup()
            {
                data = new byte[N];
                new Random(42).NextBytes(data);
            }

            [Benchmark]
            public byte[] Sha256() => sha256.ComputeHash(data);

            [Benchmark]
            public byte[] Md5() => md5.ComputeHash(data);
        }
    }
}

throws NullReferenceException exception

..
// ***** BenchmarkRunner: Finish  *****

// * Export *
  BenchmarkDotNet.Artifacts/results/Exporter.Program.Md5VsSha256-report.csv
  BenchmarkDotNet.Artifacts/results/Exporter.Program.Md5VsSha256-report-github.md
  BenchmarkDotNet.Artifacts/results/Exporter.Program.Md5VsSha256-report.html
Unhandled exception. System.NullReferenceException: Object reference not set to an instance of an object.
   at Elastic.CommonSchema.BenchmarkDotNetExporter.ElasticsearchBenchmarkExporter.<>c__DisplayClass18_0.<CreateBenchmarkDocuments>b__0(BenchmarkReport r)
   at System.Linq.Enumerable.SelectArrayIterator`2.ToList()
   at System.Linq.Enumerable.ToList[TSource](IEnumerable`1 source)
   at Elastic.CommonSchema.BenchmarkDotNetExporter.ElasticsearchBenchmarkExporter.CreateBenchmarkDocuments(Summary summary)
   at Elastic.CommonSchema.BenchmarkDotNetExporter.ElasticsearchBenchmarkExporter.ExportToLog(Summary summary, ILogger logger)
   at BenchmarkDotNet.Exporters.ExporterBase.ExportToFiles(Summary summary, ILogger consoleLogger)
   at BenchmarkDotNet.Exporters.CompositeExporter.<>c__DisplayClass5_0.<ExportToFiles>b__0(IExporter exporter)
   at System.Linq.Enumerable.SelectManySingleSelectorIterator`2.MoveNext()
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.PrintSummary(ILogger logger, ImmutableConfig config, Summary summary)
   at BenchmarkDotNet.Running.BenchmarkRunnerClean.Run(BenchmarkRunInfo[] benchmarkRunInfos)
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithDirtyAssemblyResolveHelper(Type type, IConfig config)
   at BenchmarkDotNet.Running.BenchmarkRunner.<>c__DisplayClass1_0.<Run>b__0()
   at BenchmarkDotNet.Running.BenchmarkRunner.RunWithExceptionHandling(Func`1 run)
   at BenchmarkDotNet.Running.BenchmarkRunner.Run(Type type, IConfig config)
   at Exporter.Program.Main(String[] args) in /Users/robertlyson/projects/BenchmarkDotNetExporter/Exporter/Program.cs:line 22

Process finished with exit code 6.

It's happening because of incorrect benchmark configuration (no job specified).

Core minimum for release 1.7.0

• Autogenerate from the schema definition.
• Fix up any compatibility issues with clients/usage, e.g. different loggers.

Depending on the extent of changes, for anything new can either be added as part of the main release, or added as subsequent point releases.

• Review schema changes/additions for anything new that is worth adding to Extensions
• Review schema changes/additions for anything new that is worth adding to NLog
• Review schema changes/additions for anything new that is worth adding to Serilog
• Review schema changes/additions for anything new that is worth adding to other libraries

These don't all have to be done at once, or as a block, i.e. can incrementally add to different libraries as time permits.

Note that a lot of new fields may not be relevant for libraries, e.g. the new threat.technique.subtechnique field2 may have no directly relevant default values (like many other fields).