urielha / log4stash Goto Github PK

Hi,

I'm using log4stash to write logs to elasticsearch, i have daily indices (each one with multiple shards).
I want to use routing to write to a specific shard but i didn't find a way to do it with log4stash.
Is it possible to do it with log4stash ?

From @ITFactoryAUT on January 16, 2015 9:40

Hello,
is there any documentation available?
I want to use logstash for centralizing the logs of all my application, but I have no idea how to connect the server through log4stash to my programs.
greetings

Copied from original issue: urielha/log4stash-old#3

Hi

Thanks for creating this awesome framework

The integration with the local Docker EK stack went smooth, but when I trying to connect to the AWS Kibana it get crashed with the aforementioned error and the stack trace is as the following:

System.NullReferenceException: Object reference not set to an instance of an object.
   at log4stash.WebElasticClient.CheckResponse(IRestResponse response) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\WebElasticClient.cs:line 203
   at log4stash.WebElasticClient.SafeSendRequestAsync(RequestDetails request) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\WebElasticClient.cs:line 165

My conf is

<log4net>
  <appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
    <Server>******es.amazonaws.com</Server>
    <Port>9200</Port>
    <Ssl>true</Ssl>
    <IndexName>log_test_%{+yyyy-MM-dd}</IndexName>
    <IndexType>LogEvent</IndexType>

    <AuthenticationMethod>
      <!--For AWS ElasticSearch service-->
      <Aws>
        <Aws4SignerSecretKey>***</Aws4SignerSecretKey>
        <Aws4SignerAccessKey>***</Aws4SignerAccessKey>
        <Aws4SignerRegion>***</Aws4SignerRegion>
      </Aws>
    </AuthenticationMethod>


    <ElasticFilters>
      <!-- example of using filter with default parameters -->
      <Json>
        <SourceKey>JsonRaw</SourceKey>
        <FlattenJson>false</FlattenJson>
        <!-- the separator property is only relevant when setting the FlattenJson property to 'true' -->
        <Separator>_</Separator>
      </Json>
    </ElasticFilters>
  </appender>
  <root>
    <level value="ALL" />
    <appender-ref ref="ElasticSearchAppender" />
  </root>
</log4net>

Any ideas?

Hi,

Just wanna ask regarding the integration with .NET projects, is it the same on how log4net.ElasticSearch is implemented?

Thank you :)

I have the exact same problem as described in issue #4
Elastic version: 5.6.10
Log4net version: 2.0.8
Log4stash version: 3.0.0

Hello,
I have this problem during logging:

log4net:ERROR Invalid connection to ElasticSearch
System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at log4stash.WebElasticClient.SendRequest(WebRequest webRequest, String requestString) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 152
   at log4stash.WebElasticClient.PrepareBulkAndSend(IEnumerable`1 bulk) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 129
   at log4stash.WebElasticClient.IndexBulk(IEnumerable`1 bulk) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 98
   at log4stash.ElasticSearchAppender.DoIndexNow() in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticSearchAppender.cs:line 175

I tried update for last 2.0.2-c but same issue, only line numbers changed:

log4net:ERROR Invalid request to ElasticSearch
System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
   at System.Net.HttpWebRequest.GetRequestStream()
   at log4stash.WebElasticClient.SafeSendRequest(RequestDetails request, Func`1 getRequestStream) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 169

Could you please help?

First off, thanks for this library so much it's very much appreciated. Getting this error intermittently when submitting logs to elk. TLDR is im getting timeouts and the error getting back from log4stash is misleading. It looks like this is fixed in master but master isn't pushed out to nuget?

System.NullReferenceException: Object reference not set to an instance of an object.
   at log4stash.WebElasticClient.CheckResponse(IRestResponse response) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\WebElasticClient.cs:line 206
   at log4stash.WebElasticClient.<SafeSendRequestAsync>d__d.MoveNext() in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClient\WebElasticClient.cs:line 165

I went ahead and downloaded source and attached a debugger and this is what is actually coming through is the response is timing out per the rest client but the 2.2.1 code isn't checking for a null jsonResponse. It can't deserialize the timeout so that's null. It looks like this is fixed in the master branch. Any chance of getting a push of what's in master out to nuget?

I think what's happening is something with my corporate proxy causing requests to go through but not.. responses back. I'm investigating it but in case someone else runs into the issue check your server and make sure its sending a response back - the 2.2.1 code doesn't handle error (or timeouts) properly.

      <appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
        <Server>someserverhere</Server>
        <Port>5001</Port>
        <ElasticSearchTimeout>30000</ElasticSearchTimeout>
        <ElasticFilters>
          <Filter type="log4stash.Filters.RenameKeyFilter, log4stash">
            <Key>Message</Key>
            <RenameTo>message</RenameTo>
          </Filter>
        </ElasticFilters>
      </appender>

From @yosiat on October 19, 2015 18:55

Hi,

I have read the code of ElasticClient, and I have some tips to improve performance of indexing:

• You are using .NET JavascriptSerializer, which is horribly slow and you are recreating it's instance every serialize (which clears it's internal cache), I recommend using Jil
• Instead of using WebRequest (and it's ugly Begin/End async pattern), you can use HttpClient and execute PutAsync/PostAsync - from my tests, It handles concurrency pretty well.

I will submit a pull request on the weekend If I will have time.

Copied from original issue: urielha/log4stash-old#9

In ElasticSearch 6.0 RC 2 this error occurs:

Maybe it's a similar problem like this (log4net.ElasticSearch):

bruno-garcia/log4net.ElasticSearch#86

Thanks.

Best regards
Martin

Is there a way to make the add or remove properties from the logEvent,
I would want remove unnecessary properies like AppDomain, and move all the properies of "MessageObject", outside of the warpping
{
"_index": "slog_20171218",
"_type": "LogEvent",
"_id": "v_cNaWABDFhcUDArB-sp",
"_score": 1,
"_source": {
"@timestamp": "2017-12-18T09:55:49.0362522Z",
"LoggerName": "Log4NETTest.Program",
"HostName": "ADIPTOP",
"ThreadName": "1",
"AppDomain": "Log4NETTest.exe",
"Level": "INFO",
"Message": "Log4NETTest.LogObject",
"MessageObject": {
"RawReading": null,
"Severity": null,
"Message": null,
"CorrelationId": null,
"TimeStamp": "0001-01-01T00:00:00",
"BoltName": null,
"InternalTimestamp": "0001-01-01T00:00:00",
"ReadingTimestamp": "0001-01-01T00:00:00",
"AppId": null,
"DeviceId": null,
"ProductId": null,
"Unit": null,
"TypeCode": 0,
"Name": "something new",
"Value": null
},
"log4net:Identity": "",
"log4net:UserName": "ADIPTOP\Adi",
"log4net:HostName": "Adiptop"
}
},

From @Cylindric on January 13, 2016 14:47

It would be nice if it was possible to define multiple ES nodes in the config, as a means of targeting a multi-node cluster to handle a single node failure.

Copied from original issue: urielha/log4stash-old#13

Remove (Begin/End)GetRequest async mechanism

This caused too much problems (Requests don't getting closed properly, exceptions of all kinds and more..).

• Using async await is cool but I want to preserve backward compatibility with old versions of .net - I didn't check it, so if async await approach is working also for the older .net projects I can consider this.

• Alternatively, I thought about opening configurable number of worker threads that will take a bulk request object and handle the whole communication with the Elasticsearch server synchronously.
  They all will read the requests from a queue with a configurable limit (when full all new requests will be lost into the wild universe).
  This way we can handle more easily a connection loss (I think).

May the odds be ever in the requests' favor.

Hello,
I have decided to use a template to display the specific fields that I need. I have a configuration for Log4net file and I wanted to send the same fields to Elasticsearch. In the fileappender I have something like

<layout type="log4net.Layout.PatternLayout"> <conversionPattern value="%date{ISO8601} [%-5level] [%2thread] [%property{CorrelationId}] [%logger{1}] - %message%newline%exception"/> </layout>

I want all of these fields and I added a template that will only have these properties. The template is created as well as the index but no data is added.

I tried the following:

1. Add

<IndexOperationParams>
		<Parameter>
			<Key>_id</Key>
			<Value>%{IdSource}</Value>
		</Parameter>
 this one gives nasty errors (log4net:ERROR XmlHierarchyConfigurator: Cannot find Property [Parameter] to set object on [log4stash.Configuration.IndexOperationParamsDictionary])

2. Add the values in ElasticFilters as key value pairs in the add tag
3. Add the pattern showed above

None worked I am a little bit lost as what to do in order to have the fields in my template shown and have my data in elasticsearch.

Thank you for your help @urielha .

or ES appender only?

I use log4net in my WCF service.
Should I replace it with the logstash or to add the logstash to the references?

can add an option (or maybe keynames) to preserve array when flattenjson is true.. i prefer the format the "flattenjson" option gives on the document for my object, except on array elements.

log4stash version is 3.0.0. Logging stopped working after upgrading log4net from 2.0.8 to 2.0.9. The only thing that changed is the version of log4net, everything else stayed the same. I enabled log4net debug output and I think it fails to load the appended because it is targeting 2.0.8 (as opposed to targeting 2.0.8 or higher):

log4net:ERROR Could not create Appender [ElasticSearchAppender] of type [log4stash.ElasticSearchAppender, log4stash]. Reported error follows.
System.IO.FileLoadException: Could not load file or assembly 'log4net, Version=2.0.8.0, Culture=neutral, PublicKeyToken=669e0ddf0bb1aa2a' or one of its dependencies. The located assembly's manifest definition does not match the assembly reference. (Exception from HRESULT: 0x80131040)
File name: 'log4net, Version=2.0.8.0, Culture=neutral, PublicKeyToken=669e0ddf0bb1aa2a'
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMarkHandle stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName, ObjectHandleOnStack type)
   at System.RuntimeTypeHandle.GetTypeByName(String name, Boolean throwOnError, Boolean ignoreCase, Boolean reflectionOnly, StackCrawlMark& stackMark, IntPtr pPrivHostBinder, Boolean loadTypeFromPartialName)
   at System.Type.GetType(String typeName, Boolean throwOnError, Boolean ignoreCase)
   at log4net.Util.SystemInfo.GetTypeFromString(Assembly relativeAssembly, String typeName, Boolean throwOnError, Boolean ignoreCase)
   at log4net.Repository.Hierarchy.XmlHierarchyConfigurator.ParseAppender(XmlElement appenderElement)

WRN: Assembly binding logging is turned OFF.
To enable assembly bind failure logging, set the registry value [HKLM\Software\Microsoft\Fusion!EnableLog] (DWORD) to 1.
Note: There is some performance penalty associated with assembly bind failure logging.
To turn this feature off, remove the registry value [HKLM\Software\Microsoft\Fusion!EnableLog].

log4net:ERROR Appender named [ElasticSearchAppender] not found.

Is it possible to hook-in custom authentication method ? In our ELK setup for authentication we have to set some custom header, so would need to write a custom authentication handler.

Hi,

I have to move my applications from an 2.x Elasticsearch to a new 7.6 Elasticsearch.
The new server only allows ApiKey based authentication.

I've seen the issue #44 where you explain how to implement my own authenticator. I'm using the latest Version of log4stash and cannot find any interface like log4stash.Authentication.IAuthenticationMethod. The two existing authentication classes direves directly from IAuthenticator, an implemention of ReshSharp.

I followed the pattern as in #44 but I have no idea whether my filter is found by log4stash nor whether it's treated as valid, ... The only exception I get is 401 (Forbidden). But this error occures anyway.

This is my configuration:

    <log4net>
        <appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash" >
            <Server>hostname</Server>
            <Port>443</Port>
            <Ssl>true</Ssl>
            <AuthenticationMethod>
                <Filter type="company.departement.Package.Framework.Log.Log4StashApiKeyAuthenticator, company.departement.Package.Framework.Log">
                    <ApiKey>abc123</ApiKey>
                </Filter>
            </AuthenticationMethod>
            <IndexName>application-logstash-%{+yyyy.MM.dd}</IndexName>
            <IndexType>LogEvent</IndexType>
            <Bulksize>100</Bulksize>
            <BulkIdleTimeout>15</BulkIdleTimeout>
            <IndexAsync>True</IndexAsync>
            <ElasticFilters>
            </ElasticFilters>
            <layout type="log4net.Layout.PatternLayout">
                <conversionPattern value="%date [%thread] %-5level %logger [%property{NDC}] - %message%newline" />
            </layout>
        </appender>
        <root>
            <appender-ref ref="ElasticSearchAppender" />
        </root>
    </log4net>

And this is my authenticator implementation:

using RestSharp;
using RestSharp.Authenticators;

namespace company.departement.Package.Framework.Log
{
    public class Log4StashApiKeyAuthenticator : IAuthenticator
    {
        public string ApiKey { get; set; }
        
        public void Authenticate(IRestClient client, IRestRequest request)
        {
            request.AddHeader("Authorization", $"ApiKey {ApiKey}");
        }
    }
}

Is this scenario still support? If now, how to proceed with my api key?

Thank you and kind regards,
Danny

Dear urielha,
In version 2.2.0 you added a dependency on RestSharp which is an unsigned dll. This will force strongly signed project to sign the the RestSharp dll and then recompile the log4stash project with a reference to the signed dll, which is quite a bother. Please revert to the single dependency only on the log4net dll.

Your,

Ariel

Hello!

I've started to use log4stash in our application to send log information to a AWS-Auth elastic search, but I noticed it started to log information by itself.

It logs so much info, following the correct log over console(the ones I want) is very difficult.

My log4net config file (partial and with sensitive info hidden):

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
  <configSections>
    <section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
  </configSections>

  <log4net debug="false">

    <appender name="ConsoleAppender" type="log4net.Appender.ConsoleAppender" >
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%date [%thread] %-5level %logger - %message%newline" />
      </layout>
    </appender>

    <appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
      <Server>whatever</Server>
      <Port>80</Port>
      <IndexName>happy-index</IndexName>
      <IndexType>logEvent</IndexType>
      <BulkSize>100</BulkSize>
      <BulkIdleTimeout>1000</BulkIdleTimeout>
      <IndexAsync>False</IndexAsync>
      <DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>

      <SerializeObjects>True</SerializeObjects>

      <ElasticSearchTimeout>10000</ElasticSearchTimeout>

      <FixedFields>Partial</FixedFields>

      <Ssl>false</Ssl>
      
      <AuthenticationMethod>
        <!--For AWS ElasticSearch service -->
        <Aws>
          <Aws4SignerSecretKey>abc</Aws4SignerSecretKey>
          <Aws4SignerAccessKey>123</Aws4SignerAccessKey>
          <Aws4SignerRegion>somewhere</Aws4SignerRegion>
        </Aws>
      </AuthenticationMethod>
    </appender>

    <root>
      <level value="INFO"/>
      <appender-ref ref="ElasticSearchAppender"/>
      <appender-ref ref="ConsoleAppender"/>      
    </root>
  </log4net>
</configuration>

And then, during my logging process, im greeted (in the console) with A LOT of those:

CanonicalRequest:
POST
/_bulk

content-type:application/json
host:whatever
x-amz-content-sha256:longStringHere
x-amz-date:20210601T212541Z

content-type;host;x-amz-content-sha256;x-amz-date
longStringHereToo

StringToSign:
AWS4-HMAC-SHA256
20210601T212541Z
20210601/us-east-1/es/aws4_request
anotherLongString

Signature:
numbersAndLetters

Authorization:
AWS4-HMAC-SHA256 Credential=Weeeeeee/20210601/us-east-1/es/aws4_request, SignedHeaders=content-type;host;x-amz-content-sha256;x-amz-date, Signature=veryLongStringWithSignature

When I change the root level value to WARN or higher, those logs dissapear. However, my guess is they dont show up because nothing is happening (no logs are being logged)
By checking the Aws4SignerForAuthorizationHeader class, I found many Console.WriteLine statements.

Is there any way to disable those internal loggings with a flag or something? Or just log errors?

Im not sure how I can create a PR with a quick fix for this scenario.

Hello,

I'm using log4stash appender 2.0 and my application is failing when the elasticsearch server requests are timing out.

But, I have a small patch for this how can I submit it?

Hi, I'm trying to convert some fields after parsing message through Grok pattern. Field still have "string" type in ElasticSearch instead of integer.

Here is my appender configuration:

<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
  <Server>localhost</Server>
  <Port>9200</Port>
  <IndexName>dev</IndexName>
  <ElasticFilters>
    <Filter type="log4stash.Filters.RenameKeyFilter, log4stash">
      <Key>processId</Key>
      <RenameTo>ServiceName</RenameTo>
    </Filter>
    <Grok>
      <SourceKey>Message</SourceKey>
      <Pattern>%{WORD:ResponseCode} %{WORD:Method} %{URIPATH:Url} %{WORD:ElapsedMls} %{GREEDYDATA:StatusMessage}</Pattern>
    </Grok>
    <Convert>
      <ToInt>ResponseCode</ToInt>
      <ToInt>ElapsedMls</ToInt>
    </Convert>
  </ElasticFilters>
  <filter type="log4stash.Filter.LevelRangeFilter">
    <levelMin value="INFO" />
    <levelMax value="ERROR" />
  </filter>
</appender>

I've also tried to use BASE10NUM and NUMBER instead of WORD. It doesn't work. Every time I change pattern I delete my current index in ElasticSearch before indexing it again to make sure it is fresh. So the question is how to convert field type from string to integer.

Thanks,
Alex

Hi @urielha ,

I just tried to implement my custom filter based on the existing jsonfilter. but when I add some data as an object, it ends up as some "null" value -> [] (please see the image, this is the json data at kibana).

its ok if the data is converted to its primitive data type (int, string, double). arrays and objects also become like null values.

in jsonfilter.cs

logEvent.Add(prefix, ((JValue)token).Value); // if it goes here, its ok
logEvent[key] = token; // this one will have problem;

when I compile it inside log4stash dll, it works without any problems. any ideas on this?

Thanks!

Hi, I have a testing project that reports to elasticsearch using log4net with log4stash while the developers use something else for reporting to elasticsearch (not log4net).
My idea would be to create an index common for both projects, but I see that I cannot find a common timestamp.

The developers logs have a @timestamp field why I have a properties.@timestamp and a timestamp.

Could a new field @timestamp be added to logged entries?

Thank you!

We have been successfully submitting custom properties using following log4net line, using log4stash 1.1.0

LogicalThreadContext.Stacks["UserName"].Push("jsmith")

This got broken with 1.1.1 (urielha/log4stash-old#14). So we had to use older 1.1.0. I tried to upgrade to 2.0.0 today and instead of Count=1 log4stash does not seem to submit custom values at all.

Our appender configuration:

<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
    <layout type="log4net.Layout.PatternLayout,log4net">
        <param name="ConversionPattern" value="%utcdate{ABSOLUTE} %-5level %logger - %message%newline" />
    </layout>
    <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="WARN" />
        <levelMax value="FATAL" />
    </filter>
    <Server>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</Server>
    <Port>XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX</Port>
    <IndexName>log_%{+yyyy-MM-dd}</IndexName>
    <IndexType>LogEvent</IndexType>
    <Bulksize>2000</Bulksize>
    <BulkIdleTimeout>10000</BulkIdleTimeout>
    <IndexAsync>True</IndexAsync>
    <!-- Message|Exception|Properties -->
    <FixedFields>772</FixedFields>
    <SerializeObjects>false</SerializeObjects>
    <ElasticFilters>
    </ElasticFilters>
</appender>

Is there way to turn off "The remote name could not be resolved" error being thrown if server is not available.

Our system logs messages using rollingfileappender and also ealsticsearchappender, the below error message is thrown if the elastic server is not available

log4net:ERROR Invalid request to ElasticSearch
System.Net.WebException: The remote name could not be resolved: 'server.does.not
.exists'
at System.Net.HttpWebRequest.GetRequestStream(TransportContext& context)
at System.Net.HttpWebRequest.GetRequestStream()
at log4stash.WebElasticClient.SafeSendRequest(RequestDetails request, Func`1
getRequestStream) in d:\uriel\Programing\C#\log4stash\src\log4stash\ElasticClien
t\ElasticClient.cs:line 171

From @MonDeveloper on February 10, 2016 20:20

I usually configure the IndexName with a fixed part (the ImndexName prefix) + the rolling date part.
It worked smoothly.
Now I need something more complex, I need to change the IndexName prefix from "the fixed part" to "a dynamic part" based on a message field like %logger for instance. I tried using both %logger and %{logger} but anytime it has been literally copied as part of the final IndexName and not evaluated.

Copied from original issue: urielha/log4stash-old#15

Hi,

Regarding issue #10 I have now tested it and it works fine with only one custom property. I have more than one property that needs this conversion. When I add a new convert-filter I does not log anything to ES at all?

      <ElasticFilters>
        <Convert>
          <ToString>Guid</ToString>
        </Convert>
        <Convert>
          <ToString>Property2</ToString>
        </Convert>
      </ElasticFilters> 

hi,
is there a filter to drop events ? If not, please advise if there would be a option to do so ?

Thanks.

Hi,

With reference to #10 , I had to add a ToInt converter to get all my data correctly in ES. Is this something you could add in your official release so I don't have to use a custom build?

From @Dima234 on February 8, 2016 23:15

We log custom property values using log4net with this code:
LogicalThreadContext.Stacks["UserName"].Push("jsmith")

Elasticsearch receives this as

"UserName": {
  "Count": 1
},

expected is

"UserName": "jsmith",

For the reference, correct value ('jsmith' in the example above) gets logged by RollingFileAppender, so it looks like something in log4stash causing this issue. We use latest (1.1.1) log4stash version with MS webapi project targeting .NET 4.6. Our appender config looks like this:

<appender name="ElasticSearchAppender" type="log4net.ElasticSearch.ElasticSearchAppender, log4stash">
    <layout type="log4net.Layout.PatternLayout,log4net">
        <param name="ConversionPattern" value="%utcdate{ABSOLUTE} %-5level %logger - %message%newline" />
    </layout>
    <filter type="log4net.Filter.LevelRangeFilter">
        <levelMin value="WARN" />
        <levelMax value="FATAL" />
    </filter>
    <Server>localhost</Server>
    <Port>9200</Port>
    <IndexName>log_%{+yyyy-MM-dd}</IndexName>
    <IndexType>LogEvent</IndexType>
    <Bulksize>2000</Bulksize>
    <BulkIdleTimeout>10000</BulkIdleTimeout>
    <IndexAsync>True</IndexAsync>
    <FixedFields>772</FixedFields>
    <SerializeObjects>false</SerializeObjects>
    <ElasticFilters>
        <kv>
            <SourceKey>Message</SourceKey>
            <ValueSplit>|</ValueSplit>
            <FieldSplit> ,</FieldSplit>
        </kv>
    </ElasticFilters>
</appender>

Copied from original issue: urielha/log4stash-old#14

From @jasonhnz on March 13, 2016 5:50

I have a custom object that is serialized for use in a log4net fileappender. (I'm using a Newtonsoft to serialize my object)
This object is passed in as Log.(customobject) which gets persisted correctly to the txt log file.

I also have the log4stash (elasticsearch appender) setup which sends the object to my ES instance.

The problem is, with my custom object, it appends to the text file correctly , however, because the object is already serialized, as it is transferred to ES, it is serialized again and my json is double escaped.

EG.

....
""eventType"": "Location",
""eventTypeAction"": "CheckOut",
""message"": "Checkout was approved",
.....

Which screws with the mappings of my index.

If I remove my ToString() implementation for serializing my object, it is only serialized once on the way to ES and the json is inserted correctly, however my filelog appender only receives the object , un-serialized , and thus only prints the class name.

Is there a way to disable serialization into ES if this is already done? or is this a log4net issue.

Thanks

Copied from original issue: urielha/log4stash-old#17

Hi @urielha / @jptoto

I was really wondering as there is no documentation available for this project that, is this library ready to use in production ?

From @MonDeveloper on March 11, 2016 0:36

Copied from original issue: urielha/log4stash-old#16

Hi,

I use the log4stash appender in Async mode. When the Elasticsearch server is not availaible for too long, I end up getting an OutOfMemoryException.

Is there an existing mechanism to limit the number of log entries that should be kept in memory by the appender ?

If there is no existing mechanism, I would propose to add a new parameter BulkListLimit that would prevent the exception. ElasticSearchAppender would stop adding new entries when that limit is reached.

To make the solution backward compatible, the parameter would be optional and set to no limit by default.

What do you think ?

Hey, first of all i wanted to say thank you for the project.
I don't know how to connect it to my elk, do i need to create an input in logstash and send the log from the appender to port 5044? please send me some instructions

We are getting this exception when trying to log with log4stash 3.0.0 and log4net 2.0.11.

log4net:ERROR [ElasticSearchAppender] ErrorCode: GenericFailure. Failed in DoAppend
System.ArgumentNullException: Value cannot be null.
Parameter name: input
   at System.Text.RegularExpressions.Regex.Split(String input)
   at log4stash.Filters.ConvertToArrayFilter.ValueToArray(String value)
   at log4stash.Filters.ConvertFilter.PrepareEvent(Dictionary`2 logEvent)
   at log4stash.ElasticAppenderFilters.PrepareEvent(Dictionary`2 logEvent)
   at log4stash.ElasticSearchAppender.PrepareAndAddToBulk(Dictionary`2 logEvent)
   at log4stash.ElasticSearchAppender.Append(LoggingEvent loggingEvent)
   at log4net.Appender.AppenderSkeleton.DoAppend(LoggingEvent loggingEvent)

Not sure if it is related but we use custom log4net code like
LogicalThreadContext.Stacks["FirstName"].Push("Alice")

With log4stash config like

<ElasticFilters>
    <Convert>
        <ToString>FirstName</ToString>
        <ToArray>
            <SourceKey>FirstName</SourceKey>
        </ToArray>

Edit: Code to reproduce the issue:

_log.Warn("Will be logged without FirstName");
using (LogicalThreadContext.Stacks["FirstName"].Push("Alice")) {
    _log.Warn("Will be logged with FirstName = Alice");
}
_log.Warn("This and further messages will not be logged due to ArgumentNullException in ConvertToArrayFilter");

Hello,

Is it possible to create an alias for the index ? having to send daily logs it would be nice to have them all under the same alias per application.

Can someone explain what these properties do?

<BulkSize>2000</BulkSize> <BulkIdleTimeout>10000</BulkIdleTimeout> <IndexAsync>False</IndexAsync> <DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>

I was looking into this library and in the readme it says that .netcore is supported. However the nuget package has moniker for netstandard2.0 but not a folder with actual dlls for it => https://gyazo.com/fac6bcb1bb76e1f689c640e0631b2f7e

When installed in a project which targets netstandard2.0 VS prompts for an error which is expected => https://gyazo.com/73ac8351d7f31b3ee319ac5b5bd49a12

1. Did you use this library in netstandard/.netcore project?
2. Could you please build a proper nuget package or point us how to install it?

I am unable to push logs to elasticsearch over ssl

ASP.NET Core version: 2.1
log4stash package version: 3.0.0

Elasticsearch certificate is not a self signed certificate so I have not added the SelfSignedCertificate flag in configuration. Also since it is a dev setup, the server certificate as well as the CA certificate were generated manually

<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash"> <Server>%ES_HOST%</Server> <Port>%ES_PORT%</Port> <IndexName>index-%{+yyyy-MM-dd}</IndexName> <Ssl>True</Ssl> <AuthenticationMethod> <Basic> <Username>[username]</Username> <Password>[password]</Password> </Basic> </AuthenticationMethod> </appender>

I am not sure what additional information I could provide. Would really appreciate if you could help me with this.

Thanks,
Saurabh

hi,

I'm using log4stash(2.01 and log4net 2.05) to write logs to elasticsearch with x-pack security enabled. I tried to use basic authentication method in my appender (same appender in your README) but i have a 401 error and this message in my application logs:

log4net:ERROR XmlHierarchyConfigurator: Cannot find Property [AuthenticationMethod] to set object on [log4stash.ElasticSearchAppender]

Hi. I am trying to log to our ElasticSearch, but i get the following error:

log4net: Loading Appender [ElasticSearchAppender] type: [log4stash.ElasticSearchAppender, log4stash]
log4net: Setting Property [Server] to String value [192.168.240.199]
log4net: Setting Property [Port] to Int32 value [9200]
log4net: Setting Property [IndexName] to String value [bbvanet]
log4net: Setting Property [IndexAsync] to Boolean value [True]
log4net: Setting Collection Property [AddKv] to object [log4stash.Filters.KvFilter]
log4net: Setting Property [ElasticFilters] to object [log4stash.ElasticAppenderFilters]
log4net: Created Appender [ElasticSearchAppender]
log4net: Adding appender named [ElasticSearchAppender] to logger [root].
log4net: Hierarchy Threshold []
log4net:ERROR Got error while reading response from ElasticSearch
System.Net.WebException: The remote server returned an error: (406) Not Acceptable.
at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
at log4stash.WebElasticClient.SafeGetAndCheckResponse(Func`1 getResponse) in c:\github\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 189

My config is as your example

192.168.240.199
9200

what could be?

Can you release a new NuGet package for 2.0.4? Thanks!

I faced issue logging to ElasticSearch in version 6.x after upgrade log4stash to v3.0.0
In the latest version requests that are send to Elastic doesn't contain field "_type" in body. it's not mandatory for Elastic version 7.x but without it requests fails for 6.x and lower.

Are there any plans for backward compatibility with older Elastic versions? Or maybe is there some way to make log4stash 3.0 working with older ES?

To be aligned with the specifications.
here and here

Hi. I am trying to log to our Elastic Cloud instance using 2.0.4, and I get an error:

log4net:ERROR Got error while reading response from ElasticSearch
System.Net.WebException: The underlying connection was closed: The connection was closed unexpectedly.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at log4stash.WebElasticClient.<>c__DisplayClass4.<FinishGetResponse>b__3() in c:\Uriel\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 127
   at log4stash.WebElasticClient.SafeGetAndCheckResponse(Func`1 getResponse) in c:\Uriel\log4stash\src\log4stash\ElasticClient\ElasticClient.cs:line 198
2

The appender config I have is fairly simple, and I have verified the credentials:

<appender name="ElasticSearchAppender" type="log4stash.ElasticSearchAppender, log4stash">
  <Server>myinstanceid.eu-west-1.aws.found.io</Server>
  <Port>9243</Port>
  <IndexName>remoteadmin-%{+yyyy.MM.dd}</IndexName>
  <AuthenticationMethod>
    <Basic>
      <Username>myusername</Username>
      <Password>mypassword</Password>
    </Basic>
  </AuthenticationMethod>
</appender>

Do you have an suggestions on how I might diagnose this?

Thanks.