Code Monkey home page Code Monkey logo

Comments (4)

the7thchakra avatar the7thchakra commented on July 24, 2024

I'm having a very similar issue.

from log4stash.

urielha avatar urielha commented on July 24, 2024

Can you share your configuration?

I think you can work around this if you ok with loosing few log messages in case of overflow

from log4stash.

the7thchakra avatar the7thchakra commented on July 24, 2024

I'd be happy with some lost messages right now!

If there isn't a way / plan for a flush-on-payload-size limit feature, that'd be useful. I'm using AWS OpenSearch and we're limited to 10Mb per message. With the settings below as well as some application-level message size capping, we're regularly blowing past that.

<?xml version="1.0" encoding="utf-8"?>
<log4net>
	<root>
		<level value="ALL" />
		<appender-ref ref="elasticsearch" />
	</root>
	<appender name="elasticsearch" type="log4stash.ElasticSearchAppender, log4stash">
		<Server><!-- redacted --></Server>
		<Port>443</Port>
		<AllowSelfSignedServerCert>False</AllowSelfSignedServerCert>
		<Ssl>True</Ssl>
		<AuthenticationMethod>
			<Basic>
				<Username><!-- redacted --></Username>
				<Password><!-- redacted --></Password>
			</Basic>
		</AuthenticationMethod>

		<IndexName>%{environment}-%{application}-%{log.level}-%{+yyyy-MM-dd}</IndexName>
		<IndexType>_doc</IndexType>
		<IndexAsync>False</IndexAsync>

		<BulkSize>256</BulkSize><!-- we're getting lots of payload too large messages -->
		<BulkIdleTimeout>5000</BulkIdleTimeout>
		<DropEventsOverBulkLimit>False</DropEventsOverBulkLimit>
		<ElasticSearchTimeout>90000</ElasticSearchTimeout><!-- we're getting connection rejected a lot from the server; I'd like to increase BulkSize, but with the payload too large messages, I don't think I can right now -->

		<SerializeObjects>True</SerializeObjects>

		<FixedFields>Partial</FixedFields>

		<ElasticFilters>
			<Remove>
				<Key>log4net:HostName</Key>
			</Remove>
			<Remove>
				<Key>log4net:Identity</Key>
			</Remove>

			<Rename>
				<Key>AppDomain</Key>
				<RenameTo>application</RenameTo>
			</Rename>
			<Rename>
				<Key>HostName</Key>
				<RenameTo>host.name</RenameTo>
			</Rename>
			<Rename>
				<Key>Level</Key>
				<RenameTo>log.level</RenameTo>
			</Rename>
			<Rename>
				<Key>log4net:UserName</Key>
				<RenameTo>UserName</RenameTo>
			</Rename>
			<Rename>
				<Key>LoggerName</Key>
				<RenameTo>log.logger</RenameTo>
			</Rename>
			<Rename>
				<Key>Message</Key>
				<RenameTo>message.text</RenameTo>
			</Rename>
			<Rename>
				<Key>ThreadName</Key>
				<RenameTo>thread.name</RenameTo>
			</Rename>
			<Rename>
				<Key>UserName</Key>
				<RenameTo>server.user.name</RenameTo>
			</Rename>
		</ElasticFilters>
	</appender>
</log4net>

from log4stash.

urielha avatar urielha commented on July 24, 2024

Hi,

We have a mechanism to drop events over the bulk limit, but you will have to play with it in order to achieve optimal solution.

Please read about DropEventsOverBulkLimit there: #77 (comment)

Basically you will have to set BulkIdleTimeout to a more smaller timeout (depends on your amount of log messages per seconds) and set DropEventsOverBulkLimit to true

from log4stash.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.