Comments (9)
I am facing similar issue and haven't found any solution yet.
Found another link with same issue which was closed with no resolution
#201
from aws-fluent-plugin-kinesis.
Hi @pradeepbhadani, @singhajit89
Thank you for your feedback!
Do you have any enhancement requests for this plugin? The maximum size of the record payload, up to 1 MB is a limit of Kinesis Data Streams. Even if this plugin could split records longer than 1MB, the consumer of Kinesis Data Streams would get those partial records as different independent records.
I would appreciate it if you have any good idea. Thank you!
from aws-fluent-plugin-kinesis.
Hi @simukappu,
I haven't raise enhancement request yet for this plugin but will definitely raise based on feedback on recommended work around/solution below.
I am using this plugin to send the record to Kinesis Firehose which has exactly same 1 MB limitation and exception in td-agent.log file when record size limit exceeded similar to issue reported on #201
Fact:
You are absolutely right if the plugin could split records longer than 1MB, the consumer would get those partial records as different independent records.
My use Case:
The application we have at our environment has several occurrence where the record size is greater than 1MB. Due to Kinesis 1 MB limitation the td-agent.log file keeps on printing the exception and then there are other adverse impact on respective application server due to log file size getting exploded in GB's and running out of space.
To avoid log explosion haven't configure my application to send logs to Kinesis Firehose.
Recommended Solution:
- Printing exception in td-agent.log configurable for class "ExceedMaxRecordSizeError" in kinesis.rb file.
- This will allow us to use the plugin and send the logs of the application to Kinesis and if there is any record of size greater than 1 MB at least it would not print the exception and fill the td-agent.log
- Or we can just print as ExceedMaxRecordSizeError in td-agent.log file instead of Message and actual record as exception.
The above will help me to use my application to send all the logs to Kinesis excluding the record of size greater than 1 MB and also not filling up the td-agent.log file.
Thank you!
from aws-fluent-plugin-kinesis.
Thank you for the details. I understood what the critical issue is.
We will consider to enhance this plugin to handle ExceedMaxRecordSizeError without printing contents of the records. The cause of this issue seems to be here.
from aws-fluent-plugin-kinesis.
Hi @singhajit89,
Does log_truncate_max_size configuration parameter become a solution for your issue? When you set this parameter, error log printed by "SkipRecordError" including "ExceedMaxRecordSizeError" will be truncated to length of log_truncate_max_size here.
What do you think about your recommended solution?
from aws-fluent-plugin-kinesis.
Hi @simukappu,
I had already tried using the above configuration log_truncate_max_size a month back but that didn't worked for me.
But recently I made a change in kinesis.rb file which now working as expected. For the record greater than 1MB its now only printing the exception name & size and skipping to print the record_message content in td-agent.log file which is helping to keep the file log file size low and also allowing us to see application logs in elasticsearch(excluding record of size greater than 1MB)
Original block----
def to_s
super + ": " + @record_message
end
end
class KeyNotFoundError < SkipRecordError
def initialize(key, record)
super "Key '#{key}' doesn't exist", record
end
end
class ExceedMaxRecordSizeError < SkipRecordError
def initialize(size, record)
super "Record size limit exceeded in #{size/1024} KB", record
end
end
Changes that is working
def to_s
super + ": "
end
end
class KeyNotFoundError < SkipRecordError
def initialize(key, record)
super "Key '#{key}' doesn't exist", record
end
end
class ExceedMaxRecordSizeError < SkipRecordError
def initialize(size, record)
super "ExceedMaxRecordSizeError in #{size/1024} KB", record
end
end
For Example: Using the above if the record size exceeds the kinesis firehose limit it prints below in td-agent.log file
ExceedMaxRecordSizeError in 1157 KB:
from aws-fluent-plugin-kinesis.
Hi @simukappu,
How can I get the above changes committed in actual repo or what is the procedure to do it ?
from aws-fluent-plugin-kinesis.
If we remove @record_message from to_s method, there will be no way to know the content of error record from standard logs. We are providing log_truncate_max_size parameter to shorten log size. We would appreciate it if you could feedback why log_truncate_max_size parameter does not work for you and why you would like to remove @record_message from standard log format. Thank you for your feedback!
from aws-fluent-plugin-kinesis.
Closing this issue for now. Please reopen if required.
from aws-fluent-plugin-kinesis.
Related Issues (20)
- Detect and throw throtlling and other firehose exceptions. HOT 2
- google-protobuf recent release 3.12.0 is breaking fluent-plugin-kinesis HOT 3
- Record size limit exceeded in 1883 KB HOT 3
- Gem dependency issue latest aws kinesis/core gems. HOT 3
- Add placeholder support for the stream name with kinesis_firehose (too)
- Can't use process_credentials without supplying aws_key_id and aws_sec_key HOT 1
- Ruby 2.7 incompatibility HOT 10
- "divided by 0" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb HOT 2
- Handle IRSA credentials on kinesis plugins HOT 1
- Plan to support AWS Glue Schema Registry integration
- Windows support HOT 1
- 3.4.1 tag missing? HOT 1
- When using shared_credentials, plugin does not seem to honor session token expiry HOT 7
- Encryption in transit HOT 1
- cannot load such file -- aws-sdk-core path="aws-sdk-core" error_class=LoadError error="cannot load such file -- aws-sdk-core HOT 2
- always 1 log are lost when using this plugin HOT 1
- Send events to different Kinesis Data Firehose delivery streams based on custom field HOT 2
- Logs periodically not sending to kinesis using kinesis_streams plugin HOT 1
- Ruby version compatibility issues with aws-sdk-ruby HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-fluent-plugin-kinesis.