Comments (2)
Putting json format records to kinesis, it runs for some time and then start producing warning for retry.
No idea what's going wrong behind the scene.
This warning message means that PutRecords API of Kinesis Data Streams or PutRecordBatch API of Kinesis Data Firehose returned failed records because of some kind of error, such as ProvisionedThroughputExceeded.
Gem code:
aws-fluent-plugin-kinesis/lib/fluent/plugin/kinesis_helper/api.rb
Lines 89 to 109 in d5d7c69
Kinesis API reference:
https://docs.aws.amazon.com/kinesis/latest/APIReference/API_PutRecords.html
https://docs.aws.amazon.com/firehose/latest/APIReference/API_PutRecordBatch.html
Can you check your cloud side metrics by CloudWatch?
Also wondering can we do firehose put directly with aggregated records
Currently, we cannot put aggregated records directly into firehose. Appreciate your feedback. See also this issue #193.
from aws-fluent-plugin-kinesis.
Closing this issue for now. Please reopen if required.
from aws-fluent-plugin-kinesis.
Related Issues (20)
- google-protobuf recent release 3.12.0 is breaking fluent-plugin-kinesis HOT 3
- Record size limit exceeded in 1883 KB HOT 3
- Gem dependency issue latest aws kinesis/core gems. HOT 3
- Add placeholder support for the stream name with kinesis_firehose (too)
- Can't use process_credentials without supplying aws_key_id and aws_sec_key HOT 1
- Ruby 2.7 incompatibility HOT 10
- "divided by 0" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/output.rb HOT 2
- Handle IRSA credentials on kinesis plugins HOT 1
- Record size limit exceeded in 1586 KB HOT 9
- Plan to support AWS Glue Schema Registry integration
- Windows support HOT 1
- 3.4.1 tag missing? HOT 1
- When using shared_credentials, plugin does not seem to honor session token expiry HOT 7
- Encryption in transit HOT 1
- cannot load such file -- aws-sdk-core path="aws-sdk-core" error_class=LoadError error="cannot load such file -- aws-sdk-core HOT 2
- always 1 log are lost when using this plugin HOT 1
- Send events to different Kinesis Data Firehose delivery streams based on custom field HOT 2
- Logs periodically not sending to kinesis using kinesis_streams plugin HOT 1
- Ruby version compatibility issues with aws-sdk-ruby HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from aws-fluent-plugin-kinesis.