Comments (4)
Requesting for a quick solution.
Most of our deliverables are stuck due to this issue.
from fluentd.
@SohailLS Sorry for my late response.
Could you please send the issue report to https://github.com/fluent/fluent-plugin-opensearch/issues as well?
Because it looks like the issue of fluent-plugin-opensearch
.
https://medium.com/redbox-techblog/tuning-fluentd-retries-avoiding-duplicate-documents-in-elasticsearch-e7cb9630a453 says retrying can cause duplication.
The output plugin fails to send data and retries, but Opensearch is actually properly received the data.
This is what is happening in your environment as well, right?
If so, one thing that could be done on the Fluentd itself (not fluent-plugin-opensearch
) is to disable retries.
(There is a concern, though, that it is not possible to retry data that actually failed to send...)
You can set retry_max_times 0
to disable retries. (https://docs.fluentd.org/output#control-retrying)
<match xxx>
...
<buffer xxx>
...
retry_max_times 0
</buffer>
</match>
Or, you can also have it sent to a secondary just in case.
(This adjusts the parameters to switch to the secondary from the first retry.)
<match xxx>
...
<buffer xxx>
...
retry_type periodic
retry_max_times 3
retry_secondary_threshold 0.1
retry_randomize false
</buffer>
<secondary>
...
</secondary>
</match>
from fluentd.
This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 7 days
from fluentd.
This issue was automatically closed because of stale in 7 days
from fluentd.
Related Issues (20)
- Windows: td-agent 4.5.2 Too many open files HOT 1
- Buffer files remain after restart of fluentd HOT 5
- stdout P followed my infinite loop of "\\\\\\\\\" in cloudwatch log stream for fluentd
- Compression Support in out_http plugin
- fluent/lib/ruby/3.2.0/win32/registry.rb:910:in `encode': U+767E to ASCII-8BIT in conversion from UTF-16LE to UTF-8 to ASCII-8BIT (Encoding::UndefinedConversionError) HOT 1
- DiskPressure due to 80 GB /home/fluentd/core. files
- Set log_path in system config HOT 2
- Startup error using docker startup to specify profile location
- Out_file short flush interval with timekey 1h in buffer HOT 5
- in_tail error after upgrading from 3.8.1 to td-agent-4.5.2 HOT 3
- Fluentd logs HOT 1
- Support AWS SigV4 in the http output plugin HOT 3
- Buffer: v1.16.4: Emit error by IOError HOT 8
- Match directive not working HOT 1
- What will be the impact after removal of OpenSSL c_rehash script from td-agent
- in_exec: Can't handle non-ASCII characters output HOT 1
- Syslog TLS: [client_cert_auth false] settings is not applied if [insecure true] is not set. HOT 4
- [YAML] 'log_level' is deprecated parameter name. use '@log_level' instead // parameter '$log_level' is not used HOT 3
- Broken hadoop_snappy compression in some cases HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fluentd.