stevehipwell / fluentd-aggregator Goto Github PK

As the Elasticsearch plugin currently doesn't support AWS sign V4, this plugin extends from elastic search and puts a wrapper around it to authorise sending logs to AWS Elasticsearch when the access is set as restricted.

Our setup of fluentd requires modifying logs, so they are indexed and put to S3 and ES in a specific order with some custom metadata. This plugin helps us achieve that

I tried to use your latest image in one of our aks and I noticed the fluentd pods where crashing. Here is the error given in the pod's log

/usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser/loader.rb:58:in initialize': No such file or directory @ rb_sysopen - /fluentd/etc/fluent.yaml (Errno::ENOENT) from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser/loader.rb:58:in open'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser/loader.rb:58:in open' from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser/loader.rb:58:in load'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config/yaml_parser.rb:51:in parse' from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/config.rb:38:in build'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/supervisor.rb:810:in configure' from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/lib/fluent/command/fluentd.rb:349:in <top (required)>'
from internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb:85:in require' from <internal:/usr/lib/ruby/3.1.0/rubygems/core_ext/kernel_require.rb>:85:in require'
from /usr/lib/ruby/gems/3.1.0/gems/fluentd-1.15.3/bin/fluentd:15:in <top (required)>' from /usr/bin/fluentd:25:in load'
from /usr/bin/fluentd:25:in `

We plan to use Fluentd to send logging record to azure event hub by apache Kafka method.
Can you add this plugin to image?
https://github.com/fluent/fluent-plugin-kafka

Please can you add the fluent-plugin-azure-storage-append-blob-lts plugin ti the Docker image.

https://github.com/elsesiy/fluent-plugin-azure-storage-append-blob-lts

gem install fluent-plugin-azure-storage-append-blob-lts

Add https://github.com/htgc/fluent-plugin-azureeventhubs , so the team can use output to eventhub

We would like to be able to enrich the logs from our application running on accurnt-dev subscriptions with geo info when collected and parsed by fluentd. Please add fluent-plugin-geoip plugin to the image