Code Monkey home page Code Monkey logo

fluent-plugin-dynatrace's Introduction

fluent-plugin-dynatrace, a plugin for fluentd

This project is developed, maintained and supported by Dynatrace.

A fluentd output plugin for sending logs to the Dynatrace Generic log ingest API v2. For more information, see the Dynatrace documentation on streaming logs with Fluentd.

Requirements

  • An instance of fluentd >= v1.0 from which logs should be exported
  • Ruby version >= 2.7.0
  • An ActiveGate with the Generic log ingest API v2 enabled as described in the Dynatrace documentation
  • A Dynatrace API token with the logs.ingest (Ingest Logs) scope

Installation

The plugin is published on Rubygems at https://rubygems.org/gems/fluent-plugin-dynatrace/.

To install it, run the following command:

fluent-gem install fluent-plugin-dynatrace

If you are using td-agent, run:

td-agent-gem install fluent-plugin-dynatrace

Configuration options

Below is an example configuration which sends all logs with tags starting with dt to Dynatrace.

<match dt.**>
  @type              dynatrace
  active_gate_url    https://{your-environment-activegate}:8021/e/{your-tenant}/api/v2/logs/ingest
  api_token          api_token
  ssl_verify_none    false
  inject_timestamp   false
</match>

If configured with custom <buffer> settings, it is recommended to set flush_thread_count to 1. The output plugin is limited to a single outgoing connection to Dynatrace and multiple export threads will have limited impact on export latency.

match directive

  • required

The match directive is required to use an output plugin and tells fluentd which tags should be sent to the output plugin. In the above example, any tag that starts with dt. will be sent to Dynatrace. For more information see how do match patterns work?.

@type

  • required

The @type directive tells fluentd which plugin should be used for the corresponding match block. This should always be dynatrace when you want to use the Dynatrace output plugin.

active_gate_url

  • required

This is the full URL of the Generic log ingest API v2 endpoint on your ActiveGate.

api_token

  • required

This is the Dynatrace API token which will be used to authenticate log ingest requests. It should be assigned only the logs.ingest (Ingest Logs) scope.

ssl_verify_none

  • optional
  • default: false

It is recommended to leave this optional configuration set to false unless absolutely required. Setting ssl_verify_none to true causes the output plugin to skip certificate verification when sending log ingest requests to SSL and TLS protected HTTPS endpoints. This option may be required if you are using a self-signed certificate, an expired certificate, or a certificate which was generated for a different domain than the one in use.

inject_timestamp

  • optional
  • default: false

If set to true, the timestamp of the log message will be injected into the body of the message in a format readable by Dynatrace.

Example

An full example demonstrating how to set up Fluentd on Kubernetes and export logs to Dynatrace can be found in the example folder.

Development

fluent-plugin-dynatrace supports Ruby versions >= 2.7.0 but it is recommended that at least 2.7.2 is used for development. Ruby versions can be managed with tools like chruby or rbenv.

Install Dependencies

bundle install

Run All Tests

rake test

Run Specific Tests

# Run one test file
rake test TEST=test/plugin/out_dynatrace_test.rb

Code Style Checks

# Check for code style violations
rake rubocop

# Fix auto-fixable style violations
rake rubocop:auto_correct

Run all checks and build

# Runs rubocop, tests, and builds the gem
rake check

fluent-plugin-dynatrace's People

Contributors

arminru avatar dyladan avatar jarekorlowski avatar matheusneder avatar meibensteiner avatar pichlermarc avatar pirgeo avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

fluent-plugin-dynatrace's Issues

send_to_dynatrace error

Hello,

I'm getting the error below while trying to run fluentd to send logs to dynatrace. I'm following the example in this repository.

2022-01-20 20:15:40 +0000 [warn]: #0 got unrecoverable error in primary and no secondary error_class=NoMethodError error="undefined method `downcase' for nil:NilClass"
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/uri/generic.rb:1546:in `use_proxy?'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/uri/generic.rb:1540:in `find_proxy'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1109:in `proxy_uri'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:1096:in `proxy?'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:936:in `connect'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:930:in `do_start'
  2022-01-20 20:15:40 +0000 [warn]: #0 /usr/local/lib/ruby/2.6.0/net/http.rb:925:in `start'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-dynatrace-0.1.2/lib/fluent/plugin/out_dynatrace.rb:118:in `send_to_dynatrace'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluent-plugin-dynatrace-0.1.2/lib/fluent/plugin/out_dynatrace.rb:90:in `write'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/output.rb:1138:in `try_flush'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/output.rb:1450:in `flush_thread_run'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin/output.rb:462:in `block (2 levels) in start'
  2022-01-20 20:15:40 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/2.6.0/gems/fluentd-1.13.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-01-20 20:15:40 +0000 [warn]: #0 bad chunk is moved to /tmp/fluent/backup/worker0/object_3fc275f56d70/5d6092a4e6a8c6fdf6164cf3216e9ca2.log

The content of /tmp/fluent/backup/worker0/object_3fc275f56d70/5d6092a4e6a8c6fdf6164cf3216e9ca2.log is like this (with some special characters):

bstractLogger","loggerName":"br.com.myorganization.retorno.ExceptionMappers","lev
el":"ERROR","message":"My log message","threadName":"executor-thread-0","threadId":21,"mdc":{},"ndc":"","hostName":"my-deployment-name-575cfcbf6-845mn"
,"processName":"quarkus-run.jar","processId":1}
�stream�stdout�docker��container_id�@6e6f34b143b443166d98b4c2f41247c7965b3a1fc00c7cc3f3325b21f2c50c91�kubernetes��container_name� my-deployment-name�namespace_name�axxxx�pod_name�0my-deployment-name-575cfcbf6-845mn�container_image�>my-registry/my-deployment-name:56057�container_image_idْdock
er-pullable://my-registry/my-deployment-name@sha256:d89870a4a995282867bfb6ea2263e936ff1d235d9943576dc4914ad45525982a�pod_id�$953a2f34-ab49-4d25-b17f-86ef
c1e07b23�pod_ip�10.244.9.1�host�my-k8s-node�labels��app� my-deployment-name�chart�&my-deployment-name-56057�heritage�Helm�pod-template
-hash�575cfcbf6�release� my-deployment-name�revision�5�master_url�https://10.96.0.1:443/api�namespace_id�$091e265e-4228-42a6-a4b5-833fc890a473��a���▲"*s��log�☺�{"tim
estamp":"2022-01-20T17:15:37.505-03:00","sequence":788,"loggerClassName":"org.apache.logging.log4j.spi.AbstractLogger","loggerName":"br.com.myorganization.MyClassName","level
":"INFO","message":"My log message [1]","threadName":"executor-thread-0","threadId":21,"mdc":{},"ndc":"","hostName":"xxxx
oconta-575cfcbf6-845mn","processName":"quarkus-run.jar","processId":1}
�stream�stdout�docker��container_id�@6e6f34b143b443166d98b4c2f41247c7965b3a1fc00c7cc3f3325b21f2c50c91�kubernetes��container_name� my-deployment-name�namespace_name�axxxx�pod_name�0my-deployment-name-575cfcbf6-845mn�container_image�>my-registry/my-deployment-name:56057�container_image_idْdock
er-pullable://my-registry/my-deployment-name@sha256:XXXX�pod_id�$953a2f34-ab49-4d25-b17f-86ef
c1e07b23�pod_ip�10.244.9.1�host�my-k8s-node�labels��app� my-deployment-name�chart�&my-deployment-name-56057�heritage�Helm�pod-template
-hash�575cfcbf6�release� my-deployment-name�revision�5�master_url�https://10.96.0.1:443/api�namespace_id�$091e265e-4228-42a6-a4b5-833fc890a473��a���▲L�ꄣlog�☺�{"time
stamp":"2022-01-20T17:15:37.508-03:00","sequence":789,"loggerClassName":"org.apache.logging.log4j.spi.AbstractLogger","loggerName":"br.com.myorganization.retorno.ExceptionMappers","lev
el":"ERROR","message":"My log message","threadName":"executor-thread-0","threadId":21,"mdc":{},"ndc":"","hostName":"my-deployment-name-575cfcbf6-845mn"
,"processName":"quarkus-run.jar","processId":1}
�stream�stdout�docker��container_id�@6e6f34b143b443166d98b4c2f41247c7965b3a1fc00c7cc3f3325b21f2c50c91�kubernetes��container_name� my-deployment-name�namespace_name�axxxx�pod_

I tried some different versions of dynatrace plugin, including 0.1.6 and 0.1.5

Validate config

After #29 it's clear that the error messages received from a missing active gate url is not sufficient. We should attempt to do at least some validation that a URL is passed.

Fluentbit compatibility

Is your feature request related to a problem? Please describe.
Is there a plan to make this plugin compatible with fluentbit? Fluentbit seems to be the successor to fluentd due to its fast and lightweight footprint

ActiveGate log ingest container not starting up.

Activegate Log ingest container creation - example/activegate.yaml

containers:
- name: activegate
image: ${YOUR_ENVIRONMENT_URL}/linux/activegate

Is the above one a valid image after we change the environment_url ?

below is the mount path that we provide on activegate.yaml

   - name: activegate-configuration
      mountPath: /var/lib/dynatrace/gateway/config_template/custom.properties

Pod/container creation is erroring out saying the below error

MountVolume.SetUp failed for volume "activegate-configuration" : configmap references non-existent config key: custom.properties

Specifying timeframe

I'm ingesting this log line:

[2021-09-06 10:00:00 AEST] INFO This is my first log line...

I have the following specified in td-agent.conf:

<source>
  @type tail
  path C:/path/to/your/log/file.log
  pos_file C:/path/to/your/log/file.log.pos
  tag first.*
  <parse>
    @type regexp
	  expression /^\[(?<time>[^\]]+)\] (?<loglevel>[^\s]+) (?<payload>.+)$/
  </parse>
</source>

<match first.**>
  @type              dynatrace
  active_gate_url    https://ACTIVEGATE_IP:9999/e/DT_TENANT_ID/api/v2/logs/ingest
  api_token          ***
  ssl_verify_none    true
</match>

I can see the log entries in Dynatrace but the timestamp is always the current time and doesn't reflect the time parameter.

Is there a way to configure this?

The plugin breaks when trying to process a malformed json

Describe the bug

We are using the fluent-plugin-dynatrace along with the one to send the logs to Logz.io (fluent-plugin-logzio). One of our applications, some times generates logs that include malformed json, that we have to see in our logs analysis tool. These logs are successfully processed by the plugin fluent-plugin-logzio and sent to Logz.io, but when being processed by fluent-plugin-dynatrace, an exception is raised and that makes these logs to never be sent to Dynatrace.

To Reproduce

  1. Attempt to send, from the fluent-dynatrace-plugin, the following records to Dynatrace:
records = [
  {
    "host"=>"10.0.0.X",
    "msg"=>"A regular message",
  },
  {
    "host"=>"10.0.0.X",
    "msg"=>"V\x92\xA1\u000F\xF0ܱ\u0013,\x82",
  } 
]

The second record is a malformed json. That leads to the following error message:

2023-12-14 08:20:32 +0000 [warn]: #0 failed to flush the buffer. retry_times=0 next_retry_time=2023-12-14 08:20:33 +0000 chunk="<chunk_id>" error_class=JSON::GeneratorError error="source sequence is illegal/malformed utf-8"
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:158:in `to_json'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:158:in `serialize'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:149:in `send_records'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:140:in `block in synchronized_send_records'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:139:in `synchronize'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:139:in `synchronized_send_records'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluent-plugin-dynatrace-0.2.0/lib/fluent/plugin/out_dynatrace.rb:109:in `write'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1225:in `try_flush'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluentd-1.16.3/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
  2023-12-14 08:20:32 +0000 [warn]: #0 /opt/rh/rh-ruby30/root/usr/share/gems/gems/fluentd-1.16.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'

Expected behaviour

We would expect that also malformed JSON messages are serialised and sent to Dynatrace from the plugin fluent-plugin-dynatrace. These logs are also relevant for us to analyse. Some times, a malformed json could be exactly the root cause of a problem that we are investigation.
This behaviour is the one we have with our current logs analysis tool (Logz.io), and we also need that on Dynatrace, given that it will be our primary logs analysis tool in the near future.

Desktop (please complete the following information):
OS:

  • The issue happens on a container (based on image centos:7), running on the following host:
# cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
  • The issue is reproducible via unit tests also in a macOS 13.5.1

The container has the following versions running:

  • ruby 3.0.2p107 (2021-07-07 revision 0db68f0233) [x86_64-linux]
  • gem fluent-plugin-beats (1.1.0)
  • gem fluent-plugin-dynatrace (0.2.0)
  • gem fluentd (1.16.3)

Additional context
The idea to use Yajl to serialize the JSON messages came from the implementation of other fluent plugins, that use that approach, such as:

Enhance the supportability messaging

The project page mentions the following

This project is developed and maintained by Dynatrace R&D.

Can the messaging be further enhanced to indicate whether the project is fully supported by Dynatrace or not so its supportability is clear? Example,

This project is developed, maintained and supported by Dynatrace R&D.

Dynatrace customers tends to be uncertain if they'll receive support or not should they come across issues.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.