Code Monkey home page Code Monkey logo

newrelic-fluentd-output's Introduction

Community Plus header

fluent-plugin-newrelic

A Fluentd output plugin that sends logs to New Relic

This project is provided AS-IS WITHOUT WARRANTY OR SUPPORT, although you can report issues and contribute to the project here on GitHub.

Examples

Please see the examples directory for ways to build a Docker image with the New Relic output plugin and other configuration types that could be useful in your environment.

Prerequisites

Fluentd >= v1.0

Installation

Add the plugin to your fluentd agent:

fluent-gem install fluent-plugin-newrelic

If you are using td-agent:

td-agent-gem install fluent-plugin-newrelic

For more info, review Fluentd's official documentation.

Configuration

Required plugin configuration

This plugin must be configured with either a New Relic API Insert key, or a New Relic License key. If both types of keys are specified, the API Insert key will take precedence.

To specify an API Insert key, either set the api_key property in the configuration, or set the NEW_RELIC_API_KEY environment variable. If both are specified, the configuration property will take precedence.

To specify a license key, either set the license_key property in the configuration, or set the NEW_RELIC_LICENSE_KEY environment variable. If both are specified, the configuration property will take precedence.

Optional plugin configuration

Property Description Default value
base_uri New Relic ingestion endpoint https://log-api.newrelic.com/log/v1

EU plugin configuration

If you are running this plugin in the eu set the base_uri to https://log-api.eu.newrelic.com/log/v1.

Fields

  • To make Kubernetes log forwarding easier, any log field in a log event will be renamed to message, overwriting any message field. Kubernetes logs have their messages in a log field, while we want messages in a message field.

Example

Add one of the following blocks to your Fluentd config file (with your specific key), then restart Fluentd.

Using Insights Inserts Key

Example using Insights Insert key:

<match **>
  @type newrelic
  api_key <NEW_RELIC_INSIGHTS_INSERT_KEY>
</match>

Getting your New Relic Insights Insert key: https://insights.newrelic.com/accounts/<ACCOUNT_ID>/manage/api_keys

Using License Key

Example using License key:

<match **>
  @type newrelic
  license_key <NEW_RELIC_LICENSE_KEY>
</match>

Getting your New Relic license key: https://rpm.newrelic.com/accounts/<ACCOUNT_ID>

Playground

To provide a sandbox environment where you can play with the plugin while development or testing we've added a docker-compose environment that would help to run fluend with the plugin and send some logs. More info

Community

New Relic hosts and moderates an online forum where customers can interact with New Relic employees as well as other customers to get help and share best practices. Like all official New Relic open source projects, there's a related Community topic in the New Relic Explorers Hub: Log forwarding

A note about vulnerabilities

As noted in our security policy, New Relic is committed to the privacy and security of our customers and their data. We believe that providing coordinated disclosure by security researchers and engaging with the security community are important means to achieve our security goals.

If you believe you have found a security vulnerability in this project or any of New Relic's products or websites, we welcome and greatly appreciate you reporting it to New Relic through HackerOne.

If you would like to contribute to this project, please create a branch directly in this repository and review these guidelines.

License

newrelic-fluentd-output is licensed under the Apache 2.0 License.

Copyright

  • Copyright(c) 2019 - New Relic
  • License
    • Apache License, Version 2.0

newrelic-fluentd-output's People

Contributors

bangle-98 avatar bmcfeely avatar danielsig727 avatar danybmx avatar icpenguins avatar jcsobrino avatar jodstrcil avatar jsubirat avatar luckslovez avatar matildasmeds avatar melissaklein24 avatar mnevilleoneill avatar ngotim avatar rittneje avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

ngotim danielsig727 umbocv icpenguins mnevilleoneill hedcler saidmb bangle-98 matildasmeds jsubirat gbloquel jimmyw isabella232 pablomoreno61 ghas-results newrelic-csec nr-dsharma jcsobrino

newrelic-fluentd-output's Issues

Missing milliseconds in output

Consider the following config :

<match **>
  @type copy
  @id output_copy
  <store>
    @type newrelic
    num_threads 4
    license_key xxx
    base_uri https://log-api.eu.newrelic.com/log/v1
    <buffer>
      @type file
      flush_at_shutdown true
      flush_mode immediate
      flush_thread_count 32
      flush_thread_interval 1
      flush_thread_burst_interval 1
      retry_forever true
      retry_type exponential_backoff
      path /var/log/td-agent/example
    </buffer>
  </store>
  <store>
    @type file
    path /tmp/output.log
  </store>
</match>

For the same log event, I have different timestamp values.

In my file : "timestamp":1612454131940

on newrelic :
image

So at first sight it looks like the issue is within the newrelic plugin ?

Thanks !

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 0 1 6 0 7

Warning #

Click to see rules

⚠️ third-party-notices-file-exists #

A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. Below is a list of files or patterns that failed:

  • THIRD_PARTY_NOTICES*
  • THIRD-PARTY-NOTICES*
  • THIRDPARTYNOTICES*

Passed #

Click to see rules

✅ license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

✅ readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

✅ readme-starts-with-community-plus-header #

The first 5 lines contain all of the requested patterns. (README.md). The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/.

✅ readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

✅ readme-contains-discuss-topic #

Contains a link to the appropriate discuss.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate discuss.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

✅ code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

[Repolinter] Open Source Policy Issues

Repolinter Report

🤖This issue was automatically generated by repolinter-action, developed by the Open Source and Developer Advocacy team at New Relic. This issue will be automatically updated or closed when changes are pushed. If you have any problems with this tool, please feel free to open a GitHub issue or give us a ping in #help-opensource.

This Repolinter run generated the following results:

❗ Error ❌ Fail ⚠️ Warn ✅ Pass Ignored Total
0 2 1 4 0 7

Fail #

❌ readme-starts-with-community-plus-header #

The README of a community plus project should have a community plus header at the start of the README. If you already have a community plus header and this rule is failing, your header may be out of date, and you should update your header with the suggested one below. For more information please visit https://opensource.newrelic.com/oss-category/. Below is a list of files or patterns that failed:

  • README.md: The first 5 lines do not contain the pattern(s): Open source Community Plus header (see https://opensource.newrelic.com/oss-category).
    • 🔨 Suggested Fix: prepend the latest code snippet found at https://github.com/newrelic/opensource-website/wiki/Open-Source-Category-Snippets#code-snippet-2 to file

❌ readme-contains-forum-topic #

Doesn't contain a link to the appropriate forum.newrelic.com topic (README.md). New Relic recommends directly linking the your appropriate forum.newrelic.com topic in the README, allowing developer an alternate method of getting support. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

Warning #

Click to see rules

⚠️ third-party-notices-file-exists #

A THIRD_PARTY_NOTICES.md file can be present in your repository to grant attribution to all dependencies being used by this project. This document is necessary if you are using third-party source code in your project, with the exception of code referenced outside the project's compiled/bundled binary (ex. some Java projects require modules to be pre-installed in the classpath, outside the project binary and therefore outside the scope of the THIRD_PARTY_NOTICES). Please review your project's dependencies and create a THIRD_PARTY_NOTICES.md file if necessary. For JavaScript projects, you can generate this file using the oss-cli. For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. Below is a list of files or patterns that failed:

  • THIRD_PARTY_NOTICES*
  • THIRD-PARTY-NOTICES*
  • THIRDPARTYNOTICES*
  • LICENSES/THIRD_PARTY_NOTICES*
  • LICENSES/THIRD-PARTY-NOTICES*
  • LICENSES/THIRDPARTYNOTICES*

Passed #

Click to see rules

✅ license-file-exists #

Found file (LICENSE). New Relic requires that all open source projects have an associated license contained within the project. This license must be permissive (e.g. non-viral or copyleft), and we recommend Apache 2.0 for most use cases. For more information please visit https://docs.google.com/document/d/1vML4aY_czsY0URu2yiP3xLAKYufNrKsc7o4kjuegpDw/edit.

✅ readme-file-exists #

Found file (README.md). New Relic requires a README file in all projects. This README should give a general overview of the project, and should point to additional resources (security, contributing, etc.) where developers and users can learn further. For more information please visit https://github.com/newrelic/open-by-default.

✅ readme-contains-link-to-security-policy #

Contains a link to the security policy for this repository (README.md). New Relic recommends putting a link to the open source security policy for your project (https://github.com/newrelic/<repo-name>/security/policy or ../../security/policy) in the README. For an example of this, please see the "a note about vulnerabilities" section of the Open By Default repository. For more information please visit https://nerdlife.datanerd.us/new-relic/security-guidelines-for-publishing-source-code.

✅ code-of-conduct-should-not-exist-here #

New Relic has moved the CODE_OF_CONDUCT file to a centralized location where it is referenced automatically by every repository in the New Relic organization. Because of this change, any other CODE_OF_CONDUCT file in a repository is now redundant and should be removed. Note that you will need to adjust any links to the local CODE_OF_CONDUCT file in your documentation to point to the central file (README and CONTRIBUTING will probably have links that need updating). For more information please visit https://docs.google.com/document/d/1y644Pwi82kasNP5VPVjDV8rsmkBKclQVHFkz8pwRUtE/view. Did not find a file matching the specified patterns. All files passed this test.

subsecond info is not sent to NewRelic API

Hi,

I found that all the log entries sent to NewRelic Logs lost their subsecond info in the timestamp. So all the timestamps in the Logs dashboard only have integer second numbers.

After a bit of digging, I found out that the nsec field in the timestamp is not utilized when forming the request data to NR API. I have a commit that works for me danielsig727@48602db . If you're fine with it, I can create a PR for it. Thanks

Logs are not visible in UI

I can see the response with 202 in plugin but logs don't appear in UI. Same works if i call the http endpoint and send the same logs.

tag releases

Currently releases of this repository are not tagged. Please tag the master branch with the version from lib/newrelic-fluentd-output/version.rb on every release. It would also be helpful to tag all the historical releases.

Plugin does not restrict uploaded data to the 1MB API limit

The plugin takes whatever log records are passed in a chunk from fluentd, compresses them all, and sends the results to New Relic. Unfortunately there is a 1MB limit on those transfers and chunks (in our environment) can be rather large - recently seen 150MB. There's no way that compresses down enough. And, based on the file buffering we use, the default chuck size for that is 256MB.

It was pointed out to me that the newrelic fluent-bit plugin compensates for this. If the compressed size is over 1MB, it splits the log records in half and tries to compress and send each half (recursively, down to a record being unable to compress under 1MB, which it then drops).

This is a request for similar behavior...this output plugin should compensate for whatever data is being fed it and coping with the API limitations of the logging service at New Relic. Until this happens, we cannot rely on the log stream being fed into New Relic. :-/

Thanks!

time must be a Fluent::EventTime (or Integer): Float

I'm trying to forward GELF input into the new relic, and I get this error message:

log-collector_1    | 2021-11-16 11:22:44 +0000 [error]: #0 ignore emit error in object:3ff5c0d7625c error_class=ArgumentError error="time must be a Fluent::EventTime (or Integer): Float"

I've tried adding time and timestamp fields and formatting it differently with record_transformer but no luck. Any idea on what am I doing wrong?
This is the output @type stdout:

log-collector_1    | 2021-11-16 11:22:44.334700107 +0000 fde: {"version":"1.0","host":"c7e0cd518756","short_message":"this is my test cool message","full_message":"this is my test cool message","level":4,"facility":"Payment","file":"/var/www/html/app/Plugin/Config/Controller/ConfigAppController.php","line":63,"appname":"App","topic":"Supercool","url":"app.fotograf.de.ivan.lan/config_dashboard/webinars?_=1637061762943","ref":"http://app.fotograf.de.ivan.lan/config_dashboard/index","ip":"172.20.0.0","pid":"60405","env":"docker","user_agent":"Mozilla/5.0 (X11; Fedora; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0","trace":"long trace,"message":"this is my test cool message"}

do not chunk on timestamp

Currently by default this plugin will chunk the logs based on timestamp.

newrelic-fluentd-output/lib/fluent/plugin/out_newrelic.rb

Line 40 in be92321

config_set_default :chunk_keys, ['timestamp']
This means it will spam the logging API based upon the granularity of your timestamps. For example, if your timestamps have the granularity of seconds, it will send logs every second. This is apparently too much for the server to handle at scale, so it basically drops everything.

Instead, this plugin should default to chunking over a larger time window. Or at least this should be mentioned in the README. https://docs.fluentd.org/configuration/buffer-section#time

FlutendD restart times out due to newrelic plugin

From td-agent/td-agent.log
Every other plugin shuts down correctly but it seems like newrelic plugin fails to shut down in time.

2020-01-29 19:06:32 +0000 [info]: #0 shutting down input plugin type=:tail plugin_id="object:3f9685074364"
2020-01-29 19:06:32 +0000 [info]: #0 shutting down input plugin type=:tail plugin_id="object:3f96845c9a68"
2020-01-29 19:06:32 +0000 [info]: #0 shutting down input plugin type=:tail plugin_id="object:3f9684d3a8ec"
2020-01-29 19:16:31 +0000 [warn]: #0 thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::NewrelicOutput title=:enqueue_thread thread=#<Thread:0x00007f2d05bfb808@/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:70 aborting> error=nil
2020-01-29 19:16:31 +0000 [warn]: #0 thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::NewrelicOutput title=:flush_thread_0 thread=#<Thread:0x00007f2d05bfbb00@/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:70 aborting> error=nil
2020-01-29 19:16:31 +0000 [warn]: #0 thread doesn't exit correctly (killed or other reason) plugin=Fluent::Plugin::TailInput title=:event_loop thread=#<Thread:0x00007f2d05bf9760@/opt/td-agent/embedded/lib/ruby/gems/2.4.0/gems/fluentd-1.3.3/lib/fluent/plugin_helper/thread.rb:70 aborting> error=nil

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.