datadog / datadog-agent Goto Github PK

Main repository for Datadog Agent

License: Apache License 2.0

Shell 0.66% Go 71.58% Python 3.02% C 20.16% Ruby 0.92% HTML 0.58% C++ 0.49% JavaScript 1.07% CSS 0.06% Swift 0.03% Makefile 0.01% Dockerfile 0.06% Java 0.04% Batchfile 0.05% PowerShell 0.18% CMake 0.03% GDB 0.01% SCSS 0.04% HCL 0.01% C# 1.03%

monitoring go datadog apm-agent apm-instrumentation distributed-tracing logging metrics observability open-telemetry

datadog-agent's Introduction

Datadog Agent

The present repository contains the source code of the Datadog Agent version 7 and version 6. Please refer to the Agent user documentation for information about differences between Agent v5, Agent v6 and Agent v7. Additionally, we provide a list of prepackaged binaries for an easy install process here

Note: the source code of Datadog Agent v5 is located in the dd-agent repository.

Documentation

The general documentation of the project, including instructions for installation and development, is located under the docs directory of the present repo.

Getting started

To build the Agent you need:

Go 1.21 or later. You'll also need to set your $GOPATH and have $GOPATH/bin in your path.
Python 3.9+ along with development libraries for tooling. You will also need Python 2.7 if you are building the Agent with Python 2 support.
Python dependencies. You may install these with pip install -r requirements.txt This will also pull in Invoke if not yet installed.
CMake version 3.12 or later and a C++ compiler

Note: you may want to use a python virtual environment to avoid polluting your system-wide python environment with the agent build/dev dependencies. You can create a virtual environment using virtualenv and then use the invoke agent.build parameters --python-home-2=<venv_path> and/or --python-home-3=<venv_path> (depending on the python versions you are using) to use the virtual environment's interpreter and libraries. By default, this environment is only used for dev dependencies listed in requirements.txt.

Note: You may have previously installed invoke via brew on MacOS, or pip in any other platform. We recommend you use the version pinned in the requirements file for a smooth development/build experience.

Note: You can enable auto completion for invoke tasks. Use the command below to add the appropriate line to your .zshrc file. echo "source <(inv --print-completion-script zsh)" >> ~/.zshrc

Builds and tests are orchestrated with invoke, type invoke --list on a shell to see the available tasks.

To start working on the Agent, you can build the main branch:

Checkout the repo: git clone https://github.com/DataDog/datadog-agent.git $GOPATH/src/github.com/DataDog/datadog-agent.
cd into the project folder: cd $GOPATH/src/github.com/DataDog/datadog-agent.
Install go tools: invoke install-tools (if you have a timeout error, you might need to prepend the GOPROXY=https://proxy.golang.org,https://goproxy.io,direct env var to the command).
Create a development datadog.yaml configuration file in dev/dist/datadog.yaml, containing a valid API key: api_key: <API_KEY>. You can either start with an empty one or use the full one generated by the Agent build from Step 5 (located in cmd/agent/dist/datadog.yaml after the build finishes).
Build the agent with invoke agent.build --build-exclude=systemd.

By default, the Agent will be built to use Python 3 but you can select which Python version you want to use:
- invoke agent.build --python-runtimes 2 for Python2 only
- invoke agent.build --python-runtimes 3 for Python3 only
- invoke agent.build --python-runtimes 2,3 for both Python2 and Python3
You can specify a custom Python location for the agent (useful when using virtualenvs):
```
invoke agent.build \
  --python-runtimes 2,3 \
  --python-home-2=$GOPATH/src/github.com/DataDog/datadog-agent/venv2 \
  --python-home-3=$GOPATH/src/github.com/DataDog/datadog-agent/venv3
```
Running invoke agent.build:
- Discards any changes done in bin/agent/dist.
- Builds the Agent and writes the binary to bin/agent/agent.
- Copies files from dev/dist to bin/agent/dist. See https://github.com/DataDog/datadog-agent/blob/main/dev/dist/README.md for more information.
If you built an older version of the agent, you may have the error make: *** No targets specified and no makefile found. Stop.. To solve the issue, you should remove CMakeCache.txt from rtloader folder with rm rtloader/CMakeCache.txt.

Please note that the trace agent needs to be built and run separately.

Please refer to the Agent Developer Guide for more details. For instructions on setting up a windows dev environment, refer to Windows Dev Env.

Testing

Run unit tests using invoke test.

invoke test --targets=./pkg/aggregator

You can also use invoke linter.go to run just the go linters.

invoke linter.go

When testing code that depends on rtloader, build and install it first.

invoke rtloader.make && invoke rtloader.install
invoke test --targets=./pkg/collector/python

Run

You can run the agent with:

./bin/agent/agent run -c bin/agent/dist/datadog.yaml

The file bin/agent/dist/datadog.yaml is copied from dev/dist/datadog.yaml by invoke agent.build and must contain a valid api key.

Run a JMX check

In order to run a JMX based check locally, you must have:

A copy of a JMXFetch jar copied to dev/dist/jmx/jmxfetch.jar
java available on your $PATH

For detailed instructions, see JMX checks

Contributing code

You'll find information and help on how to contribute code to this project under the docs/dev directory of the present repo.

License

The Datadog agent user space components are licensed under the Apache License, Version 2.0. The BPF code is licensed under the General Public License, Version 2.0.

datadog-agent's People

Contributors

Stargazers

Watchers

Forkers

jeremybaumont jeremy-lq sullerandras ravirdv jyee unifio snyk-se masci feeeenng byronwolfman tikyau jmassey-datadog nagarjuna8 rtloeffler rlaveycal ncracker int-tt julienbalestra mdrollette therc cw-sakamoto confluentinc ewdurbin sylr olenm radekl christofferh winebarrel ms4720 vladlosev erikmack lkysow yitnobe dianna531963 jvanlieshout wannapongkjp kenny-house dsahni skarlsson johananl jeffjo overbryd s-maj anderserikbojsen hmhco danielkasen krasnopolsky etsangsplk pulberg jbarciauskas bmoylan nikaxi c2h5oh btseytlin s177437 wowi42 gnana btamayo jenunifer munisystem snowflakekiller minamijoyo fotinakis mokabaddi jstanton617 eduardorha jchen1 stackvista jafsari janl rikonen bryanbolles chulkilee stonith guizmaii seratch jayrapson toming90 kagayanon bluemutedwisdom lostcolony stevenolen christian-marie p-leger fossabot tolbrino hudsonmd minato128 hatappi darnaut anzecesar qingsongyao laurentmoix bewiwi lonyzone thebeege yasra002 potgieterdl fraserthompson gianrubio

datadog-agent's Issues

[dogstream] Implement core go parsers

Explore the opportunity to implement "long running checks" to support stuff like dogstatsd and JMX fetch

In the old agent we let Supervisor orchestrate external processes like JMX or gohai.
We should explore the opportunity to do something similar reusing the Check interface

[scheduler] Avoid to reschedule a check if it's already waiting in the pipeline

ATM the scheduler knows nothing about the state of checks, it just send them to a channel at certain time intervals.

We would like to avoid to schedule a check when the same check is already waiting to be executed.

One way could be providing checks with a lock: the scheduler acquires the lock, the executor releases it.

Port memory check form agent5

Port the memory check to the new agent:

see if the GoPsutil package abstract win/linux or if we still need to have 2 check

[dogstatsd] Clean old contexts from sampler.contexts

Add SunOS support to gopsutils

For now we only need the memory check but we need to see with the maintainer how he would accept the support of SunOS.

[core] Switch logging library

Move from go-logging to seelog

Restore go_expvar check to its original state, using rates not gauges

[bug] Memory leak

The agent6 seems to leak memory (cf agent6 testing board in production).

[aggregator] Allow aggregator to handle `rate` metrics from checks

This requires some changes on the way the aggregator works with metrics from agent checks (currently it handles them exactly like dogstatsd metrics, which doesn't work for rate metrics)

[loader] If python raises an `Import Error` the loader keeps going

We should double check what happens when a Python module for a check contains an Import error. @hush-hush assigning to you b/c you were able to reproduce, let me know if I can help

Provide a way to keep state between checks

Provide a memcache-like system exposed through both Go and Python APIs so checks can persist their state across runs (no need to persist across restarts of the agent)

[dogstream] Add benchmarks

Ideally to be able to compare it to:

different versions of itself ;)
the agent 5 python dogstream
fluentd
mtail

[bug] Investigate why the agent6 got `NaN` at json convertion

The agent6 got NaN exception at json convertion when flushing.

Set up an apt repo

On a new S3 bucket on our AWS demo account.

make our gitlab pipeline push new packages there
make our QA env pull packages from there

build.sh script does not work with go <1.6

build script should set GO15VENDOREXPERIMENT env var before launching go build so that go compiler can found the packages Glide retrieves

[scheduler] do not wait for scheduling timer on stop

When the Schedulers stops, jobs runners wait for the scheduling timer before evaluating the stop condition. This means if a check has a 180s interval, we need to wait three minutes until it stops.

Investigate python interpreter errors when concurrent/fast calls are made

100% chance to reproduce with a Benchmark to any python func

Add ENV vars config provider for global configuration

It's something that needs to be done for the checks too at some point. Priority on the global config file because it could be useful during development

[dogstatsd] Perf benchmark

[aggregator] Have the same suite of tests as Agent 5

All the metric types need extensive tests, at least on par to what we currently have on Agent 5

[dogstatsd] Investigate using SO_REUSEPORT

See stripe/veneur#25

Add support for handling PIDfiles

This is something required by systemd and also useful within the agent itself to decide what to clean up and when

[aggregator] Support dd-specific tags: hostname and device name

[aggregator] Separate Agent and Dogstatsd metrics

All formatted as API payloads for now, but should be kept separately in the aggregator/forwarder

Implement basic python check to collect memory/gc metrics from python runtime

Would be useful to track the resources that the Agent uses, a bit like the go_expvar check.

Especially since it looks like there's currently a mem leak on the Agent that doesn't come from the go runtime.

[check][aggregator] Make checks identify themselves to the aggregator

Currently the aggregator assigns an arbitrary ID to each Sender that's registered to it. This allows the aggregator to know which check a metric comes from.

It would make more sense for each check to have an ID that they would pass to the Sender/aggregator to identify themselves, i.e. the responsibility of identifying a check should be assumed by the check itself instead of by the Sender/aggregator.

The check could then call GetSender(id) to initialize a Sender (id type TBD)

[omnibus] Use S3 caching

See https://trello.com/c/WGnaykJx/1019-omnibus-use-s3-caching

Rename `Configure` to `Init`

Configure method on the Check interface was meant to support configuration injection on the checks but it's also useful to execute code only once in a check lifetime (e.g. computing the # of CPUs in the memory check). To better support the idea, we should rename the method to something more meaningful like Init or Initialize

counters
sets
dd histograms
statsd histograms (ms)