sentient / statsdbeat Goto Github PK

View Code? Open in Web Editor NEW

4.0 3.0 4.0 30.76 MB

A beat to get statsd udp messages into Elastic Search

License: Other

Makefile 2.18% Go 78.26% Python 3.41% Shell 0.40% Batchfile 0.97% Jinja 14.77%

statsd-backends elasticsearch statsd beat

statsdbeat's Introduction

Statsdbeat

Welcome to Statsdbeat.

Ensure that this folder is at the following location: ${GOPATH}/src/github.com/sentient/statsdbeat

Getting Started with Statsdbeat

Requirements

Golang > 1.17

Init Project

To get running with Statsdbeat and also install the dependencies, run the following command:

mage build

It will create a clean git history for each major step. Note that you can always rewrite the history if you wish before pushing your changes.

To push Statsdbeat in the git repository, run the following commands:

git remote set-url origin https://github.com/sentient/statsdbeat
git push origin master

For further development, check out the beat developer guide.

Build

To build the binary for Statsdbeat run the command below. This will generate a binary in the same directory with the name statsdbeat.

make

Run

To run Statsdbeat with debugging output enabled, run:

./statsdbeat -c statsdbeat.yml -e -d "*"

Test

To test Statsdbeat, run the following command:

make testsuite

alternatively:

make unit-tests
make system-tests
make integration-tests
make coverage-report

The test coverage is reported in the folder ./build/coverage/

Update

Each beat has a template for the mapping in elasticsearch and a documentation for the fields which is automatically generated based on fields.yml by running the following command.

make update

Cleanup

To clean Statsdbeat source code, run the following command:

make fmt

To clean up the build directory and generated artifacts, run:

make clean

Clone

To clone Statsdbeat from the git repository, run the following commands:

mkdir -p ${GOPATH}/src/github.com/sentient/statsdbeat
git clone https://github.com/sentient/statsdbeat ${GOPATH}/src/github.com/sentient/statsdbeat

For further development, check out the beat developer guide.

Packaging

The beat frameworks provides tools to crosscompile and package your beat for different platforms. This requires docker and vendoring as described above. To build packages of your beat, run the following command:

make release

This will fetch and create all images required for the build process. The whole process to finish can take several minutes.

statsdbeat's People

Contributors

Stargazers

Watchers

Forkers

mwik ashisheric codefreezr overbryd

statsdbeat's Issues

Support for OpenDistro resp. OSS-Version of Elasticsearch

Do you support the new OSS Version of Elasticsearch, for e.g. OpenDistro for Elasticsearch?

[_doc, doc] _Type Issue

Since a couple of weeks we have some Elastic-Instances on which we did not get any statsd-Messages into ElasticSearch. All this Messages end with:

{"type":"illegal_argument_exception","reason":"Rejecting mapping update to [statsd-6.3.3-2020.11.23] as the final mapping would have more than 1 type: [_doc, doc]"}

We have made no changes on purpose since around 1 Year or so.

Any Ideas? Even how we can try to find the cause or a work-a-round?

ELS: 7.4.2
Statsdbeat: 6.3.3
OS: Amazon Linux

statsdbeat 7.1.2 ignores config

We have configured as follow inside statsdbeat.yml (Using statsdbeat 7.1.2 against a ELS 7.4.2.):

index: statsd-%{[beat.version]}-%{+yyyy.MM.dd}

...

setup:
  template:
    name: statsd
    pattern: statsd-*

This would assume we would could create an index pattern in the form of "statsd-*"

But the resulting Index was for e.g. in the Form of:
statsdbeat-7.1.2-2020.11.25-000001

This breaks all our Visuals.

It seems work with a config on a 7.1.2 (beat) and 7.8.1 (ELS). Any Ideas?

Failed making a beat

We receive the following error:
2019-03-15T11:27:58.931Z ERROR [statsdbeat] beater/statsdbeat.go:71 Failed making a beat{error 25 0 Expecting 2 or 3 parts of | but was 1} 2019-03-15T11:27:58.931Z ERROR [statsdbeat] beater/statsdbeat.go:71 Failed making a beat{error 25 0 strconv.Atoi: parsing "0.0005": invalid syntax}
The statsd flavor is etsy.

Any ideas?

Feature Request

Can you compile also 64bit RPM's in a release?

Question: Is TCP or UDP used to write stats in pipelines ?

Hello,

I need above clarification. statsd uses UDP to send packets to statsdbeat, but what statsdbeat uses to send packets to logstash pipelines ?
What changes are required to support TCP if not already implemented.
What happens to stats which couldn't write into pipeline because of network Issues ? I guess you don't accumulate and retry.

Thanks,
Rahul.

Help with testing for a PR/feature

Hi @sentient,

I started to prepare a PR, for supporting an alternate way of tagging statsd messages.
In my case, the only major problem why I have a hard time replacing Datadog metrics with the Elastic stack.

The datadog datagram format is widely used in many statsd clients I know of (e.g. traefik loadbalancer).
It is described here https://docs.datadoghq.com/developers/dogstatsd/datagram_shell/#datagram-format

and so with my limited knowledge of beats development and go I set out to implement this.

This is the patch I came up with: https://gist.github.com/Overbryd/4fe442429bbdaff8f1334251fd607827

Unfortunately, I cannot run the tests and I need some help getting my local "beats" / "statsdbeats" development environment setup.

I checked out your repository in my GO_PATH and all, fetched the necessary dependencies,
but running go test -run '' does not get me any further.

Would you mind to enlighten me about your dev setup for statsdbeat?

Kind regards,

Lukas