elastic / beats-docker Goto Github PK

View Code? Open in Web Editor NEW

165.0 342.0 55.0 123 KB

Official Beats Docker images

Makefile 32.62% Python 61.70% Shell 5.68%

beats metricbeat docker filebeat heartbeat monitoring

beats-docker's Introduction

This repository is no longer used to generate the official Beats Docker image from Elastic.

To build Beats docker images for pre-6.6 releases, switch branches in this repo to the matching release.

beats-docker's People

Contributors

Stargazers

Watchers

Forkers

frekele emjaybee83 lightspeedretail dxawnw pvledoux whereismyjetpack exekias multani asusdin87 maxchristl quantonganh landro ahefnawi stkuehnl dchittibala sergey-pronin mool pinguo-zhuanming philbour otdevops orocklin domderen huoyinghui nephel ltagliamonte frntn andrewkroh yantailee jazzimc alienonesecurityllc lcaballero section-io nordstrom meiwangqing richievos doytsujin apolloclark kvch gauntletwizard kayazas123 bl4kk4t lifeofguenter hyperion-hyn piyushss alexandruavadanii pawelprazak samaws1 vkamlesh mdelapenya changsongyang isabella232 gygygyy codystout

beats-docker's Issues

Running Packetbeat in Kubernetes

Hey Elastic!

I wasn't sure where a good place for this was, but I thought other users might benefit from how I setup packetbeat in k8s.

Here is my daemonset configuration. Note that using a daemonset instead of a Pod ensures that each physical host gets a packetbeat container

apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: packetbeat-v5.2.1
  namespace: kube-system
  labels:
    k8s-app: packetbeat
    kubernetes.io/cluster-service: "true"
    version: v5.2.1
spec:
  template:
    metadata:
      labels:
        k8s-app: packetbeat
        kubernetes.io/cluster-service: "true"
        version: v5.2.1
    spec:
      hostNetwork: true
      containers:
      - name: packetbeat
        image: docker.elastic.co/beats/packetbeat:5.2.1
        command:
          - '/bin/sh'
          - '-c'
          - '/usr/share/packetbeat/packetbeat -e -c /etc/packetbeat/packetbeat.yml -path.logs /var/log/packetbeat'
        securityContext:
          #privileged: true
          capabilities:
            add:
              - NET_ADMIN
        env:
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        resources:
          limits:
            memory: 200Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlogpacketbeat
          mountPath: /var/log/packetbeat
        - name: packetbeatconf
          mountPath: /etc/packetbeat
          readOnly: true
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlogpacketbeat
        hostPath:
          path: /var/log/packetbeat
      - name: packetbeatconf
        configMap:
          name: packetbeat-conf

One of the things I've done for metricbeat is create an put my ElasticSearch endpoint in an ExternalName Service, so that in my metricbeat.yml, I can point my elasticsearch destination to this dns name rather than the actual dns name. This is the preferred pattern for routing network traffic to destinations outside your k8s cluster. This works great for metricbeat, however, breaks for packetbeat. I suspect, because packetbeat requires hostNetwork: true, it bypasses kube-dns, which holds k8s service information. The work around is to point packetbeat directly to your elasticsearch, not using k8s services. Here is my k8s service for reference if someone is able to figure out how to use k8s services on the host network, that would be fantastic.

apiVersion: v1
kind: Service
metadata:
  name: elasticsearch-logging
  namespace: kube-system
  labels:
    kubernetes.io/cluster-service: "true"
    kubernetes.io/name: Elasticsearch
spec:
  type: ExternalName
  externalName: your-elasticsearch-hostname.us-west-2.es.amazonaws.com
  ports:
  - name: esrest
    port: 9200
    protocol: TCP
    targetPort: 80

Then in your metricbeat.yml/packetbeat.yml configuration

output.elasticsearch:
  hosts: ["elasticsearch-logging:"]

Mapping definition for [error] has unsupported parameters

I am using a lightly customized installation of all 6.0.0-beta1 beats.

However, When attempting to start them, they all spew this error:

 2017/08/22 19:58:16.455956 client.go:465: WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"Failed to parse mapping [_default_]: Mapping definition for [error] has unsupported parameters:  [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]","caused_by":{"type":"mapper_parsing_exception","reason":"Mapping definition for [error] has unsupported parameters:  [properties : {code={type=long}, message={norms=false, type=text}, type={ignore_above=1024, type=keyword}}]"}}

I have set setup.template.overwrite: true in all of the config files, and I have also run docker run --add-host "elasticsearch:xxx.xxx.xx.xx" docker.elastic.co/beats/filebeat:6.0.0-beta1 setup --template

I am not sure how to interpret this error at all.

Running on amazonlinux, docker 17.03.1-ce, elasticsearch 5.

Running Metricbeat in kubernetes

Running metricbeat in kubernetes, beats.hostname is set to the container name, e.g. "metricbeat-v5.2.1-3bhfr".

When running metricbeats in multiple hosts, you will not be able to distinguish one host from another. Furthermore, if metricbeat dies for any reason, it will be assigned a new container name, e.g. "metricbeat-v5.2.1-XXXXX"

Feature: Add Ubuntu 16.04 base image support

I forked the repo and converted it to Ubuntu 16.04 base image:
https://github.com/apolloclark/packer-elk-docker

What’s the best way to open a PR for this?

Auditbeat 'operation not permitted', even with highly elevated container

The Auditbeat image currently fails with 'operation not permitted' even when:

The container process runs as root
The container is started with --privileged
The container is granted all capabilities (--cap-add=ALL)

# docker run --privileged -u root --cap-add=ALL --rm -it docker.elastic.co/beats/auditbeat:6.0.0-beta2
2017/09/06 04:16:05.881553 beat.go:430: INFO Home path: [/usr/share/auditbeat] Config path: [/usr/share/auditbeat] Data path: [/usr/share/auditbeat/data] Logs path: [/usr/share/auditbeat/logs]
2017/09/06 04:16:05.881593 beat.go:457: INFO Beat metadata path: /usr/share/auditbeat/data/meta.json
2017/09/06 04:16:05.881623 metrics.go:23: INFO Metrics logging every 30s
2017/09/06 04:16:05.881915 beat.go:437: INFO Beat UUID: a1c35d76-0446-4282-93b8-4c9f8d17750f
2017/09/06 04:16:05.881941 beat.go:192: INFO Setup Beat: auditbeat; Version: 6.0.0-beta2
2017/09/06 04:16:05.882124 client.go:123: INFO Elasticsearch url: http://elasticsearch:9200
2017/09/06 04:16:05.882479 module.go:80: INFO Publisher name: 9faf05cdf86d
2017/09/06 04:16:05.882515 metricbeat.go:37: INFO Register [ModuleFactory:[], MetricSetFactory:[audit/file, audit/kernel]]
2017/09/06 04:16:05.882670 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.kernel metricset is a beta feature
2017/09/06 04:16:05.883142 cfgwarn.go:23: WARN EXPERIMENTAL: The audit.file metricset is an experimental feature
2017/09/06 04:16:05.883347 beat.go:264: INFO auditbeat start running.
2017/09/06 04:16:13.740069 audit_linux.go:81: ERR [audit.kernel] failed to delete existing rules: operation not permitted

Very interesting. I feel like there's something I don't know about audit permissions. Any thoughts @andrewkroh?

Unable to run filebeat in docker with yml file mapping

I get error while executing the below command -

sudo docker run -v /etc/filebeat/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.4.0

container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:57: mounting \"/etc/filebeat/filebeat.yml\" to rootfs \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs\" at \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs/usr/share/filebeat/filebeat.yml\" caused \"not a directory\"""
docker: Error response from daemon: oci runtime error: container_linux.go:262: starting container process caused "process_linux.go:339: container init caused "rootfs_linux.go:57: mounting \"/etc/filebeat/filebeat.yml\" to rootfs \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs\" at \"/app/docker/devicemapper/mnt/e9a0e8d10a77d4f658c82511f60be50f4afe5598df8bb2f9c168f15d52ff0fd1/rootfs/usr/share/filebeat/filebeat.yml\" caused \"not a directory\"""
: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type.
ERRO[0000] error waiting for container: context canceled

WARN Can not index event / mapper_parsing_exception / failed to parse [system.process.cgroup.memory.memsw.limit.bytes] / json_parse_exception / Numeric value xxx out of range of long...

Hi,

I'm running MetricBeat as explained here: https://www.elastic.co/guide/en/beats/metricbeat/current/running-on-docker.html#monitoring-host

On a (hardened) Debian 8 machine from OVH French provided.

In MetricBeats' log I get a lot of errors like the following:
WARN Can not index event (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse [system.process.cgroup.memory.memsw.limit.bytes]","caused_by":{"type":"json_parse_exception","reason":"Numeric value (18446744073709551615) out of range of long (-9223372036854775808 - 9223372036854775807)

Same error occurs for:
system.process.cgroup.memory.kmem_tcp.limit.bytes
system.process.cgroup.memory.memsw.limit.bytes
system.process.cgroup.memory.stats.hierarchical_memory_limit.bytes

For confirmed bugs, please report:

Beat Version: MetricBeat 6.0.0 & 6.0.1
Operating System: Debian 8 / Linux xxx 3.14.32-xxxx-grs-ipv6-64 #1 SMP Tue Jun 30 18:50:21 CEST 2015 x86_64 GNU/Linux
Docker Version: 1.6.2
Steps to Reproduce:

run with
host$docker run --name "MBeats"
--volume=/proc:/hostfs/proc:ro \
--volume=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro \
--volume=/:/hostfs:ro \
--net=host
docker.elastic.co/beats/metricbeat:6.0.1 -system.hostfs=/hostfs
Log into guest
host$docker exec -it "MBeats" /bin/bash
Watch logs
guest$ tail -f logs/metricbeat

Metricbeat Docker didn't work with `system.hostfs`

The Metricbeat Docker didn't work with system.hostfs:

docker run -it -e ELASTICSEARCH_URL=http://elasticsearch:9200 -v=/proc:/hostfs/proc:ro -v=/sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro -v=/:/hostfs:ro docker.elastic.co/beats/metricbeat:5.3.0 -system.hostfs=/hostfs

Error:

docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "exec: \"-system.hostfs=/hostfs\": stat -system.hostfs=/hostfs: no such file or directory".

Cannot share filebeat.yml with docker container

Please post all questions and issues on https://discuss.elastic.co/c/beats
before opening a Github Issue. Your questions will reach a wider audience there,
and if we confirm that there is a bug, then you can open a new issue.

For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.

For confirmed bugs, please report:

Beat Version:
Operating System: Ubuntu 16.04; Linux client 4.4.0-119-generic #143-Ubuntu SMP Mon Apr 2 16:08:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
Docker Version: Docker version 18.03.1-ce, build 9ee9f40
Steps to Reproduce:
According to https://www.elastic.co/guide/en/beats/filebeat/current/running-on-docker.html
when I run:
docker run -v /root/filebeat-docker/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.2.4

I get docker: Exiting: error unpacking config data: more than one namespace configured accessing 'output' (source:'filebeat.yml')

filebeat run on k8s in ds , error: line 8: exec: filebeat: not found

filebeat run on k8s in ds , using the yaml file:
https://raw.githubusercontent.com/elastic/beats/6.4/deploy/kubernetes/filebeat-kubernetes.yaml

but error:
/usr/local/bin/docker-entrypoint: line 8: exec: filebeat: not found

Provide Alpine based images

Your new official filebeat docker image runs at 245MB.
A simple build based on Alpine goes up to less than 20MB.

Please move your beat images to Alpine.

I'm adding the Dockerfile I currently use for filebeat below.
(adapted from https://github.com/primait/docker-filebeat)

FROM alpine:3.5

ENV FILEBEAT_VERSION=5.3.0 \
    FILEBEAT_SHA1=c6f56d1a938889ec9f5db7caea266597f625fcc1

RUN mkdir /etc/filebeat && \
  apk add --no-cache wget && \
  wget -q https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-${FILEBEAT_VERSION}-linux-x86_64.tar.gz --no-check-certificate -O filebeat.tar.gz && \
  echo "${FILEBEAT_SHA1}  filebeat.tar.gz" | sha1sum -c - && \
  tar xzvf filebeat.tar.gz && \
  cp filebeat-*/filebeat /usr/local/bin && \
  rm -rf filebeat*

WORKDIR /etc/filebeat
COPY filebeat.yml /etc/filebeat/
COPY filebeat.template.json /etc/filebeat/
COPY filebeat.template-es2x.json /etc/filebeat/
COPY inject.sh /usr/local/bin

ENTRYPOINT []
CMD [ "filebeat", "-e" ]

Use ENTRYPOINT, not just CMD in Dockerfile.

Provide a nice ENTRYPOINT, so that people can do things like:

docker run docker.elastic.co/beats/metricbeat:5.3.0 -system.hostfs=/hostfs

Filebeat cannot work, need help!

Hello, I cannot use docker's filebeat to get my logs.
[machine+version] centos6 filebeat6.3

If I use local filebeat service to work, It can grab any logs I need, But when I run docker's filebeat with my filebeat.yml, It cannot get logs. [This configuration can work on service filebeat start]
When the docker's filebeat runs, there are only Non-zero metrics ticking every 30s, I would like to what caused my error. Hopeful you can help me out. Thanks!
In my case, I get the logs and then output to kafka cluster.

gzip: stdin: unexpected end of file

For security vulnerabilities please only send reports to [email protected].
See https://www.elastic.co/community/security for more information.

For confirmed bugs, please report:

Beat Version: 7.0.0 and 6.3.1
Operating System: 10.13.6 (High Sierra)
Docker Version: 2.0.0.0-mac81 (29211)
Steps to Reproduce: make

 ---> Running in 1772f6cccee3

gzip: stdin: unexpected end of file
tar: Child returned status 1
tar: Error is not recoverable: exiting now
The command '/bin/sh -c curl -Lso - https://artifacts.elastic.co/downloads/beats/auditbeat/auditbeat-6.3.1-linux-x86_64.tar.gz |       tar zxf - -C /tmp &&     mv /tmp/auditbeat-6.3.1-linux-x86_64 /usr/share/auditbeat' returned a non-zero code: 2
make: *** [auditbeat] Error 2```

Feature: Add Auditbeat

Auditbeat does not currently have a Docker image available, it would be great to have one.

Use number, not string for USER

Kubernetes' MustRunAsNonRoot option requires a numeric USER.

can not set logging to file when use docker

For confirmed bugs, please report:

Beat Version: filebeat 6.2.3
Operating System: centos7
Docker Version:18.04.0-ce
Steps to Reproduce:

set the config file: /xxxx/filebeat/config/filebeat.yml like this:
logging.level: info
logging.to_files: true
logging.files:
path: /usr/share/filebeat/logs
name: filebeat
keepfiles: 7
permissions: 0664

and then run docker container:
docker run -d --restart=always --name=filebeat -v /xxxx/filebeat/logs/:/usr/share/filebeat/logs -v /xxxx/filebeat/config/filebeat.yml:/usr/share/filebeat/filebeat.yml docker.elastic.co/beats/filebeat:6.2.3

but there is no log file in "/xxxx/filebeat/logs/" on VM,also no file in "/usr/share/filebeat/logs" in the container.

kibana did not find index 404

For confirmed bugs, please report:

Beat Version: 5.3.1
Operating System: ubuntu
Docker Version: Docker version 17.04.0-ce, build 4845c56
Steps to Reproduce:

The metricbeat is writing data into the elasticsearch i can check that with http://localhost:9200/_cat/indices?v

health status index                           uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   .monitoring-kibana-2-2017.04.26 MkrSWQjBRdGEaQf4csUuog   1   1          6            0     63.7kb         63.7kb
yellow open   .kibana                         yEdBQl8JSNqNhVYMMgx77A   5   1         95           67    394.5kb        394.5kb
yellow open   metricbeat-2017.04.26           ZH5tpPqZS52D-P9TQT5mIQ   5   1       2348            0      4.4mb          4.4mb
yellow open   .monitoring-es-2-2017.04.26     SabclaYqQSmKOQ0KUCsKOQ   1   1        194           76    471.1kb        471.1kb
yellow open   .monitoring-data-2              _6MtkjMeQvGNEs0diHJ1Lg   1   1          3            0      6.9kb          6.9kb

The index is created and has some fields in kibana.

But "Discover" reports "No results found".

When i try to add the index manually the logs give a 404

kibana                       | {"type":"response","@timestamp":"2017-04-26T06:54:58Z","tags":[],"pid":7,"method":"get","statusCode":404,"req":{"url":"/elasticsearch/metricbeat-*/_mapping/field/*?_=1493189698122&ignore_unavailable=false&allow_no_indices=false&include_defaults=true","method":"get","headers":{"host":"localhost:5601","user-agent":"Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:53.0) Gecko/20100101 Firefox/53.0","accept":"application/json, text/plain, */*","accept-language":"en-US,en;q=0.5","accept-encoding":"gzip, deflate","kbn-version":"5.3.1","referer":"http://localhost:5601/app/kibana","connection":"keep-alive"},"remoteAddress":"192.168.160.1","userAgent":"192.168.160.1","referer":"http://localhost:5601/app/kibana"},"res":{"statusCode":404,"responseTime":11,"contentLength":9},"message":"GET /elasticsearch/metricbeat-*/_mapping/field/*?_=1493189698122&ignore_unavailable=false&allow_no_indices=false&include_defaults=true 404 11ms - 9.0B"}

docker-compose.yml

version: '2.1'

services:
  metricbeat:
    image: docker.elastic.co/beats/metricbeat:5.3.1
    restart: unless-stopped
    networks:
      - beats
    volumes:
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
    command: metricbeat -e -system.hostfs=/hostfs
    environment:
      - http_proxy
      - https_proxy
      - no_proxy=${no_proxy},elasticsearch
    depends_on:
      elasticsearch:
        condition: service_healthy

  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.3.1
    container_name: elasticsearch
    environment:
      - "http.host=0.0.0.0"
      - "transport.host=127.0.0.1"
    ports:
      - '127.0.0.1:9200:9200'
    networks:
       - beats
    healthcheck:
      test: ['CMD', 'curl', '-f', '-u', 'elastic:changeme', 'http://localhost:9200']

  kibana:
    image: docker.elastic.co/kibana/kibana:5.3.1
    container_name: kibana
    ports:
      - '127.0.0.1:5601:5601'
    networks:
       - beats
    depends_on:
      elasticsearch:
        condition: service_healthy

  import_dashboards:
      image: docker.elastic.co/beats/metricbeat:5.3.1
      # Using -beat "" causes it to install all dashboards in the zip and not just the metricbeat ones.
      networks:
        - beats
      command: >-
        /usr/share/metricbeat/scripts/import_dashboards
          -file /usr/share/metricbeat/beats-dashboards-5.3.1.zip
          -es http://elasticsearch:9200
          -user elastic
          -pass changeme
      depends_on:
        elasticsearch:
          condition: service_healthy

  set_default_index_pattern:
      image: docker.elastic.co/beats/metricbeat:5.3.1
      networks:
        - beats
      command: >-
        curl -XPUT http://elastic:changeme@elasticsearch:9200/.kibana/config/5.3.1
          -d '{"defaultIndex" : "metricbeat-*"}'
      depends_on:
        elasticsearch:
          condition: service_healthy

networks:
  beats:

Heartbeat requires network capabilities for ICMP

Starting an ICMP monitor when running the heartbeat docker image results in the following error:

2017/02/24 02:53:36.459997 loop.go:330: INFO IPv4 ICMP not supported: listen ip4:icmp : socket: operation not permitted
2017/02/24 02:53:36.460013 loop.go:330: INFO IPv6 ICMP not supported: listen ip6:ipv6-icmp : socket: operation not permitted
2017/02/24 02:53:36.460129 manager.go:125: ERR failed to load monitor tasks: failed to initiate IPv4 support when initializing monitor icmp(0)

This can be fixed by setting the network capabilities not only for packetbeat, but also for heartbeat.

As a workaround, the following custom docker image can be used:

FROM docker.elastic.co/beats/heartbeat:5.2.1
USER root
RUN setcap cap_net_raw,cap_net_admin=eip /usr/share/heartbeat/heartbeat
USER heartbeat

elastic / beats-docker Goto Github PK

beats-docker's Introduction

beats-docker's People

Contributors

Stargazers

Watchers

Forkers

beats-docker's Issues

Recommend Projects

Recommend Topics

Recommend Org