Code Monkey home page Code Monkey logo

local-elk-cluster's Introduction

Running local ELK Stack on Docker

This will build 5 containers:

  • 1 x Elasticsearch Master / Ingest Node
  • 2 x Elasticsearch Data Nodes
  • 1 x Kibana
  • 1 x Logstash

Versions of Elasticsearch and cluster password can be set using the env.sh.

NOTE: if you change the password you will also need to update ./pipeline/logstash.conf too

Pre-requisites

  • docker
  • docker-compose

Elasticsearch

set local environment vars

source ./env.sh

Start Cluster

docker-compose up -d

Check Status of Elasticsearch

docker-compose logs -f
curl -X GET "localhost:9200/_cat/nodes?v&pretty" -u elastic:changeme

Connect to Kibana

Open browser to http://localhost:5601/

Send some test data to Logstash

NOW=$(($(date +'%s * 1000 + %-N / 1000000')))
curl -H "Content-Type: application/json" -X POST "http://localhost:9600" -d'
  { 
    "timestamp": "'"${NOW}"'",
    "message": "This is a sample event...."
  }
'

Reload Logstash config on the fly

If you update the ./pipeline/logstash.conf file then changes will be reflected inside the Docker image. Using SIGHUP provides a convenient way to test Logstash configurations by forcing a reload of the Logstash service.

docker exec -u root logstash-01 bash -c 'kill -HUP 1'

Filebeat

Install the client

On your test environment (use following link for detailed instructions)

Copy config file for Filebeat

MAC OS

cp filebeat.yml /usr/local/etc/metricbeat/filebeat.yml

Linux

cp filebeat.yml /etc/metricbeat/filebeat.yml

Configure Index and configure System module and dashboards

filebeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["elastic:changeme@localhost:9200"]'
filebeat modules enable system
filebeat setup --dashboards

Run some logs into Logstash

filebeat -e

Metricbeat

Install the client

On your test environment (use following link for detailed instructions)

Copy config file for Metricbeat

MAC OS

cp metricbeat.yml /usr/local/etc/metricbeat/metricbeat.yml

Linux

cp metricbeat.yml /etc/metricbeat/metricbeat.yml

Configure Metricbeat Dashboards

metricbeat setup --index-management -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["elastic:changeme@localhost:9200"]'
metricbeat setup --dashboards

Run some metrics into Logstash

metricbeat -e

Cleanup

Stop containers

docker-compose down

Prune local system

REF: https://docs.docker.com/config/pruning/

docker container prune
docker network prune
docker image prune
docker volume prune

# or do this....
docker system prune  --volumes

Todo

  • Notes for testing Beats

References

Changes made to run the stack on a Linux Box

Elasticsearch require vm.max_map_count to be set as 262144. To fix the problem with vm.max_map_count too small run to increase it on the box

sudo sysctl -w vm.max_map_count=262144

local-elk-cluster's People

Contributors

bugblatter2016 avatar

Watchers

 avatar  avatar

Forkers

vovakaplenko

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.