Code Monkey home page Code Monkey logo

elk-projects's Introduction

ELK Stack

This is a simple implementation of ELK stack using docker-compose

Run

sudo docker-compose up --build

Check elasticsearch

curl localhost:9200

Check kibana

Browse to localhost:5601

Run logstash for terminal

sudo docker run -d -h logstash --name logstash --link elasticsearch:elasticsearch --net elk_default -it --rm -v "$PWD"/logstash:/config-dir logstash:7.16.3 -f /config-dir/terminal-listener.conf

Then pass in some input at the same terminal

a
test1
b
c

Then follow the steps below:

  1. browse to localhost:5601
  2. then go to http://localhost:5601/app/management/kibana/indexPatterns
  3. create an index pattern for kibana
    1. Note: Your index should have common characters of the inputs unless kibana does not create an index without any inputs.
  4. Then browse to http://localhost:5601/app/discover
  5. And you will see your first index by using the upper left dropdown you can change between your indices.

discover_01

Docker Run for elastic and kibana:

sudo docker run -d -p 9200:9200 -p 9300:9300 --rm -h elasticsearch --name elasticsearch -e xpack.security.enabled=false -e discovery.type=single-node  elasticsearch:7.16.3
sudo docker run -d -p 5601:5601  --rm -h kibana --name kibana --link elasticsearch:elasticsearch kibana:7.16.3

logstash listening to a port:

sudo docker-compose -f docker-compose.yml up --build
sudo docker run -d -h logstash --name logstash -p 9300:9300 --link elasticsearch:http://localhost:9200 -it --rm -v "$PWD"/logstash:/config-dir logstash:7.16.3 -f /config-dir/port-listener.conf

Or packing all containers in a single docker-compose

sudo docker-compose -f docker-compose-port.yml up 

Send logs to defined port on logstash container using telnet:

telnet localhost 9300

Append your messages and you will find your request appending on http://localhost:5601/app/discover as well

How to log containers using fluentd

  1. First off, create fluentd/Dockerfile & fluentd/conf/fluent.conf
  2. Then, add the following section to all the container that needs to be logged.
logging:
      driver: "fluentd"
      options:
        fluentd-address: localhost:24224
        tag: httpd.access
  1. Finally, run the following command
sudo docker-compose -f docker-compose-fluentd.yml up --build

Run APM

sudo docker-compose -f docker-compose-apm.yml up

References:

  1. https://www.youtube.com/watch?v=6bXSfjwQVIc
  2. https://medium.com/analytics-vidhya/installing-elk-stack-in-docker-828df335e421
  3. https://logz.io/blog/docker-logging/
  4. https://docs.fluentd.org/v/0.12/container-deployment/docker-compose

elk-projects's People

Contributors

pooya-mohammadi avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.