Code Monkey home page Code Monkey logo

loki-enhance-middleware's Introduction

loki-enhance-middleware

typescript Test Suite

loki-enhance-middleware hijacks log push requests sent to loki and modifies it.

Deploy

docker-compose.yaml

services:
  loki:
    image: grafana/loki:2.6.1
    restart: always
    expose:
      - "3100"
    # ...

  enhance_middleware:
      image: nihiue/loki_enhance_middleware:latest
      restart: always

      environment:
        - LOKI_HOST=http://loki:3100
        - WORKER_COUNT=1
        - PORT=3100

      expose:
        - 3100

Config LogAgent

LogAgent needs to:

  • Send requests to middleware
  • Add placeholder to log line for middle to process, see Module - Geo IP for example

promtail-config.yaml

clients:
  - url: http://enhance_middleware:3100/loki/api/v1/push

Module - Geo IP

Injects GeoIP info for any log source

Powered by maxmind and maxmind-npm

GeoIP

promtail-config.yaml

scrape_configs:
  - job_name: caddy
    pipeline_stages:
      - json:
          expressions:
            level:
            status:
            host: request.host
            method: request.method
            url: request.uri
            remote_addr: request.remote_addr

      - labels:
          level:
          status:
          method:

      - template:
          source: output_msg
          template: 'url="{{ .url }}" host="{{ .host }}" GeoIP_Source="{{.remote_addr }}"'

      - output:
          source: output_msg

GeoIP_Source="[IP]" is the placeholder, and it will be replaced by geo-ip fileds.

Log line

foo=bar GeoIP_Source="22.22.22.22" abc=xyz

Result

foo=bar geo_ip_asn="HostSlick" geo_ip_continent="North America" geo_ip_city="Ashburn" geo_ip_city_geoname_id="4744870" geo_ip_country="United States" geo_ip_country_geoname_id="6252001" geo_ip_country_iso_code="US" geo_ip_latitude="39.018" geo_ip_longitude="-77.539" abc=xyz

Module - UserAgent Detect

Parse user_agent field to structure data.

Powered by device-detector-js

UA

Placeholder: Device_UA_Source="[UA]"

Log line - Normal

Device_UA_Source="Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"

Result

ua_client="Chrome Mobile;90.0" ua_device="Google;Pixel 5" ua_os="Android;11.0"

Log line - Search Engine Bot

Device_UA_Source="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Result

ua_bot="Googlebot"

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.