loki-enhance-middleware

loki-enhance-middleware hijacks log push requests sent to loki and modifies it.

Deploy

docker-compose.yaml

services:
  loki:
    image: grafana/loki:2.6.1
    restart: always
    expose:
      - "3100"
    # ...

  enhance_middleware:
      image: nihiue/loki_enhance_middleware:latest
      restart: always

      environment:
        - LOKI_HOST=http://loki:3100
        - WORKER_COUNT=1
        - PORT=3100

      expose:
        - 3100

Config LogAgent

LogAgent needs to:

Send requests to middleware
Add placeholder to log line for middle to process, see Module - Geo IP for example

promtail-config.yaml

clients:
  - url: http://enhance_middleware:3100/loki/api/v1/push

Module - Geo IP

Injects GeoIP info for any log source

promtail-config.yaml

scrape_configs:
  - job_name: caddy
    pipeline_stages:
      - json:
          expressions:
            level:
            status:
            host: request.host
            method: request.method
            url: request.uri
            remote_addr: request.remote_addr

      - labels:
          level:
          status:
          method:

      - template:
          source: output_msg
          template: 'url="{{ .url }}" host="{{ .host }}" GeoIP_Source="{{.remote_addr }}"'

      - output:
          source: output_msg

GeoIP_Source="[IP]" is the placeholder, and it will be replaced by geo-ip fileds.

Log line

foo=bar GeoIP_Source="22.22.22.22" abc=xyz

Result

foo=bar geo_ip_asn="HostSlick" geo_ip_continent="North America" geo_ip_city="Ashburn" geo_ip_city_geoname_id="4744870" geo_ip_country="United States" geo_ip_country_geoname_id="6252001" geo_ip_country_iso_code="US" geo_ip_latitude="39.018" geo_ip_longitude="-77.539" abc=xyz

Module - UserAgent Detect

Parse user_agent field to structure data.

Placeholder: Device_UA_Source="[UA]"

Log line - Normal

Device_UA_Source="Mozilla/5.0 (Linux; Android 11; Pixel 5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/90.0.4430.91 Mobile Safari/537.36"

Result

ua_client="Chrome Mobile;90.0" ua_device="Google;Pixel 5" ua_os="Android;11.0"

Log line - Search Engine Bot

Device_UA_Source="Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

Result

ua_bot="Googlebot"

Need help with expressions.

Hi, I would love to use this, but I dont know how to setup the expression via json.

Here is an example of my json output (json is written to file per line)
{"ct_id":0,"ct_mark":0,"detected_application":303,"detected_application_name":"126.netify.google","detected_protocol":188,"detected_protocol_name":"QUIC","detection_guessed":1,"dhc_hit":true,"digest":"99b10c8d5fa0dd3fdd17abe1bfab5b82bad037f2","first_seen_at":1674703379228,"first_update_at":1674703379228,"host_server_name":"clients2.google.com","ip_nat":false,"ip_protocol":17,"ip_version":4,"last_seen_at":1674703379269,"local_ip":"10.0.5.143","local_mac":"c8:2a:dd:88:de:e3","local_origin":true,"local_port":40180,"other_ip":"142.250.176.14","other_mac":"20:6d:31:11:10:d2","other_port":443,"other_type":"remote","vlan_id":0,"interface":"br-lan","internal":true,"type":"flow"}

Here is my promtail.yml config

server:
  http_listen_port: 9080
  grpc_listen_port: 0

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://lokimiddleware:3100/loki/api/v1/push

- job_name: netify-logs
  static_configs:
  - targets:
      - localhost
    labels:
      job: netifylogs
      __path__: /var/log/netify/*log
    pipeline_stages:
      - json:
          expressions:
            detected_protocol_name: detected_protocol_name
            host_server_name: host_server_name
            local_ip: local_ip
            local_mac: local_mac
            local_port: local_port
            other_ip: local_ip
            other_ip: other_ip
            other_mac: other_mac
            other_port: other_port
      - labels:
          host_server_name:
          local_ip:
          local_mac:
          local_port:
          other_ip:
          other_mac:
          other_port:
      - template:
          source: output_msg
          template: 'detected_protocol_name="{{ .detected_protocol_name }}" local_ip="{{ .local_ip }}" local_mac="{{ .local_mac }}"  local_port="{{ .local_port }}"  GeoIP_Source="{{ .other_ip }}" other_mac="{{ .other_mac }}" other_port="{{ .other_port }}"'
      - output:
          source: output_msg

nihiue / loki-enhance-middleware Goto Github PK