teragrep / dkr_01 Goto Github PK

View Code? Open in Web Editor NEW

0.0 2.0 1.0 139 KB

Docker RELP Logging Driver

License: Apache License 2.0

Makefile 3.89% Go 93.93% Dockerfile 1.02% Shell 1.16%

dkr_01's Introduction

dkr_01

Docker Logging Driver Plugin. Sends the Docker container’s logs to the specified RELP server.

Logging options

These can be modified via the --log-opt KEY=VALUE command-line argument when using docker run.

Log option	Description	Default value	Possible values
relpHostname	Set the RELP server’s hostname	127.0.0.1	any IP address
relpPort	Set the RELP server’s port	1601	any valid port
relpTls	Use TLS connection instead of unencrypted to RELP server	false	true OR false
syslogHostname	Hostname in syslog message	Actual hostname	any string
syslogAppName	App name in syslog message	teragrep	any string
tags	View docker info in "dkr_01@48577" structured data	off	off, minimal OR full
k8sMetadata	Enrich syslog message with dkr_01_k8s@48577 structured data containing kubernetes metadata	false	true OR false
kubeletUrl	URL used to connect to the kubernetes metadata API	`https://minikube:10250`	any valid URL
k8sClientAuthEnabled	Authenticate the client for Kubernetes metadata	true	true OR false
k8sClientAuthCertPath	Path for the client.crt file for kubernetes metadata, must be accessible from the mount specified in `./plugin/config.json`	`/var/lib/minikube/certs/apiserver-kubelet-client.crt`	valid path for PEM cert
k8sClientAuthKeyPath	Path for the client.key file for kubernetes metadata, must be accessible from the mount specified in `./plugin/config.json`	`/var/lib/minikube/certs/apiserver-kubelet-client.key`	valid path for PEM key
k8sServerCertValidationEnabled	Validate the server certificate	false	true OR false
k8sServerCertPath	Path for the server certificate, must be accessible from the mount specified in `./plugin/config.json`	(empty)	valid path for cert
k8sMetadataRefreshInterval	Metadata refresh interval in seconds, 0 is always.	0	valid positive integer (>=0)
k8sMetadataRefreshEnabled	Enable data refresh more than once in the beginning	false	true OR false

Building the plugin

In git root, run:

$ chmod +x build.sh && bash build.sh

Will make the plugin and enable it. You can check if it was successful with the docker plugin ls command. The plugin’s logs can be accessed from the /run/docker/plugins/<id>/ folder.

Running the plugin with a container

To attach the log driver to a container, run the container with the --log-driver=teragrep/dkr_01:v1 parameter.

$ docker run --log-driver=teragrep/dkr_01:v1 \
    --log-opt relpHostname=127.0.0.1 \
    --log-opt relpPort=1234 \
    --log-opt relpTls=false

If using the docker-compose command, you can customize the docker-compose.yaml to contain the options.

x-logging:
    &default-logging
    options:
        relpHostname: "127.0.0.1"
        relpPort: "1601"
    driver: teragrep/dkr_01:v1
services:
    test:
        build:
            context: .
            dockerfile: Dockerfile
        logging: *default-logging

Using in Minikube

To build and use the plugin in Minikube, follow the instructions below.

(Note: In the commands below, [host]$ means that the command should be run in the host machine’s terminal, [minikube]$ means that it should be run in minikube ssh, and [testserver-pod]$ means that it should be run in the pod’s terminal.)

Start your Minikube environment with
```
[host]$ minikube start
```
Eval the docker-env to in your host machine’s terminal
```
[host]$ eval $(minikube -p minikube docker-env)
```

Build the plugin

[host]$ chmod +x build.sh && bash build.sh

If necessary, build the test server

[host]$ cd testserver && docker build . -t testserver:latest --no-cache

SSH to the minikube
```
[host]$ minikube ssh
```
You need to add the log driver into the dockerd launch command in the docker.service file
```
[minikube]$ sudo vi /lib/systemd/system/docker.service
```

Change the line "ExecStart=" to contain the --log-driver=teragrep/dkr_01:v1 option and any possible log opts.

Reload the daemon, restart docker service and restart kubelet to save these changes.

[minikube]$ sudo systemctl daemon-reload
[minikube]$ sudo service docker restart
[minikube]$ sudo systemctl restart kubelet

You can check if the logging driver is functioning with the command below in minikube ssh terminal:
```
[minikube]$ docker info | grep -i "logging driver"
```

It should display dkr_01 instead of the default jsonfile logging driver, if the plugin has been loaded correctly.

If you built the test server, you can run it with (in your host machine terminal):

[host]$ kubectl run dkrpod --image=testserver:latest --image-pull-policy=Never --restart=Never

To print logs, you can use the pod’s terminal and curl

[host]$ kubectl exec -i -t dkrpod -- /bin/bash

and enter

[testserver-pod]$ curl localhost:9003/foobar

to generate log output in the testserver pod.

Contributing

You can involve yourself with our project by opening an issue or submitting a pull request.

Contribution requirements:

All changes must be accompanied by a new or changed test. If you think testing is not required in your pull request, include a sufficient explanation as why you think so.
Security checks must pass
Pull requests must align with the principles and values of extreme programming.
Pull requests must follow the principles of Object Thinking and Elegant Objects (EO).

Read more in our Contributing Guideline.

Contributor License Agreement

Contributors must sign Teragrep Contributor License Agreement before a pull request is accepted to organization’s repositories.

You need to submit the CLA only once. After submitting the CLA you can contribute to all Teragrep’s repositories.

dkr_01's People

Contributors

Watchers

Forkers

eemhu

dkr_01's Issues

use separate relp connections for each logPair

now one connection is shared among all logPairs

add sequence number to records

multi-line events must be reconstructable

Kubernetes/minikube documentation missing

How to actually use this with Kubernetes or in minikube?

Use private network addresses on documentation

The ip 1.2.3.4 used in tests is publicly routable, https://whatismyipaddress.com/ip/1.2.3.4 and trying to connect and/or send data there is not advisable. Use something like 127.0.0.1 instead.

    --log-opt relpHostname=1.2.3.4 \

and

        relpHostname: "1.2.3.4"

consider pr to moby

https://github.com/moby/moby/tree/master/daemon/logger

peformance test dkr_01

implement performance tests and produce reports

make hostname, appname configureable

make hostname, appname configureable. in certain scenarios we want to override these from log-opts

Re-running make all causes permission issues

$ make all
### removing previously built plugin-build directory
rm -rf ./plugin-build
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/google.golang.org/[email protected]/types/descriptorpb/descriptor.pb.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/google.golang.org/[email protected]/types/pluginpb/plugin.pb.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/google.golang.org/[email protected]/types/known/durationpb/duration_test.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/google.golang.org/[email protected]/types/known/durationpb/duration.pb.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/google.golang.org/[email protected]/types/known/emptypb/empty.pb.go': Permission denied
<snip>
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/github.com/observiq/[email protected]/rfc5425/scanner.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/github.com/observiq/[email protected]/rfc5425/parser.go': Permission denied
rm: cannot remove './plugin-build/rootfs/go/pkg/mod/github.com/observiq/[email protected]/LICENSE': Permission denied
make: *** [Makefile:9: clean] Error 1

$ docker --version
Docker version 20.10.23, build %{shortcommit_cli}
$ uname -a
Linux fedora 6.2.13-200.fc37.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 26 20:15:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Running command example has wrong flag settings

$ docker run --log-driver:teragrep/dkr_01:v1 --log-opt relpHostname=127.0.0.1 --log-opt relpPort=1234 --log-opt relpTls=false example-container:123
unknown flag: --log-driver:teragrep/dkr_01:v1
See 'docker run --help'.

Should be

--log-driver=teragrep/dkr_01:v1

allow inclusion of docker log tags into structured data

https://docs.docker.com/config/containers/logging/log_tags/

add disk buffer option so that logging driver can buffer events if relp is not available

Killing -ti container with ^C causes docker to freeze completely until restart

$ docker plugin ls
ID             NAME                 DESCRIPTION              ENABLED
60d721d8488d   teragrep/dkr_01:v1   RELP Log Driver Plugin   true

$ docker plugin disable teragrep/dkr_01:v1
Error response from daemon: plugin teragrep/dkr_01:v1 is in use

$ docker stop -t 0 kontti
<nothing happens>

I executed a container initially with

$ docker run --rm -ti --name kontti --log-driver=teragrep/dkr_01:v1 --log-opt relpHostname=127.0.0.1 --log-opt relpPort=1601 --log-opt relpTls=false example-container:23
docker: Error response from daemon: failed to initialize logging driver: error creating logger: Post "http://%2Frun%2Fdocker%2Fplugins%2F60d721d8488dfb7cfa58d50fa34c146eb27d515a693f134ddd90ccda4fe77ae3%2Fdkr_01.sock/LogDriver.StartLogging": net/http: request canceled (Client.Timeout exceeded while awaiting headers).

And again with

$ docker run --rm -ti --name kontti --log-driver=teragrep/dkr_01:v1 --log-opt relpHostname=127.0.0.1 --log-opt relpPort=1601 --log-opt relpTls=false example-container:23

Note: I did not have a relp server available at the time of launching it.

Restarting docker gives these outputs:

# The docker run command
docker: error during connect: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/768ee65610544b30384d9f59bdaff58113ba3d5f5c2f31560b544030e52ef477/start": EOF.
ERRO[2170] error waiting for container: unexpected EOF  

# docker stop
error during connect: Post "http://%2Fvar%2Frun%2Fdocker.sock/v1.41/containers/kontti/stop?t=1": EOF

Restarting docker service and re-running the container allows me to show plugins, but inspecting the container stalls

$ docker plugin inspect teragrep/dkr_01:v1 | jq -r '.[0] | .Name + " -> " + .Id'
teragrep/dkr_01:v1 -> 60d721d8488dfb7cfa58d50fa34c146eb27d515a693f134ddd90ccda4fe77ae3

$ docker inspect kontti
<nothing happens>

Seems like reproduce is as follows:

Build and install plugin
Start relp server
Launch a container with -ti
Stop it abruptly with ^C
Relaunch it
-> No logs happening

Running kill -9 inside the container to kill it, or using docker stop $name and docker stop $name -t0 did not cause issues. Only with ^C:ing it from -ti container

investigate into syslog library

please find out if premade syslog library exists for go that we could use. It should allow us to avoid complexities that arise from using structured data which may be necessary in later development requests.

'Building the plugin' docs do not work out of the box

$ git clone https://github.com/teragrep/dkr_01.git
$ cd dkr_01/
$ cd plugin && chmod +rwx ./plugin-build && make all
chmod: cannot access './plugin-build': No such file or directory

plugin-build directory is made by running make all and doesn't exist on fresh repository

implement testing for kubernetes metadata enrichment

test kubernetes metadata fetching in minikube environment

it has been tested via a mock http server providing the sample json data, but not in actual production-like environment

implement kubernetes metadata enrichment

it is possible to fetch kubernetes metadata from kubelet which manages the corresponding node.

Example call to kubelet is shown bellow:

curl -k --key /var/lib/minikube/certs/apiserver-kubelet-client.key --cert /var/lib/minikube/certs/apiserver-kubelet-client.crt https://localhost:10250/pods

{
  "kind": "PodList",
  "apiVersion": "v1",
  "metadata": {},
  "items": [
    {
      "metadata": {
        "name": "etcd-minikube",
        "namespace": "kube-system",
        "uid": "f5bb3d327095d4906ada206fb0e9a699",
        "creationTimestamp": null,
        "labels": {
          "component": "etcd",
          "tier": "control-plane"
        },
        "annotations": {
          "kubeadm.kubernetes.io/etcd.advertise-client-urls": "https://192.168.39.198:2379",
          "kubernetes.io/config.hash": "f5bb3d327095d4906ada206fb0e9a699",
          "kubernetes.io/config.seen": "2023-04-28T12:28:36.042764386Z",
          "kubernetes.io/config.source": "file"
        }
      },
      "spec": {
        "volumes": [
          {
            "name": "etcd-certs",
            "hostPath": {
              "path": "/var/lib/minikube/certs/etcd",
              "type": "DirectoryOrCreate"
            }
          },
          {
            "name": "etcd-data",
            "hostPath": {
              "path": "/var/lib/minikube/etcd",
              "type": "DirectoryOrCreate"
            }
          }
        ],
        "containers": [
          {
            "name": "etcd",
            "image": "k8s.gcr.io/etcd:3.5.3-0",
            "command": [
              "etcd",
              "--advertise-client-urls=https://192.168.39.198:2379",
              "--cert-file=/var/lib/minikube/certs/etcd/server.crt",
              "--client-cert-auth=true",
              "--data-dir=/var/lib/minikube/etcd",
              "--experimental-initial-corrupt-check=true",
              "--initial-advertise-peer-urls=https://192.168.39.198:2380",
              "--initial-cluster=minikube=https://192.168.39.198:2380",
              "--key-file=/var/lib/minikube/certs/etcd/server.key",
              "--listen-client-urls=https://127.0.0.1:2379,https://192.168.39.198:2379",
              "--listen-metrics-urls=http://127.0.0.1:2381",
              "--listen-peer-urls=https://192.168.39.198:2380",
              "--name=minikube",
              "--peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt",
              "--peer-client-cert-auth=true",
              "--peer-key-file=/var/lib/minikube/certs/etcd/peer.key",
              "--peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt",
              "--proxy-refresh-interval=70000",
              "--snapshot-count=10000",
              "--trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt"
            ],
            "resources": {
              "requests": {
                "cpu": "100m",
                "memory": "100Mi"
              }
            },
            "volumeMounts": [
              {
                "name": "etcd-data",
                "mountPath": "/var/lib/minikube/etcd"
              },
              {
                "name": "etcd-certs",
                "mountPath": "/var/lib/minikube/certs/etcd"
              }
            ],
            "livenessProbe": {
              "httpGet": {
                "path": "/health",
                "port": 2381,
                "host": "127.0.0.1",
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 10,
              "timeoutSeconds": 15,
              "periodSeconds": 10,
              "successThreshold": 1,
              "failureThreshold": 8
            },
            "startupProbe": {
              "httpGet": {
                "path": "/health",
                "port": 2381,
                "host": "127.0.0.1",
                "scheme": "HTTP"
              },
              "initialDelaySeconds": 10,
              "timeoutSeconds": 15,
              "periodSeconds": 10,
              "successThreshold": 1,
              "failureThreshold": 24
            },
            "terminationMessagePath": "/dev/termination-log",
            "terminationMessagePolicy": "File",
            "imagePullPolicy": "IfNotPresent"
          }
        ],
        "restartPolicy": "Always",
        "terminationGracePeriodSeconds": 30,
        "dnsPolicy": "ClusterFirst",
        "nodeName": "minikube",
        "hostNetwork": true,
        "securityContext": {
          "seccompProfile": {
            "type": "RuntimeDefault"
          }
        },
        "schedulerName": "default-scheduler",
        "tolerations": [
          {
            "operator": "Exists",
            "effect": "NoExecute"
          }
        ],
        "priorityClassName": "system-node-critical",
        "enableServiceLinks": true
      },
      "status": {
        "phase": "Running",
        "conditions": [
          {
            "type": "Initialized",
            "status": "True",
            "lastProbeTime": null,
            "lastTransitionTime": "2023-04-28T12:28:36Z"
          },
          {
            "type": "Ready",
            "status": "True",
            "lastProbeTime": null,
            "lastTransitionTime": "2023-04-28T12:28:54Z"
          },
          {
            "type": "ContainersReady",
            "status": "True",
            "lastProbeTime": null,
            "lastTransitionTime": "2023-04-28T12:28:54Z"
          },
          {
            "type": "PodScheduled",
            "status": "True",
            "lastProbeTime": null,
            "lastTransitionTime": "2023-04-28T12:28:36Z"
          }
        ],
        "hostIP": "192.168.39.198",
        "podIP": "192.168.39.198",
        "podIPs": [
          {
            "ip": "192.168.39.198"
          }
        ],
        "startTime": "2023-04-28T12:28:36Z",
        "containerStatuses": [
          {
            "name": "etcd",
            "state": {
              "running": {
                "startedAt": "2023-04-28T12:28:38Z"
              }
            },
            "lastState": {
              "terminated": {
                "exitCode": 255,
                "reason": "Error",
                "startedAt": "2023-03-30T14:29:21Z",
                "finishedAt": "2023-04-28T12:28:25Z",
                "containerID": "docker://dfeb0a1ff0fa506a00d9170f764fb54bc77070ef5154fa0281b96db1d63e6baf"
              }
            },
            "ready": true,
            "restartCount": 4,
            "image": "k8s.gcr.io/etcd:3.5.3-0",
            "imageID": "docker-pullable://k8s.gcr.io/etcd@sha256:13f53ed1d91e2e11aac476ee9a0269fdda6cc4874eba903efd40daf50c55eee5",
            "containerID": "docker://92703e3ddeb25a2677af4daf44bd54ea42926a1483916e6fc4b682ece0c5fb65",
            "started": true
          }
        ],
        "qosClass": "Burstable"
      }
    }
  ]
}

logging driver is already aware of the containerId.

containerId can be used to extract further meta information from the kubelet response.

jq '.items[].status.containerStatuses[].containerID' kubelet.json
"docker://92703e3ddeb25a2677af4daf44bd54ea42926a1483916e6fc4b682ece0c5fb65"

jq '.items[].metadata.name' kubelet.json
"etcd-minikube"

jq '.items[].metadata.namespace' kubelet.json
"kube-system"

jq '.items[].metadata.labels' kubelet.json
{
  "component": "etcd",
  "tier": "control-plane"
}

jq '.items[].metadata.uid' kubelet.json
"f5bb3d327095d4906ada206fb0e9a699"

jq '.items[].metadata.creationTimestamp' kubelet.json
null

please extract items above.

please make following configureable:

k8s metadata extraction enabled (toggles the whole feature)
- kubelet address (i.e. https://localhost:10250)
- client auth enabled (toggles if client cert and key are to be used for authentication)
- - tls-client-auth certificate location
- - tls-client-auth key location
- server certificate validation (toggles if server certificate is validated)
- - tls-connection ca-certificate (server cafile to check server certificate against)
- metadata refresh enabled
- - metadata rerfresh interval (seconds, 0 means always, for each request)

further information will be provided what to do with the extracted information.

store the information on per container basis (into the logPair relation). it must not be refreshed if data is present unless refresh is enabled and it has expired.