Example integration of a Kafka Producer, Kafka Broker and Promtail producing test data to Grafana Cloud Logs, see architecture

Requires Docker and Docker Compose

Configure the environment variables below from your Grafana Cloud Account Logs Data Source settings:

1. Log into your Grafana Cloud account to access the Cloud Portal
2. Select the Loki Send Logs to set up and manage the Loki logging service from the Cloud Portal
3. From the Grafana Data Source setting for Logs, use the hostname of the URL, the User and Password in the following environment variables:

Edit the file: envvars-grafana-cloud-stack1.sh

export GRAFANA_LOGS_HOST="logs-prod-us-central1.grafana.net"
export GRAFANA_LOGS_USERNAME="123456"
export GRAFANA_LOGS_API_KEY="eyJrIjoi...."

source envvars-grafana-cloud-stack1.sh
./ctl.sh configure

Check the file promtail-config.yml now has the correct LOGS credentials

./ctl.sh build

./ctl.sh up

This will start the containers: zookeeper, kafka, producer and promtail

Using the Logs Data Source and LogQL Check for log messages under the labels: {job="kafka"} and {job="varlogs"}

Import the example dashboard dashboard-example-1.json into Grafana and configure the Logs Data Source

./ctl.sh down