Apache Kafka: 10 essential terms and concepts explained

Kafka Official Documentation

Kafka performance monitoring metrics

Repo no GitHub

Clonando o repositório do github via http

git clone https://github.com/malki-cedheq/python_mqtt_kafka.git

Clonando o repositório do github via ssh

git clone [email protected]:malki-cedheq/python_mqtt_kafka.git

Clonando o repositório do github via GitHub CLI

gh repo clone malki-cedheq/python_mqtt_kafka

Atualizando o repositório local

git pull

Docker

Python ^3.10

Poetry

MQTT_USERNAME=
MQTT_PASSWORD=
MQTT_PORT=
MQTT_BROKER=

KAFKA_USERNAME=
KAFKA_PASSWORD=
KAFKA_PORT=
KAFKA_BROKER=

1. docker compose -f docker-compose-mqtt.yml up -d --force-recreate
2. docker compose -f docker-compose-kafka.yml up -d --force-recreate

1. docker compose -f docker-compose.yml up -d --force-recreate

docker exec -it mosquitto_container sh

mosquitto_passwd -c /mosquitto/config/password.txt nome_usuario

poetry install

poetry shell

poetry run python3 arquivo.py

1. kafka_producer.py produz mensagens em um tópico do broker do cluster KAFKA.
2. kafka_consumer.py consome mensagens de um tópico do broker do cluster KAFKA.

1. mqtt_kafka_parallel_pub.py publica mensagens em um tópico do broker MQTT e produz a mesma mensagem no tópico de mesmo nome do broker do cluster KAFKA.
2. mqtt_kafka_parallel_sub.py subscreve em um tópico do broker MQTT, e consome as mensagens que são produzidas no mesmo tópico do broker do cluster KAFKA.

1. mqtt_kafka_bridge_pub.py publica mensagens em um tópico do broker MQTT
2. mqtt_kafka_bridge_sub.py subscreve em um tópico do broker MQTT, e ao receber uma nova mensagem no referido tópico encaminha esta mensagem, produz, no mesmo tópico do broker do cluster KAFKA.

http://localhost:9090

http://localhost:9091/metrics

http://localhost:3000

Remove todos os conteiners

docker rm -f $(docker ps -a -q)

Remove todos os volumes

docker volume rm -f $(docker volume ls -q)

Remove as redes inutilizadas

docker network rm -f $(docker network list -q)

Remove todas as imagens armazenas

docker images rmi -f $(docker images -a -q)

*SUBSTITUIR |ZOOKEEPER POR KRAF

https://hub.docker.com/r/bitnami/kafka

https://hub.docker.com/r/obsidiandynamics/kafdrop

https://www.youtube.com/watch?v=FDCTQ47oXUg

https://medium.com/python-point/mqtt-and-kafka-8e470eff606b

1. Docker compose Kafkar + Zookeeper + KafDrop

   1. https://www.youtube.com/watch?v=EV3KiPgE760

   2. https://www.youtube.com/watch?v=8aUZ1J7r8hM

2. Criando consumers e producers em Python

   https://www.youtube.com/watch?v=yT-GYVssdEg

https://www.confluent.io/ko-kr/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/

https://www.metricfire.com/blog/kafka-monitoring/

https://medium.com/@alvarobacelar/monitorando-um-cluster-kafka-com-ferramentas-open-source-a4032836dc79

https://sridharan-rg.medium.com/kafka-metrics-monitoring-with-prometheus-15b0aebea16b

https://computingforgeeks.com/monitor-apache-kafka-with-prometheus-and-grafana/

https://www.youtube.com/watch?v=XXLe0KNEbR4

https://www.youtube.com/watch?v=hL82UnQMnGM

http://www.liferaysavvy.com/2021/07/kafka-monitoring-with-prometheus.html

https://www.metricfire.com/blog/kafka-monitoring/

sudo apt update
sudo apt install default-jre
java -version
sudo apt install default-jdk
javac -version

cd /home/$USER
mkdir prometheus
cd prometheus

https://github.com/prometheus/jmx_exporter

https://github.com/prometheus/jmx_exporter/blob/main/example_configs/zookeeper.yaml

https://github.com/prometheus/jmx_exporter/blob/main/example_configs/kafka-2_0_0.yml

https://github.com/prometheus/jmx_exporter/blob/main/example_configs/kafka-0-8-2.yml

https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/

https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.18.0/jmx_prometheus_javaagent-0.18.0.jar

KAFKA_OPTS="-javaagent:/prometheus/jmx_prometheus_javaagent-0.18.0.jar=$PROMETHEUS_PORT:/prometheus/kafka_broker.yml"

https://github.com/LiferaySavvy/kafka-monitoring/blob/master/prometheus.yml