Some simple explorations, providing easy to understand reusable examples to get to grips when starting with Kafka.

docker compose up -d
Will start the local Kafka cluster, a data generator, Conduktor UI for interacting with Kafka on localhost:8080.

Running the TFL script will populate a topic with tube details from the TFL API, see below for more.

The consumer folder gives examples of how to consume this data that you can copy.

Data generator, live stream of data from the next tube approaching the Stockwell underground station from the Transport for London API.

This can be used as a Kafka producer for the local stack when an internet connection is available.

Steps involved in it's creation included:

1. Navigating the TFL documentation site to find what endpoints held the data I wanted, how frequently it's updated and how to auth to the API
2. Query the TFL API to grab the station codes depending which station I wanted to watch
3. Query another part of the API to get the tube data, in this case incoming Victoria line tubes to Stockwell station
4. Improve the reliability by building in back-offs and what to do with empty responses
5. Capture and transform the API response into a usable JSON message
6. Produce the JSON message into Kafka

Screenshot of my app polling the TFL API. Screenshot from Conduktor's UI of the message in Kafka.

This leverages an adjusted Conduktor Docker compose stack for local development, containing a Kafka cluster and tooling.

Aiven and Quix for some producer & consumer examples to model my own around.