troy-west / thimble Goto Github PK
View Code? Open in Web Editor NEWA Clojure Sandbox for Streaming Data Platforms
License: Eclipse Public License 2.0
A Clojure Sandbox for Streaming Data Platforms
License: Eclipse Public License 2.0
Hello,
Thanks a lot for the library, it has been quite helpful. I found this library after watching on your talk.
I just wanted to underline two littles places where the code might want to be updated for newer version of Kafka, we might want to adapt the start-broker function:
(ns troy-west.thimble.kafka
(:require [clojure.java.io :as io]
[integrant.core :as ig]
[troy-west.thimble.zookeeper :as zookeeper])
(:import (java.io File)
(java.util Map)
(kafka.server KafkaServerStartable KafkaConfig)
(org.apache.kafka.clients.admin AdminClient NewTopic KafkaAdminClient)
(org.apache.kafka.clients.consumer KafkaConsumer)
(org.apache.kafka.clients.producer KafkaProducer ProducerRecord)))
(defn start-broker
[zk-state topics dir config]
(let [config (assoc (merge default-broker-config config)
"zookeeper.connect" (zookeeper/server-address zk-state)
"log.dir" (.getAbsolutePath ^File dir))
broker (KafkaServerStartable. (KafkaConfig. config))]
(.startup broker)
(let [admin-host (str (get config "host.name") ":" (get config "port"))
admin-client ^AdminClient (AdminClient/create ^Map {"bootstrap.servers" admin-host})]
(when (seq topics)
(let [n-parts (Integer/parseInt (get config "num.partitions"))
r-factor (short (Integer/parseInt (get config "default.replication.factor")))]
(try
(.get (.all (.createTopics admin-client
(map #(NewTopic. %1 n-parts r-factor)
topics))))
(catch Exception ex
(str "caught exception: " (.getMessage ex))))))
{:config config
:broker broker
:admin-client admin-client})))
What changes is the try catch, because some topics might already exists (ugly hack), this is useful for multiple broker settings. The other point is the constructor for NewTopic accepts a short as last argument and not an int anymore.
I tested this under Kafka 2.7.0 and it seemed to work.
Best regards,
David
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.