spotify / docker-kafka Goto Github PK
View Code? Open in Web Editor NEWKafka (and Zookeeper) in Docker
License: Apache License 2.0
Kafka (and Zookeeper) in Docker
License: Apache License 2.0
I get a LeaderNotAvailableException when I use the console producer from the docker host. I'm on Fedora and start the container like this:
docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=127.0.0.1 --env ADVERTISED_PORT=9092 spotify/kafka
Then I try to write to it:
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
resulting in:
WARN Error while fetching metadata [{TopicMetadata for topic test -> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2016-06-01 09:57:20,149] WARN Error while fetching metadata [{TopicMetadata for topic test -> No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo) [2016-06-01 09:57:20,149] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
What am I doing wrong?
It would be awesome if the docker image had more tags than just "latest". How difficult would it be for there to be different tags for different versions? I'm working with some other systems to do not yet support kafka 0.10.
Where's the volume mount option for docker kafka?
Hi,
I am new to the Kafka docker, so I want to increase the maximum client connection in the Kafka image to be unlimited
that need to set the zookeeper configuration(/etc/zookeeper/conf/zoo.cfg
) by adding this statement(maxClientCnxns=0
), can you guy give me a suggestion on this, how can I do that
Thanks,
I tried
git clone https://github.com/spotify/docker-kafka
cd docker-kafka/
cd kafka
docker build -t kafka:0.10.1.0 .
and I get the following error
Get:6 http://deb.debian.org jessie/main amd64 Packages [9098 kB]
Err http://deb.debian.org jessie-backports/main amd64 Packages
Err http://deb.debian.org jessie-backports/main amd64 Packages
Err http://deb.debian.org jessie-backports/main amd64 Packages
Err http://deb.debian.org jessie-backports/main amd64 Packages
Err http://deb.debian.org jessie-backports/main amd64 Packages
404 Not Found
Fetched 10.1 MB in 17s (583 kB/s)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-updates/InRelease Unable to find expected entry 'main/binary-amd64/Packages' in Release file (Wrong sources.list entry or malformed file)
W: Failed to fetch http://deb.debian.org/debian/dists/jessie-backports/main/binary-amd64/Packages 404 Not Found
E: Some index files failed to download. They have been ignored, or old ones used instead.
The command '/bin/sh -c apt-get update && apt-get install -y zookeeper wget supervisor dnsutils && rm -rf /var/lib/apt/lists/* && apt-get clean && wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz' returned a non-zero code: 100
Kafka failed to start with this error message:
2017-08-16 18:21:57,172 CRIT Supervisor running as root (no user in config file)
2017-08-16 18:21:57,173 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2017-08-16 18:21:57,173 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2017-08-16 18:21:57,181 INFO RPC interface 'supervisor' initialized
2017-08-16 18:21:57,181 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2017-08-16 18:21:57,181 INFO supervisord started with pid 1
2017-08-16 18:21:58,185 INFO spawned: 'zookeeper' with pid 7
2017-08-16 18:21:58,187 INFO spawned: 'kafka' with pid 8
2017-08-16 18:21:58,201 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:21:59,208 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2017-08-16 18:21:59,210 INFO spawned: 'kafka' with pid 28
2017-08-16 18:21:59,214 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:01,218 INFO spawned: 'kafka' with pid 29
2017-08-16 18:22:01,225 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:04,231 INFO spawned: 'kafka' with pid 30
2017-08-16 18:22:04,238 INFO exited: kafka (exit status 127; not expected)
2017-08-16 18:22:05,239 INFO gave up: kafka entered FATAL state, too many start retries too quickly
Does anyone encountered that, how can I solve this problem?
First off let me say thank you for this great image!
A few days ago I was running the image and it was working well.
Lately, though, I've been trying to follow the most basic steps (as listed in the Run section at dockerhub) and it's been erroring with
root@2995acf13535:/opt/kafka_2.11-0.8.2.1/bin# ./kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2016-08-25 12:06:33,784] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
Hi There
[2016-08-25 12:06:36,183] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2016-08-25 12:06:36,214] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2016-08-25 12:06:36,217] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
Just to verify that I'm not doing anything ultra dumb, in linux I'm,
localhost
because I'm in linux): docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=localhost --env ADVERTISED_PORT=9092 spotify/kafka
docker exec -it stupefied_wing bash
cd /opt/kafka_2.11-0.8.2.1/bin/
./kafka-console-producer.sh --broker-list localhost:9092 --topic test
[2016-08-25 12:12:35,959] WARN Property topic is not valid (kafka.utils.VerifiableProperties)
Thanks again, any help would be greatly appreciated.
Hi!
Could you publish your image to the Docker registry? Thanks!
Hi,
I was able to use the spotify/kafka image and run a kafka/zookeeper container. When i scale it using marathon, it creates additional containers each with kafka and zookeeper. How do make the additional containers part of the same cluster?
Thanks!
Sonali
Hi thanks for sharing this piece of work!
I have a question:
export KAFKA=docker-machine ip \
docker-machine active``:9092
kafka-console-producer.sh --broker-list $KAFKA --topic test
Where do you execute those scripts? Outside of the container there is no such shell script. But how to execute it?
Thanks
Hi,
I'm trying to build docker image from source but it failed as following
Sending build context to Docker daemon 9.216 kB
Step 1 : FROM java:openjdk-8-jre
---> 0a290b7fd4c4
Step 2 : ENV DEBIAN_FRONTEND noninteractive
---> Using cache
---> c9c7a60507de
Step 3 : ENV SCALA_VERSION 2.11
---> Using cache
---> 6b1df17b1b9b
Step 4 : ENV KAFKA_VERSION 0.10.0.1
---> Using cache
---> c28d56fc3f16
Step 5 : ENV KAFKA_HOME /opt/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION"
---> Using cache
---> 69728c9aac29
Step 6 : RUN apt-get update && apt-get install -y zookeeper wget supervisor dnsutils && rm -rf /var/lib/apt/lists/* && apt-get clean && wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz
---> Running in 159de5710043
Err http://security.debian.org jessie/updates InRelease
Err http://security.debian.org jessie/updates Release.gpg
Could not resolve 'security.debian.org'
Err http://httpredir.debian.org jessie InRelease
Err http://httpredir.debian.org jessie-updates InRelease
Err http://httpredir.debian.org jessie-backports InRelease
Err http://httpredir.debian.org jessie Release.gpg
Could not resolve 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-updates Release.gpg
Could not resolve 'httpredir.debian.org'
Err http://httpredir.debian.org jessie-backports Release.gpg
Could not resolve 'httpredir.debian.org'
Reading package lists...
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/InRelease
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/InRelease
W: Failed to fetch http://security.debian.org/dists/jessie/updates/InRelease
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/InRelease
W: Failed to fetch http://security.debian.org/dists/jessie/updates/Release.gpg Could not resolve 'security.debian.org'
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie/Release.gpg Could not resolve 'httpredir.debian.org'
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-updates/Release.gpg Could not resolve 'httpredir.debian.org'
W: Failed to fetch http://httpredir.debian.org/debian/dists/jessie-backports/Release.gpg Could not resolve 'httpredir.debian.org'
W: Some index files failed to download. They have been ignored, or old ones used instead.
Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package zookeeper
E: Unable to locate package supervisor
E: Unable to locate package dnsutils
The command '/bin/sh -c apt-get update && apt-get install -y zookeeper wget supervisor dnsutils && rm -rf /var/lib/apt/lists/* && apt-get clean && wget -q http://apache.mirrors.spacedump.net/kafka/"$KAFKA_VERSION"/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -O /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz && tar xfz /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz -C /opt && rm /tmp/kafka_"$SCALA_VERSION"-"$KAFKA_VERSION".tgz' returned a non-zero code: 100
Something wrong with getting packages ?
Thanks,
Kevin.
I´m trying to run Kafka and Zookepper using Docker following the steps defined here https://github.com/spotify/docker-kafka
But after run the docker command just as describe I don't understand for what for the export are, and I´m not able to publish anything pointing to localhost:9092` and I receive a timeout response.
This is the code of the publisher that I´m using https://github.com/politrons/Kafka/blob/master/src/main/scala/Producer.scala
Any idea if I´m doing something wrong?
It would be very nice if the container (and so the start up script) would support the creation of topics like the wurstmeister/kafka container does, i.e. "--env KAFKA_CREATE_TOPICS=eventsourcing-traffic-store:1:1".
Great tool for dev!
However I see issues in stability for the container if I run for a long time. Am using this out of the box with no custom configuration.
Is there any way to add more verbose logging easily? The consumers seem to go into a loop trying to get the leader.
Hi,
For reasons I don't yet understand, when I do a docker restart container_id, I can no longer send messages to the topics previously created (the kafka client gets the following: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.).
If I connect to the running container and just stop the kafka process directly, supervisord successfully restarts it and I don't have any issues, so I'm assuming this probably is docker-specific.
Hi,
I'm trying to get your Kafka container running for testing purposes, but I always run into the issue that I get an LeaderNotAvailableException if I follow the run steps as described on your Github page.
bin ./kafka-console-producer.sh --broker-list $KAFKA --topic test
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
Message
[2015-05-07 11:30:13,965] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,027] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,027] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
[2015-05-07 11:30:14,154] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,166] WARN Error while fetching metadata [{TopicMetadata for topic test ->
No partition metadata for topic test due to kafka.common.LeaderNotAvailableException}] for topic [test]: class kafka.common.LeaderNotAvailableException (kafka.producer.BrokerPartitionInfo)
[2015-05-07 11:30:14,166] ERROR Failed to collate messages by topic, partition due to: Failed to fetch topic metadata for topic: test (kafka.producer.async.DefaultEventHandler)
Are there plans to support clustering Kafka via Docker swarm mode, which is new in Docker v1.12.0? With the new swarm you can define services and automatically scale them up or down. Plus docker handles load balancing automatically between instances of the service (default round robin).
docker service create --replicas 1 --name kafka spotify/kafka
docker service scale kafka=3
Intro: https://docs.docker.com/engine/swarm/
Concepts: https://docs.docker.com/engine/swarm/key-concepts/
I'm trying to run the kafka container on a AWS docker machine, when the container starts it successfully starts zookeeper but fails to start kafka. Here is the console output:
> docker-machine create --driver amazonec2 --amazonec2-access-key $AWS_ACCESS_KEY_ID --amazonec2-secret-key $AWS_SECRET_ACCESS_KEY --amazonec2-vpc-id $VPC_ID --amazonec2-region $REGION aws01
> eval $(docker-machine env aws01)
> docker run -p 2181:2181 -p 9092:9092 --env ADVERTISED_HOST=`docker-machine ip \`docker-machine active\`` --env ADVERTISED_PORT=9092 spotify/kafka
/usr/lib/python2.7/dist-packages/supervisor/options.py:296: UserWarning: Supervisord is running as root and it is searching for its configuration file in default locations (including its current working directory); you probably want to specify a "-c" argument specifying an absolute path to a configuration file for improved security.
'Supervisord is running as root and it is searching '
2015-11-20 10:47:03,112 CRIT Supervisor running as root (no user in config file)
2015-11-20 10:47:03,112 WARN Included extra file "/etc/supervisor/conf.d/kafka.conf" during parsing
2015-11-20 10:47:03,112 WARN Included extra file "/etc/supervisor/conf.d/zookeeper.conf" during parsing
2015-11-20 10:47:03,121 INFO RPC interface 'supervisor' initialized
2015-11-20 10:47:03,121 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2015-11-20 10:47:03,122 INFO supervisord started with pid 1
2015-11-20 10:47:04,124 INFO spawned: 'zookeeper' with pid 8
2015-11-20 10:47:04,125 INFO spawned: 'kafka' with pid 9
2015-11-20 10:47:04,171 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:05,172 INFO success: zookeeper entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2015-11-20 10:47:05,173 INFO spawned: 'kafka' with pid 37
2015-11-20 10:47:05,196 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:07,200 INFO spawned: 'kafka' with pid 45
2015-11-20 10:47:07,224 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:10,228 INFO spawned: 'kafka' with pid 53
2015-11-20 10:47:10,251 INFO exited: kafka (exit status 1; not expected)
2015-11-20 10:47:11,252 INFO gave up: kafka entered FATAL state, too many start retries too quickly
Any idea what is causing this and how to fix it?
For me the example with kafka-console-producer.sh
and kafka-console-consumer.sh
does not seem to work. No output from producer and no output from consumer. And when I close the consumer it tells me Processed a total of 0 messages
.
# variables for all terminals
export IP=$(ifconfig eth0 | grep "inet addr:" | cut -d ":" -f2 | cut -d " " -f1)
export KAFKA=$IP:9092
export ZOOKEEPER=$IP:2181
# terminal 1
docker run \
--name kafka \
--restart=always \
-p 2181:2181 \
-p 9092:9092 \
--env ADVERTISED_HOST=$IP\
--env ADVERTISED_PORT=9092 \
spotify/kafka
# terminal 2
docker run -it --rm spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-producer.sh --broker-list $KAFKA --topic test
# terminal 3
docker run -it --rm spotify/kafka /opt/kafka_2.11-0.10.1.0/bin/kafka-console-consumer.sh --zookeeper $ZOOKEEPER --topic test
I'd like to use dynamic port binding (aka docker run -P ...
).
Is there a way to get this working?
I guess the dynamically allocated port must be advertised in zk?
Any other things must be done for that?
THanks!
log.retention.hours and log.retention.bytes provide specification as to how long items are retained and can be useful to control in development/testing environments.
Dear @mattnworb & @mbruggmann could it make sense to archive this repo to make it more clear that it is no longer maintained and does not work with current versions of Kafka (clients).
See also: #102
Thanks a lot for your consideration
Not being that experienced with kafka it would be nice to have a VOLUME exposed for easily setting up a data container.
Does anyone successfully used this on AWS?
Hey guys,
super-thanks for creating this, saved me a TON of time!
I just noticed that the instructions in the README may be slightly improved, by using the following (I was having trouble running this on a VM where I hadn't installed Kafka, so wanted to run both consumer/producer inside two separate containers, still using this image - but avoid using the "hack" of using localhost
everywhere).
I also believe that Kafka will have trouble if the ADVERTISED_HOST
does not reflect the IP of the container.
Anyways, this is how I ran it - please let me know if you would like me to update the README and I'd be happy to submit a PR.
# Launch the "main" Kafka container, this runs the server;
# it also changes the `hostname` to "kafka" and sets an appropriate entry to /etc/hosts:
$ docker run --rm --env ADVERTISED_HOST=kafka --env ADVERTISED_PORT=9092 \
--name kafka -h kafka spotify/kafka
To launch the Producer (this used to fail with a kafka.common.LeaderNotAvailableException
when posting to a topic):
$ docker run --rm -it --name producer --link kafka spotify/kafka \
/opt/kafka_2.11-0.8.2.1/bin/kafka-console-producer.sh --broker kafka:9092 --topic test
similarly, the consumer can be run with:
$ docker run --rm -it --name consumer --link kafka spotify/kafka \
/opt/kafka_2.11-0.8.2.1/bin/kafka-console-consumer.sh --zookeeper kafka:2181 --topic test
Also, I would suggest adding the /opt/kafka_xxx
directory to the container's PATH
in the Dockerfile
(again, happy to send a PR, if you guys want me to).
Hope this helps!
"2.0.0 is the latest release. The current stable version is 2.0.0."
--https://kafka.apache.org/downloads
Hi,
I seem to be unable to use different port mappings for this image in order to run different environments on the same machine. In conjunction with spring-kafka:2.1.4.RELEASE i get the following error:
2018-04-10 19:57:38.191 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 4 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.208 INFO 13318 --- [ Test worker] o.a.kafka.common.utils.AppInfoParser : Kafka version : 1.0.0
2018-04-10 19:57:38.208 INFO 13318 --- [ Test worker] o.a.kafka.common.utils.AppInfoParser : Kafka commitId : aaa7af6d4a11b29d
2018-04-10 19:57:38.228 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 2 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.299 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 5 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.337 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 3 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.406 WARN 13318 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-1, groupId=notifications] Error while fetching metadata with correlation id 6 : {notifications.topic=LEADER_NOT_AVAILABLE}
2018-04-10 19:57:38.445 WARN 13318 --- [ad | producer-1] org.apache.kafka.clients.NetworkClient : [Producer clientId=producer-1] Error while fetching metadata with correlation id 4 : {notifications.topic=LEADER_NOT_AVAILABLE}
...
and so on
docker-compose:
kafka:
container_name: integration_kafka
image: spotify/kafka:latest
ports:
- "9093:9092"
- "2182:2181"
environment:
ADVERTISED_HOST: 127.0.0.1
ADVERTISED_PORT: 9093
TOPICS: notifications.topic
GROUP_ID: notifications
bootstrap.yml:
spring:
kafka:
consumer:
auto-offset-reset: earliest
group-id: notifications
bootstrap-servers:
- 127.0.0.1:9093
Only if i run the container with the default mappings 9092:9092, 2181:2181 and the bootstrap servers set to 127.0.0.1:9092 it seems to work as intended. I'd be very thankful if you could point out my mistake... or care to correct yours ;-)
Hey!
It would be nice to update Dockerfile to publish an image using Kafka 0.9.
Unfortunately this repo is high ranked in search engines.
For those coming here, I am using this repository: https://github.com/lensesio/fast-data-dev
It is very complete and works! 🍺
Hi
I'm newbie to kafka.
I run this docker, and in almost three hours my docker lvs get to 100% making it unavailable
How can I change this docker to an external volume to not exhaust my docker?
Hi,
I have used repository image of kafka from docker hub( docker pull spotify/kafka) and run through Docker tool box(Kitematic) and now kafka is running with the existing setup.No changes I did . My requirement is I need to create a topic and produce the topic and consume the topic . How we can proceed from docker tool box or CLI .
Please advise I am new for docker but I am familiar about normal setup of kafka without docker .
By running it in docker cloud, or my localhost virtual machine.,
the logs show both zookeeper and kafka server running correctly, but I can not create topic by using
"kafka-console-producer.sh --broker-list $KAFKA --topic test"
since in the default bin file, it has no kafka command there.
Also, how we set up when we wanna multiple zookeeper instances running ?
I attempted to put
kafka:
image: spotify/kafka
ports:
- "9092:9092"
- "2181:2181"
hostname: kafka
expose:
- "9092"
- "2181"
environment:
TOPICS: test-topic
But this failed to work, is there something that I'm missing? (I tried with quotes around the topic as well)
Also, if anyone has had any luck using Spring to connect to a topic, I would greatly appreciate some help connecting to the topics.
I am new to kafka and zoo keeper. If I run 3 instances of this docker, will they be related through a network interface. Basically is it equivalent to a cluster of kafka on top of a cluster of zoo keeper.
Please update your start script to avoid failures due to these lines:
https://github.com/spotify/docker-kafka/blob/master/kafka/scripts/start-kafka.sh#L20
We have #listeners now.
Hi, does this image support Kafka cluster? It means that running multiple broker at the same cluster. I do not see any place where can set the configuration for it.
Thanks.
In general, it is not best practice to leave the user as root
for Docker containers, since you buy yourself several security issues with that. I hope you are not running it like this in your production servers :-D
The solution is to set up a user in the Dockerfile
and give it just the necessary permissions and then run Kafka/Zookeeper with these limited permissions.
It comes first on search engine results for 'docker kafka'!
Cannot build docker image.
Error in fetch kafka.consumer.ConsumerFetcherThread$FetchRequest@224dc69c (kafka.consumer.ConsumerFetcherThread)
java.nio.BufferUnderflowException
at java.nio.HeapByteBuffer.get(HeapByteBuffer.java:145)
Why cant I get a topic to be consumed
I am running this on AWS ami
The recent update to 0.8.2.1 broke some scripts because the bin
directory moved. Would be helpful to tag the container version and ensure the install location is consistent for a given tag
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.