Cannot send messages when the topic is set long time after the broker about librdkafka HOT 4 CLOSED

When "rd_kafka_brokers_add" is called, it will request metadata from Kafka clusters. If the code "sleep(1)" is put after "rd_kafka_brokers_add", the topics information will be ignored when the broker threads receive metadata since "rd_kafka_topic_new" has not been called so far. (Here, the metadata must be received within one second after the request is sent.)

When "rd_kafka_topic_new" is called later, it will not request metadata from Kafka clusters again. Therefore, rdkafka can not send messages because it does not know to which broker the topic corresponds.

I guess you may make a mistake. The upper debug output you pasted above corresponds to the case that you add the code "sleep(1)" while the lower corresponds to the case without the "sleep".

You can see the output "Ignore unknown topic rb_event" in the upper debug infomation since "rd_kafka_topic_new" has not been called at that time. The output "new topic: rb_event" indicates the "rd_kafka_topic_new" is called later.

Solving this issue is easy. Just swap the position of "rd_kafka_topic_new" and "rd_kafka_brokers_add". Call "rd_kafka_topic_new" firstly and then call "rd_kafka_brokers_add". Now the code "sleep(1)" will not affect the result.

from librdkafka.

You are right in both auwzb. I swapped the debug output and I didn't realize I could just set the topic before the brokers. Anyway, I think the correct behavior is you can add a topic anywhere in the code, so I'll let the issue open by the moment.
Thanks a lot for the workaround!

from librdkafka.

Hi, eugpermar. I am glad that my comment helped.
You suggestion requires that rdkafka must request metadata when a topic is added.

from librdkafka.

I will push a fix for this soon.

from librdkafka.