Broke Kafka replication when requesting ack != -1 and a message larger than 1000000 bytes about librdkafka HOT 6 CLOSED

Make sure to configure message.bytes.max equally on both brokers and producers, that means providing .. -X "message.max.bytes=NNNNNNNN" .. command line argument to rdkafka_performance.

In code you would use rd_kafka_conf_set(rk_conf, "message.bytes.max", "NNNNNNNN", ...)

from librdkafka.

Ok, but why would all subsequent runs of the binary fail thereafter?

On Friday, January 3, 2014, Magnus Edenhill wrote:

Make sure to configure message.bytes.max equally on both brokers and
producers, that means providing .. -X "message.max.bytes=NNNNNNNN" ..command line argument to rdkafka_performance.

In code you would use rd_kafka_conf_set(rk_conf, "message.bytes.max",
"NNNNNNNN", ...)

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-31510241
.

from librdkafka.

It sounds to me that one or more of your brokers, but not all, have failed in some way.
Thats why required.acks = 1 works - only the receiving broker will ack it, but -1 doesnt - all ISRs will need to ack it.

Did you change message.bytes.max on ALL brokers in your cluster and restart them?
It would be fair to guess that mismatching max message sizes would frustrate a cluster while syncing messages.

from librdkafka.

I'll have my colleague check the broker config. However, since new topics
on this cluster are fine until I run the procedure mentioned, I think
all brokers
are at least up and running- seems to be on a per topic basis.

On Friday, January 3, 2014, Magnus Edenhill wrote:

It sounds to me that one or more of your brokers, but not all, have failed
in some way.
Thats why required.acks = 1 works - only the receiving broker will ack it,
but -1 doesnt - all ISRs will need to ack it.

Did you change message.bytes.max on ALL brokers in your cluster and
restart them?
It would be fair to guess that mismatching max message sizes would
frustrate a cluster while syncing messages.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-31524132
.

from librdkafka.

Any news on this?

from librdkafka.

At this point I'd say just close it. Either I did something wrong with the
max msg size locally or there was something else with the broker. will
reopen if I encounter it again.

On Wed, Jan 29, 2014 at 1:00 AM, Magnus Edenhill
[email protected]:

Any news on this?

Reply to this email directly or view it on GitHubhttps://github.com//issues/48#issuecomment-33558841
.

from librdkafka.