`-e` flag ignored in case of many active partitions about kcat HOT 8 CLOSED

A

Huhm, yeah, so when EOF is reached for a partition the internal counter of how many partitions have reached EOF is increased, but kafkacat doesn't actually stop consuming that partition, so in your case where the partitions are all active it will most likely continue consuming & printing messages for that partition. This should be considered a bug.

B

However, if this happens for all the partitions the exit-on-EOF check should trigger, even if some are back to consuming, so this should work as expected.
On the other hand, with active partitions, the chance that EOF is actually reached depends on if kafkacat is faster than the sum of producers, and as you say might have a low chance.

So fixing the bug (stop consuming when EOF is reached) should sort this issue out according to A, but the use of -e for active partitions might be somewhat limited according to B.

from kcat.

Right. At least it would be great to fix A)

One possible (but probably overkill) solution for B) is to have another flag that would terminate not by timeout (which might not happen ever, in case producer is faster) but once "last" offset is reached (determined upon start).

from kcat.

Re B: ah, you mean like a -c <receive_mgs_cnt> ?
E.g., stop consuming after this many messages received.

That option is currently global (not per partition) though.

Can you elaborate on your use case?

from kcat.

Re B: ah, you mean like a -c <receive_mgs_cnt>

No, actually. AFAIU, B) is about the case when partition grows faster than kafkacat can read from it. In this case -e is difficult to implement only using timeout. We also might need to save latest offsets and stop once we reach them. At least, this is what I expect from -e. However, there might be someone, for whom -e means reading until all partition timeouts are reached (topic fully read at some point in time). So, hypothetically, another flag might be considered, like -E, stop by reaching initial offsets, and -e stop when all partitions timeout.

Actually, the very latter option would be kinda weird. Imagine some partitions are slower than others. If we stop reading by timeout some partitions would stop sooner, while other would keep going much longer. This can be confusing.

So, to my mind, reading all latest offsets upfront, and stop once they are reached would be the most straightforward option.

from kcat.

Okay, I understand now what you mean by initial offset. The EOF offset at the time kafkacat was started.
This is a bit tricky because this information is not available to the application (apart from in the stats, but that's a bit slow).

Is the use for this only for -e to work when producers are too fast?

from kcat.

This is a bit tricky because this information is not available to the application (apart from in the stats, but that's a bit slow).

Can't kafkacat use OffsetRequest?
http://goo.gl/fUHtnN

Is the use for this only for -e to work when producers are too fast?

Yes, which is probably why fixing A) is more than enough for now, until some real use case for this come up.

from kcat.

The underlying protocol requests are not exposed by librdkafka, kafkacat doesn't really know anything about the protocol. But I'll look into an API for acquiring the offsets for a topic and partition.

Fair enough, I'll fix the -e message count (A).

from kcat.

This should be fixed now, please give it a shot.

from kcat.