-o when consuming from multiple partitions? about kcat HOT 10 CLOSED

It applies to all of the partitions, which is highly impractical if it werent for the symbolic offsets:

• beginning
• end
• -NNNN (tail)
• stored

from kcat.

So, it doesn’t work with manually entered offsets? Or, how does stored work with kafkacat? Is there a way to tell it to store and read offsets in a directory (like kafkatee?)

On Oct 16, 2014, at 12:18 PM, Magnus Edenhill [email protected] wrote:

It applies to all of the partitions, which is highly impractical if it werent for the symbolic offsets:

beginning
end
-NNNN (tail)
stored
—
Reply to this email directly or view it on GitHub.

from kcat.

It doesnt work with absolute offsets, no, or.. it works in the way that it'll use that offset for all partitions ,which is probably wrong.

Use -X topic.offset.store.path=dir to specify a directory.

I'm guessing you want something like:
kafkacat .... -t muh_topic -o 192 -p 0 -o 935 -p 1 -o 512 -p 2

• partition 0: offset 192
• partition 1: offset 935
• partition 2: offset 512

right?

from kcat.

Ah! offset.store.path will work just fine, I think. Just tested this and I think it works.

Question, if I use -e, or -c , will the offset files be written to when kafkacat exits? Or will they only be written to every auto.commit.interval.ms?

On Oct 16, 2014, at 1:07 PM, Magnus Edenhill [email protected] wrote:

It doesnt work with absolute offsets, no, or.. it works in the way that it'll use that offset for all partitions ,which is probably wrong.

Use offset.store.path to specify a directory.

I'm guessing you want something like:
kafkacat .... -t muh_topic -o 192 -p 0 -o 935 -p 1 -o 512 -p 2

partition 0: offset 192
partition 1: offset 935
partition 2: offset 512
right?

—
Reply to this email directly or view it on GitHub.

from kcat.

Both, every commit.interval.ms and at shutdown.

from kcat.

What happens if SIGINT (ctrl-C?)

from kcat.

It will exit cleanly (and commit).

from kcat.

Huhm, maybe it wont, it will destroy rdkafka but doesnt wait for completion. I should fix that.

from kcat.

Do you desire the per-partition -o offset switch?

from kcat.

No, I don’t think so… That might be useful for debugging now and then, but the stored offset files should do just fine.

On Oct 17, 2014, at 3:27 AM, Magnus Edenhill [email protected] wrote:

Do you desire the per-partition -o offset switch?

—
Reply to this email directly or view it on GitHub.

from kcat.