Kafka indexing service duplicate key exception in druid_pendingSegments table about druid HOT 7 OPEN

Thanks for reporting this issue, @baaaaast !
I will try to debug this and get back to you.

from druid.

A couple days ago we upgraded to v30. Started to notice this error too. It turned out there is increased number of failed tasks. Now thinking of rolling back to v28, most because of kafka ingestion.

from druid.

A couple days ago we upgraded to v30. Started to notice this error too. It turned out there is increased number of failed tasks. Now thinking of rolling back to v28, most because of kafka ingestion.
Did you try to set the property druid.indexer.tasklock.batchSegmentAllocation=false to the overlord ?

from druid.

ruid.indexer.tasklock.batchSegmentAllocation=false

No we did not yet. Wondering how (much) it can hit the ingestion performance.

from druid.

I haven't seen any difference at all: no lag for millions of messages consumed and no extra load on the overlord side.
I just don't have any more failed tasks

from druid.

I haven't seen any difference at all: no lag for millions of messages consumed and no extra load on the overlord side. I just don't have any more failed tasks

Thanks, sure we'll try out and get back here. For the context, normally we ingest few millions messages per second.

from druid.

Setting druid.indexer.tasklock.batchSegmentAllocation=false fixed the duplicate key error.
Looking forward to the complete resolution of this github issue 👍

from druid.