Ack at the start of the pipeline about broadway_rabbitmq HOT 7 CLOSED

I was curious on this topic - e.g. for Kafka, when should we commit the offset so that producer can keep going? Also where to replay the message?

Edit: I realized this is for rabbitmq producer, not broadway

from broadway_rabbitmq.

I don't really know with Kafka as I don't have experience with it :/

from broadway_rabbitmq.

@whatyouhide I think this funtionality would be exclusive to RabbitMQ. RabbitMQ can ackcnowledge the message early on and then put "an empty acknowledger" for the message going forward, so I will move this issue to the RabbitMQ project.

Also, regarding RabbitMQ, I think we should have "multiple ack modes" and allow the user to choose. On start you would do something llike:

ack: :confirm | :confirm_on_receive | :nothing,
nack: :reject_with_requeue | :reject | :confirm | :nothing | ...

But you can also change those modes on the fly. This allows you to control what happens in case of exceptions (the one you set at the module level) and also what happens when you explicitly call fail.

from broadway_rabbitmq.

@josevalim okay, let me get it right so I can help with the implementation. You can really do one of three things if you want to ack messages:

• have no_ack set to true, meaning that RabbitMQ will consider the message acked once it believes it has been delivered to a consumer. I believe we probably want to support this option.
• ack at any point of the pipeline: when the message arrives, or during processing, or after processing but before batching, or in between two operation done in the processing, and so on. Maybe we should provide a way to ack the message explicitly and provide something like ack: :manual when configuring the producer so that we can use the no-op acknowledger in that case.
• have Broadway ack automatically at the end of the pipeline unless the message is failed explicitly.

I am not a fan of the :confirm-based names since in RabbitMQ confirms are something else. Thoughts? How would we explicitly ack the message in your opinion?

from broadway_rabbitmq.

Right, the idea is that we will have two configs (ack | nack) but those values will be copied as default values to the acknowledge data. You will be able to change those values on a message on the fly at any moment. My suggestions for values for ack/nack were just suggestions, I am not familiar with the best names for RabbitMQ, so whatever you propose.

ack at any point of the pipeline

This needs to be a Broadway feature, I would probably work/discuss it separately

have Broadway ack automatically at the end of the pipeline

This will be possible once we implement this issue. The idea is that you can do something like:

message
|> configure_ack(nack: :nothing)
|> failed("explicitly fail due to x")

Let's have a meeting where we can discuss it once I am back home.

from broadway_rabbitmq.

@josevalim sounds good. Ping me tomorrow (Monday) or the day after and we can talk about this :)

from broadway_rabbitmq.

Closing this since we implemented ack_immediately (dashbitco/broadway#108) and configure_ack (dashbitco/broadway#109) in Broadway. I've opened #40 to track implementation of the configure/3 callback in the RabbitMQ producer.

from broadway_rabbitmq.