Support for queue bindings about broadway_rabbitmq HOT 9 CLOSED

hi @msaraiva and @narrowtux yes, is recommended to have a separate connection for both consuming and publishing messages to rabbitmq (e.g having 1 for producers and 1 for consumers as described here https://www.cloudamqp.com/blog/2018-01-19-part4-rabbitmq-13-common-errors.html) then, you can use the producers connection for setting up the queues for you

from broadway_rabbitmq.

side note: creating queues automatically has its own advantages and disadvantages, for example what we have done in the past is having a GenServer with :transient and :ignore to setup the queues for us once, see https://github.com/esl/ex_rabbit_pool/blob/master/lib/worker/setup_queue.ex , but it only works with durable queues, because if you use non_durable queues, and there is a network partition or disconnect then you lost all the queues from the rabbit side and you are not able to re-create them on your side without some extra logic for handling edge cases

from broadway_rabbitmq.

Hi @narrowtux, thanks for reporting this issue. I have a couple of questions:

• Is this problem reported somewhere you know so I can get more details? Is it related to the erlang amqp client? It would be great to get more context before discussing any changes to the API.
• Does this happens only when sharing the same connection/channel to publish and consume messages?
• I'm still not sure if Broadway should handle those bindings. Aren't they necessary only when publishing messages? Broadway only consumes messages.

I might be missing something here, so forgive me if that's the case.

Thanks!

from broadway_rabbitmq.

The problem was hard to reproduce on other machines, and this was a relatively simple fix that works well.

To give more details about what we were doing:

We have a hierarchy of folders and this hierarchy is reflected in rabbitmq using one exchange per folder which subscribes to each child-folder-exchange.

At first, we only set up the exchanges and the bindings when we created new folders, but that lead to the problem that messages just wouldn't appear in queues bound to the exchanges, we confirmed that the exchanges, queues and bindings were set up correctly using the RabbitMQ management interface.

We then switched to the approach that all parts of the route should be re-declared every time we start consuming a queue. So we have functions that create all dependent exchanges and then the new queue and add bindings to the correct exchanges. This works recursively until the exchange where the message is published is reached.

After that, the routing has worked perfectly.

To answer your specific questions:

Is this problem reported somewhere you know so I can get more details? Is it related to the erlang amqp client? It would be great to get more context before discussing any changes to the API.

We tried reporting the problem on the rabbitmq slack but as it's really hard to describe and reproduce, nobody replied.

Does this happens only when sharing the same connection/channel to publish and consume messages?

Our application opens one connection but uses one channel per queue.

I'm still not sure if Broadway should handle those bindings. Aren't they necessary only when publishing messages? Broadway only consumes messages.

I thought that because this library offers to supply declare options, it would make sense because if the queue that broadway would consume from doesn't exist before it calls declare, it has to set up bindings to exchanges as well.

from broadway_rabbitmq.

@narrowtux thanks for the detailed information.

Our application opens one connection but uses one channel per queue.

Ok. But does it use the same connection to publish and consume? If so, this could be related. I believe it's recommended to use separate connections for publishing and consuming.

@josevalim, @sescobb27, I'd love to hear your thoughts on this one.

from broadway_rabbitmq.

@sescobb27 thanks! That was helpful.

Due to the fact that the connections/channels created by ExRabbitPool can also be used for publishing, I can see why you need something like ExRabbitPool.Worker.SetupQueue to set up the queues and bindings on startup. However, broadway doesn't share any connection and only consumes messages.

I thought that because this library offers to supply declare options, ...

@narrowtux, yeah, I see what you mean. Maybe we shouldn't supply declare options at all. I believe most scenarios for broadway requires that the queue already exists so we probably don't need to declare it and just let it crash if it doesn't.

What do you guys think?

from broadway_rabbitmq.

from broadway_rabbitmq.

@msaraiva yeah exactly, I agree that in the context of Broadway i would let it crash if the queue isn't already there, we have the SetupQueue stuff only for users convenience so is optional (users need to include that setup queue in their supervision tree and pass configuration to it e.g ExRabbitPool.Worker.SetupQueue.start_link({pool_id, queues_config}))

from broadway_rabbitmq.

Great! Let's remove it.

from broadway_rabbitmq.