Use a shared connection among Broadway producers about broadway_rabbitmq HOT 8 CLOSED

Hi @msaraiva sure, perfect, let me know if you need something, we are willing to help, the pooling strategy is not required tho, it is not an application, is just a library, so if you need to only use the RabbitConnection module without the pooling it will also work and is also really well tested, if not, we can help you implement this feature on Broadway.

from broadway_rabbitmq.

I wonder if it's worth making the supervision tree more complicated to save a few MB in memory. As far as I can remember, the RabbitMQ documentation states that after you hit around 50 number of concurrent connections/channels you stop getting any further improvement. So we're talking about 5MB. In my tests, this number is even lower (around 20). So I believe it's an acceptable tradeoff. Maybe we should wait until we have some evidence that this change would bring some relevant improvements. What do you guys think?

from broadway_rabbitmq.

I think we should close this as now we have functionality about custom opts per producer. We could introduce a separate producer that shares it, but it is definitely not the focus ATM.

from broadway_rabbitmq.

Hi, at Erlang Solutions we have been working on a connection pooling strategy for RabbitMQ called ExRabbitPool which handles the case you are pointing to.

We have a pool of connections for distributing the load across multiple RabbitMQ connections, each connection has it's own pool of channels which we link to. That way we can handle crashes by spawning a new channel.

Also we monitor clients so if a client owning a channel crash we drop that channel and create a new one (channels are stateful so they shouldn't be reused), we use this module RabbitConnection for that purpose. If this sounds good to you, you can check ExRabbitPool, and we can work implementing this on Broadway or just adapting our library to Broadway, whatever is better.

from broadway_rabbitmq.

Hi @sescobb27 !

Thanks for the helpful information! I'll definitely take a look at ExRabbitPool. The pool idea sounds interesting but since Broadway producers are already a pool by themselves, it would be nice if we don't have a pool (broadway producers) calling another pool. I'll take a closer look at ExRabbitPool code and see if I can get some insights from it.

Thanks again!

from broadway_rabbitmq.

Hey @msaraiva, I've been taking a look at this. I think the ideal strategy would be to have a small pool of connections (to improve resilience in case some connections go down and to improve TCP throughput), each maintaining a few channels open. Each Broadway producer should be tied to a channel. For example, if I want a RabbitMQ producer with 50 stages, I could have 50 channels (one per producer) spread over 5 connections with 10 channels open per connection. Wdyt?

I've been thinking of a good API for this and I'm not sure because right now we rely on Broadway to start the producers. I think a good compromise could be to require users to start the connection pool themselves and pass it down when starting the producers:

# In my supervision tree:
children = [
  # ...,
  {BroadwayRabbitMQ.ConnectionPool, name: MyConnectionPool},
  MyBroadway
]

# In the Broadway pipeline:
{BroadwayRabbitMQ.Producer,
 queue: ...,
 connection_pool: MyConnectionPool}

This could also retain compatible behaviour because it would mean that if you don't pass the :connection_pool option, each producer stage will open its own connection.

However, I haven't thought about how this would affect the supervision tree. I think that if a connection goes down, all channels/producers attached to it should go down too. Thoughts?

from broadway_rabbitmq.

We want to avoid shutting down producers as much as possible, so if no state gets corrupted, we should simply reconnect.

from broadway_rabbitmq.

I think the problem here might be that we can't use a single connection for all of the producer stages (because of scalability + reliability) but at the same time, I could easily spawn 100 producer stages but 100 respective RabbitMQ TCP connections might be a bit much. Channels kind of mediate that. I'm fine with one connection per producer stage though, maybe it's just a matter of good docs :)

from broadway_rabbitmq.