Oban notifier eats up postgres connections about oban HOT 6 CLOSED

The error is coming from queue producers using your ecto pool (not the notifier, which uses a dedicated postgrex connection). The second error could be coming from anywhere really, but it seems to be well after the fact when it has gathered too many connections.

Oh well! Thanks for digging through those logs.

I'm going to release v0.11 today. You won't need to run from master for long.

from oban.

Thanks for the report! That is a nasty bug.

I'm fairly sure this has to do with the circuit breaker getting into a loop and recreating the connection. The problem is that I can't recreate it so far—I haven't seen this issue anywhere else so far. Working on recreation and a fix currently.

Out of curiosity, how many nodes are you running?

from oban.

The circuit breaker logs a structured message with the error message that tripped the circuit. Do you have any logs with those error reports?

from oban.

We are running 3 nodes.

Also now, after restarting all of them, the issue seems to have disappeared (for now).
What might also be interesting is that our underlying infrastructure provider seemed to have some network issues today, so I assume the issue only happens when the connections to postgres break.

Now that our infrastructure provider has fixed the issue and after the server restart, we are back to a normal amount of connections.

About the logs I will go and see what I find, but I can't promise anything, since our system is not yet live on production and we are still working on our kibana setup 🙈.

from oban.

@andreasknoepfle Glad to hear it has resolved itself. Hopefully this isn't something you run into again!

In any case, I've made a change to how the circuit breaker handles reconnection. It should prevent this from happening in the future. I say should here because I can't replicate the issue with tests 🤷‍♂

from oban.

So I looked through the logs and just found lots and lots of those:

{source=oban, error=connection not available and request was dropped from queue after 2997ms. You can configure how long requests wait in the queue using :queue_target and :queue_interval. See DBConnection.start_link/2 for more information, message=Circuit temporarily tripped}

And then afterwards it is a sea of those, where I can't say if there is anything valuable in between:

"Postgrex.Protocol (#PID<0.5139.0>) failed to connect: ** (Postgrex.Error) FATAL 53300 (too_many_connections) remaining connection slots are reserved for non-replication superuser connections"

Thanks for looking into this ❤️ 🙏
Going to update to your latest commit now and will report back when this ever happens again :)

from oban.