Sent notification buffer overrun in production about pushy HOT 15 CLOSED

Can you provide any additional details? How many threads are in your event loop group (if you've customized it), and how many concurrent connections are you using? What was the reason for the rejection? I'm most troubled by this (from your mailing list post):

Seems that pushmanager won't work after error occurred.

Can you be more specific as to what seems to be going wrong? Does it simply stop taking notifications from the public queue? Is there anything in the log output after the error?

Thanks!

from pushy.

I've not customized the number of threads and there is only one concurrent connection to apns.
The reason for the rejection was an invalid token.
I get the apnsConnection error three times, no further log output.

Yes, I think (could not test in particular) that no further notifications are taken from the queue, because no device has received one of the missing notifs.

I'll try to give you some more details tomorrow.
Thank you for your updates on #76

from pushy.

no further notifications are taken from the queue, because no device has received one of the missing notifs.

I don't mean to be difficult, but could you clarify what you mean by "missing notifications?" I think what's happening here is something like this:

1. Assume a sent notification buffer capacity of 100.
2. You send 1,000 notifications.
3. After all 1,000 notifications are sent, the gateway rejects the very first notification.
4. 900 notifications are lost because they have already been pushed out of the sent notification buffer. The 100 still in the buffer get re-enqueued.

@fkoebel please let me know if the above could plausibly explain the problem you're seeing. Regardless, this is obviously a serious problem because we're losing notifications. That said, our options are limited because the APNs gateway never acknowledges successful notifications. Our choices are:

1. Just pick a larger size for the sent notification buffer. This reduces the chances of overrunning the buffer, but at the cost of more memory (and doesn't guarantee that we'll never overrun). This is mostly what we're doing now, although #76 will make it easier to change the buffer size.
2. "Flush" connections by shutting them down with a known-bad notification whenever their sent notification buffer becomes full. This strikes me as reeeeeally inefficient (and may even be viewed as a DoS attack in high-throughput situations), but does guarantee that we'll never lose notifications due to buffer overruns.

One of the other things we're doing in #76 is adding better logging for this case. I'm curious as to how badly we're overrunning the buffer, and that logging will certainly help us understand the issue more clearly (though having somebody lose notifications in production is, obviously, not the way we'd like to get that information).

from pushy.

Yes, that's exactly what I think too.

Just for my comprehension:

1. Assume buffer size of 100
2. Sending the first 100 notifications (assuming sequence number 40 for an invalid notif)
3. Notification buffer gets emptied, because there is obviously no rejection (or did you immediately notice a rejected notif?)
4. Another 100 notifications will be taken from the queue to the buffer
5. Rejected notifiation registered (number 40), searching for sequence number 40 in the buffer, but this notif is not anymore in the buffer -> the error occurred.

Am I right?

from pushy.

You're mostly right. The twist is that we don't completely empty the buffer when it gets full. If we fill the buffer (still assuming a capacity of 100 notifications), then send one more notification, the oldest notification will get bumped from the buffer and the new one will be added. In other words, the buffer will contain notifications 2 through 101.

But otherwise, yes, what you wrote is accurate.

from pushy.

Thank you for your reply making the error more clearly.
I'll try different buffer sizes and will report which size works for me to prevent the error.

from pushy.

I'll try different buffer sizes and will report which size works for me to prevent the error.

Excellent. Thanks kindly.

I know I mentioned on the mailing list that you could edit the hard-coded buffer size in ApnsConnection. If you'll be experimenting with buffer sizes, though, I'd strongly encourage you to use the approach in #76 both for ease of use and for better logging in case something continues to go wrong (and we'll hope it doesn't).

from pushy.

Yes, I forgot to mention that I actually work on configurable_sent_buffer_size branch

from pushy.

For what it's worth, we just merged #76 into master, so please feel free to use that, if you'd like.

from pushy.

@fkoebel Any news on your end? Have you had luck with the changes in #76?

from pushy.

I think #76 offers half of the solution here, and #118 offers the rest. The idea is that you can (after #118 gets merged) set your own sent notification buffer size, and also configure connections to close after a set number of send attempts. As long as the number of send attempts is less than or equal to the sent notification buffer size, we'll never lose notifications to buffer overruns. That said, there are still other ways to lose notifications (i.e. non-graceful shutdowns).

from pushy.

We've hit this this week at Swrve: https://app.getsentry.com/team-swrve/jvm-production/group/22614269/ (I've made the Sentry issue public)

from pushy.

We've hit this this week at Swrve.

Acknowledged. Looking forward to getting the next version of Pushy out the door so people can use #118 if they need it. I'm afraid it's the best workaround we can offer given the limitations of APNs.

from pushy.

Oops, I've just read back over my comment above it sounds like I'm complaining! I just wanted to add more data to help you guys understand the issue 👍

from pushy.

I just wanted to add more data to help you guys understand the issue

No worries! That's definitely the spirit in which it was received. Sorry if I came across as defensive!

from pushy.