Handling failures in the async engine? about open-service-broker-azure HOT 4 CLOSED

From @krancour on October 16, 2017 20:17

This is a duplicate of #10

but the best practice is to make steps idempotent which makes failure handling less important (because the steps can be re-run with minimal negative consequence)

This is true to a very limited extent. There are lots of things that could go wrong that will continue to fail no matter how many times they are retried.

Here's a real world example (for which we should probably open a separate issue, tbh)...

On a normal CI run, I saw provisioning fail because either a UUID or randomly generated name had the misfortune of having the character sequence xbox occur (in that precise order) within it. This actually caused provisioning to fail because ARM seems to disallow names that resemble MSFT brands in any way.

No amount of retrying this would make it succeed. I'm certain there are many, many other unanticipated errors that would suffer from the same fate. As it stands now, there are no retries at all for this reason.

For clarity, the reason we (currently) try to make tasks idempotent isn't due to explicit failure of a task, but rather due to the possibility that the process executing a task (or waiting on a task to complete) just completely tips over and dies (for presumably unrelated reasons, like an OOM, for instance). The reliable queue pattern we use will reassign the outstanding task to another worker and that's the current impetus for creating idempotent tasks.

We still absolutely need to tackle this. I only wanted to add some clarity around the current behavior and design choices that were made very early on.

from open-service-broker-azure.

@krancour sorry for the dupe, and thanks for the added clarity! Feel free to close if appropriate.

from open-service-broker-azure.

From @krancour on October 16, 2017 20:23

No worries on the dupe. I think I'll close #10 instead because this thread has a more valuable level of context in it that #10 now.

from open-service-broker-azure.

Close as async engine was separated into its own project via #560.

from open-service-broker-azure.