Feature request: On batch processing, fill in processor result even if BatchProcessingError is raised about powertools-lambda-python HOT 5 OPEN

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

from powertools-lambda-python.

Thanks for raising this @nico00.

I guess it depends on what you're trying to do with them – Lambda will auto redrive to reprocess these messages at a service level.

from powertools-lambda-python.

That's correct but in such case Lambda is limited to two retries, while DynamoDb stream allows up to 10,000 retries. On the other side I see no cons in filling in processor result just before raising BatchProcessingError.

from powertools-lambda-python.

hey @nico00, please allow me to ask some clarifying questions

BatchProcessingError is raised when all records failed to be processed. In such case, processor response appears empty, as all records have been successfully processed.

It's technically a Lambda invocation failure, as recommended by the Lambda team. The Lambda Poller picks up the error and considers the entire batch a failure, there is no empty response in this case.

Are you experiencing an empty response instead of a BatchProcessingError? If so, it'd be a bug/regression on our side.

This would give the programmer the freedom to decide what to do according to various business cases.

Would you be able to expand with one or more examples to help us picture this better?

I'm trying to understand whether you want to intercept a BatchProcessingError - like you can with the context manager today - or something else entirely?

Thanks a lot!

from powertools-lambda-python.

Also, before I forget, thank you for creating a feature request :) We always appreciate hearing from customers and learning what additional use cases can be unblocked (or made easier!) for everyone

from powertools-lambda-python.