Streamline integration of SQS Batch Processing with Sentry

[charlax]

Is your feature request related to a problem? Please describe.

The current behavior reraises exception as SQSBatchProcessingError, which hides the original exception and prevents it from showing up correctly in Sentry:

SQSBatchProcessingError
Not all records processed succesfully. 1 individual errors logged separately below.

Traceback (most recent call last):
  File "/var/task/urllib3/connectionpool.py", line 445, in _make_request
    six.raise_from(e, None)
  File "<string>", line 3, in raise_from
  File "/var/task/urllib3/connectionpool.py", line 440, in _make_request
    httplib_response = conn.getresponse()
  File "/var/task/sentry_sdk/integrations/stdlib.py", line 102, in getresponse
    rv = real_getresponse(self, *args, **kwargs)
  Fi...

Describe the solution you'd like

The best solution would be to be compatible by default with Sentry if it's there (I presume this is one of the most used exception reporting solutions for Python).

Another solution would be to provide a hook into the exception handler, to give us the opportunity to call Sentry.

Describe alternatives you've considered

I looked into subclassing PartialSQSProcessor and overriding _process_record, which is what I might do if you don't think that's worth looking into.

[charlax]

I was actually pretty simple, although it would still be nice to have a no-code integration:

from aws_lambda_powertools.utilities.batch import PartialSQSProcessor
from sentry_sdk import capture_exception


class SQSProcessor(PartialSQSProcessor):
    def failure_handler(self, record: Event, exception: Tuple) -> Tuple:  # type: ignore
        capture_exception()
        logger.exception("got exception while processing SQS message")
        return super().failure_handler(record, exception)  # type: ignore

Btw: I took the existing types, Tuple misses its type parameters.

[to-mc]

Hi @charlax, thanks for raising an issue! The functionality surrounding the SQSBatchProcessingError is a bit tricky for 2 reasons:

1. We have to execute _process_record multiple times, and multiple exceptions can be raised during a single execution. There's no good way (I'm aware of) to raise from multiple exceptions, so we wrap everything into the SQSBatchProcessingError. The "root" exceptions should be logged though since we fixed Exception handling in batch processing doesn't work as expected #275 in v1.10.2.
2. By default, logging statements from within the powertools package are suppressed - as per best practices for libraries. This means we can't log the exceptions individually as they happen without users needing to manually set the package logger.

I think the solution you've found is a good one, arguably cleaner than passing hooks/callbacks around. We could always add something to the docs for folks wanting to implement similar logic, do you think that would help?

[charlax]

Yes, this was actually very simple with the failure_handler, adding this example to the doc would be enough I think! Thanks for your answer!

[heitorlessa]

Hey @charlax Added an example - Could you tell us whether this suffice?

We're also refreshing the navigation it should be easier to find it - Full PR: #308

[charlax]

Yes, this is great! Thanks a lot!

[heitorlessa]

This is now available in 1.11.0 - Literally just launched ;) 🎉