SnsSqsEnvelope/SnsNotificationModel - Missing fields for FIFO SNS -> FIFO SQS -> Lambda

[plondino]

Expected Behaviour

I have a lambda triggered by a FIFO SQS which is subscribed to a FIFO SNS (see https://docs.aws.amazon.com/sns/latest/dg/fifo-message-delivery.html for this format).

The payload I receive is:

    "Records": [
        {
            "messageId": "4136f084-6249-4a9a-89d8-e824bfa8c237",
            "receiptHandle": "AQEBKimXhFVwYo/BVlwqfx2KHOosqbnMpK8sT3Wo/9VL49dnWMcDgHtllmcYbL02YFakmTCK+1vLCtMNRidlZyLGFv5Z0L8H0Zw/xvdIteyXZiAR7m49u5lFp4f0nfst70Qykasg6dHkoz9s1xmEh03HIRJlNcOWarVF8jneaCoCw3mz77ehoY5WWNaVmw0hbZRlVpu8MLJ2k2RiIPNgJlShASyJ5KD0YTfUQFNC5T7657qiBwlGyFVfZs6BxYKB5Tv8QZy3n7OSjBZTsI7Db35gMlpMmVWcHyBv7XBjf6vdARp42oopStqYbDjcQhEYyl1taPPmfk9pCY4lwxaYQOoGiQ==",
            "body": "{\n  \"Type\" : \"Notification\",\n  \"MessageId\" : \"20aefd0b-ab3a-5911-b009-a6ca92db2664\",\n  \"SequenceNumber\" : \"10000000000000028000\",\n  \"TopicArn\" : \"arn:aws:sns:us-east-1:1234567890:topic.fifo\",\n  \"Message\" : \"...\",\n  \"Timestamp\" : \"2022-10-12T21:54:41.629Z\",\n  \"UnsubscribeURL\" : \"https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:1234567890:topic.fifo\"\n}\n",
            "attributes": {
                "ApproximateReceiveCount": "1",
                "SentTimestamp": "1665611681790",
                "SequenceNumber": "18873140664247791872",
                "MessageGroupId": "123456",
                "SenderId": "AIDAYRRVD2ENU4DSO2WBX",
                "MessageDeduplicationId": "CID1234",
                "ApproximateFirstReceiveTimestamp": "1665611681790"
            },
            "messageAttributes": {},
            "md5OfBody": "0d429e273a5948f84e670ef9beeef0e0",
            "eventSource": "aws:sqs",
            "eventSourceARN": "arn:aws:sqs:us-east-1:1234567890:queue.fifo",
            "awsRegion": "us-east-1"
        }
    ]
}

When I unwrap the SNS message I get:

{
  "Type" : "Notification",
  "MessageId" : "20aefd0b-ab3a-5911-b009-a6ca92db2664",
  "SequenceNumber" : "10000000000000028000",
  "TopicArn" : "arn:aws:sns:us-east-1:1234567890:topic.fifo",
  "Message" : "...",
  "Timestamp" : "2022-10-12T21:54:41.629Z",
  "UnsubscribeURL" : "https://sns.us-east-1.amazonaws.com/?Action=Unsubscribe&SubscriptionArn=arn:aws:sns:us-east-1:1234567890:topic.fifo"
}

I would expect the parser.envelopes.SnsSqsEnvelope or the SnsNotificationModel to be able to validate and unwrap this.

Current Behaviour

I get an error:

Traceback (most recent call last):
  File "/Users/plondino/code/poc/misc/slack_tests.py", line 33, in <module>
    parser.parse(d, model=BusinessEvent, envelope=parser.envelopes.SnsSqsEnvelope)
  File "/Users/plondino/code/poc/misc/venv/lib/python3.9/site-packages/aws_lambda_powertools/utilities/parser/parser.py", line 158, in parse
    return envelope().parse(data=event, model=model)
  File "/Users/plondino/code/poc/misc/venv/lib/python3.9/site-packages/aws_lambda_powertools/utilities/parser/envelopes/sns.py", line 74, in parse
    sns_notification = SnsNotificationModel.parse_raw(body)
  File "pydantic/main.py", line 549, in pydantic.main.BaseModel.parse_raw
  File "pydantic/main.py", line 526, in pydantic.main.BaseModel.parse_obj
  File "pydantic/main.py", line 342, in pydantic.main.BaseModel.__init__
pydantic.error_wrappers.ValidationError: 3 validation errors for SnsNotificationModel
SigningCertUrl
  none is not an allowed value (type=type_error.none.not_allowed)
Signature
  field required (type=value_error.missing)
SignatureVersion
  field required (type=value_error.missing)

I found #265 which seemed similar but in this case the fields are actually missing in the SNS I receive in AWS, not just renamed.

Code snippet

from aws_lambda_powertools.utilities import parser
parser.parse(d, model=MyModel, envelope=parser.envelopes.SnsSqsEnvelope)

Possible Solution

No response

Steps to Reproduce

See above

AWS Lambda Powertools for Python version

latest

AWS Lambda function runtime

3.9

Packaging format used

PyPi

Debugging logs

No response

[boring-cyborg]

Thanks for opening your first issue here! We'll come back to you as soon as we can.
In the meantime, check out the #python channel on our AWS Lambda Powertools Discord: Invite link

[heitorlessa]

thanks for raising that to us @plondino! That's because SNS FIFO changed their contract from non-FIFO thus the breakage. I'll create a PR marking these fields as Optional, make a patch release to unblock you ASAP, and create an issue to create two FIFO Models (SNS/SQS) to do this properly in the future.

cc @ran-isenberg for FYI

[heitorlessa]

Until this is released, you can also optionally set Raw Message Delivery in SNS subscription to SQS. That will configure SNS to only send the body payload to SQS with no metadata - from here you only need to use SQS Model

[heitorlessa]

@plondino we're making a release now (v1.31.1) to include this fix. We've also recreated SNS FIFO + SQS FIFO + Lambda to ensure there wasn't anything else missing for the validation.

Lambda Layer v39 should be available in all regions in approximately 15m

[github-actions]

This is now released under 1.31.1 version!