KinesisFirehoseResponse missing Metadata field

[chris-smith-zocdoc]

Describe the feature

The metadata field of KinesisFirehoseResponse is used for the dynamic partitioning feature in Kinesis Firehose

Reference implementation in golang https://github.com/aws/aws-lambda-go/blob/1ba09afd7523e0d5a8a2e2fb58ddb8ec8b92ebb7/events/firehose.go#L36-L41 added in aws/aws-lambda-go#395

Use Case

Used to extract partition keys/values for dynamic partitioning feature in Kinesis Firehose

Proposed Solution

Add KinesisFirehoseResponseRecordMetadata class with field partitionKeys as Dictionary<string, string>

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

AWS .NET SDK and/or Package version used

Amazon.Lambda.KinesisFirehoseEvents 2.1.0

Targeted .NET Platform

.net 6

Operating System and version

AmazonLinux

[ashishdhingra]

@chris-smith-zocdoc Thanks for the PR. Please advise if you have tested this end-to-end in actual scenario.

[chris-smith-zocdoc]

@ashishdhingra I have not yet

[ashishdhingra]

@ashishdhingra I have not yet

Somehow looking at the example at https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html, the data type of the metadata entry doesn't appear to be string, even though Go SDK models it as string. So would appreciate if end-to-end testing could be done before we accept the PR.

EDIT: It appears to be JQ expression based on information at https://aws.amazon.com/blogs/big-data/kinesis-data-firehose-now-supports-dynamic-partitioning-to-amazon-s3/. However, end-to-end testing is needed.

[chris-smith-zocdoc]

@ashishdhingra The JQ expression isn't relevant to this pr, thats an alternative way of providing the partitioning data without the transformation lambda.

I just confirmed this pr works as intended in our test account.

My test lambda returns the partition information like this

string eventType = ...

output.Metadata = new KinesisFirehoseResponse.FirehoseResponseRecordMetadata
{
        PartitionKeys = new Dictionary<string, string>
        {
            {"event_type", eventType},
        }
};

Relevant section of cdk configuration for CfnDeliveryStream.extendedS3DestinationConfiguration

prefix: "event_type=!{partitionKeyFromLambda:event_type}/year=!{timestamp:yyyy}/month=!{timestamp:MM}/day=!{timestamp:dd}/hour=!{timestamp:HH}/",
dynamicPartitioningConfiguration: {
   enabled: true
},
processingConfiguration: {
    enabled: true,
    processors: [{
      type: 'Lambda',
      parameters: [
        {
          parameterName: 'LambdaArn',
          parameterValue: functionArn,
        }
      ],
    }],
  },

Here event_type comes from the transformation lambda (push in my example), the rest is using variables made available by firehose

Which results in a file under the path

s3://<bucket_name>/event_type=push/year=2022/month=10/day=27/hour=00/<filename>.gz

[chris-smith-zocdoc]

@ashishdhingra I've completed end to end testing, can you please take a look?

[normj]

PR has been merged and released in version 2.2.0 of Amazon.Lambda.KinesisFirehoseEvents. Thanks for the PR!

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.