Skip to content

Commit 81f2d69

Browse files
Alexander Schuerendreamorosi
Alexander Schueren
andauthored
docs(idempotency): add batch integration to idempotency docs (#1676)
* add batch integration to idempotency docs * revert unintended IDE formatting changes * chore: fixed imports / highlight --------- Co-authored-by: Andrea Amorosi <[email protected]>
1 parent e1a3452 commit 81f2d69

File tree

2 files changed

+98
-4
lines changed

2 files changed

+98
-4
lines changed

Diff for: docs/snippets/idempotency/workingWithBatch.ts

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
import {
2+
BatchProcessor,
3+
EventType,
4+
processPartialResponse,
5+
} from '@aws-lambda-powertools/batch';
6+
import type {
7+
Context,
8+
SQSBatchResponse,
9+
SQSEvent,
10+
SQSRecord,
11+
} from 'aws-lambda';
12+
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
13+
import {
14+
IdempotencyConfig,
15+
makeIdempotent,
16+
} from '@aws-lambda-powertools/idempotency';
17+
18+
const processor = new BatchProcessor(EventType.SQS);
19+
20+
const dynamoDBPersistence = new DynamoDBPersistenceLayer({
21+
tableName: 'idempotencyTable',
22+
});
23+
const idempotencyConfig = new IdempotencyConfig({
24+
eventKeyJmesPath: 'messageId',
25+
});
26+
27+
const processIdempotently = makeIdempotent(
28+
async (_record: SQSRecord) => {
29+
// process your event
30+
},
31+
{
32+
persistenceStore: dynamoDBPersistence,
33+
config: idempotencyConfig,
34+
}
35+
);
36+
37+
export const handler = async (
38+
event: SQSEvent,
39+
context: Context
40+
): Promise<SQSBatchResponse> => {
41+
idempotencyConfig.registerLambdaContext(context);
42+
43+
return processPartialResponse(event, processIdempotently, processor, {
44+
context,
45+
});
46+
};

Diff for: docs/utilities/idempotency.md

+52-4
Original file line numberDiff line numberDiff line change
@@ -79,10 +79,10 @@ As of now, Amazon DynamoDB is the only supported persistent storage layer, so yo
7979

8080
If you're not [changing the default configuration for the DynamoDB persistence layer](#dynamodbpersistencelayer), this is the expected default configuration:
8181

82-
| Configuration | Default value | Notes |
83-
| ------------------ |:--------------|-----------------------------------------------------------------------------------------|
84-
| Partition key | `id` | The id of each idempotency record which a combination of `functionName#hashOfPayload`. |
85-
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console. |
82+
| Configuration | Default value | Notes |
83+
| ------------------ | :------------ | -------------------------------------------------------------------------------------- |
84+
| Partition key | `id` | The id of each idempotency record which a combination of `functionName#hashOfPayload`. |
85+
| TTL attribute name | `expiration` | This can only be configured after your table is created if you're using AWS Console. |
8686

8787
???+ tip "Tip: You can share a single state table for all functions"
8888
You can reuse the same DynamoDB table to store idempotency state. We add the Lambda function name in addition to the idempotency key as a hash key.
@@ -767,6 +767,54 @@ This means that we will raise **`IdempotencyKeyError`** if the evaluation of **`
767767
}
768768
```
769769

770+
### Batch integration
771+
772+
You can easily integrate with [Batch](batch.md) utility by using idempotency wrapper around your processing function.
773+
This ensures that you process each record in an idempotent manner, and guard against a [Lambda timeout](#lambda-timeouts) idempotent situation.
774+
775+
???+ "Choosing a unique batch record attribute"
776+
In this example, we choose `messageId` as our idempotency key since we know it'll be unique.
777+
Depending on your use case, it might be more accurate [to choose another field](#choosing-a-payload-subset-for-idempotency) your producer intentionally set to define uniqueness.
778+
779+
780+
781+
=== "Integration with batch processor"
782+
783+
```typescript hl_lines="27 31-34 41"
784+
--8<-- "docs/snippets/idempotency/workingWithBatch.ts"
785+
```
786+
787+
=== "Sample event"
788+
789+
```json hl_lines="4"
790+
{
791+
"Records": [
792+
{
793+
"messageId": "059f36b4-87a3-44ab-83d2-661975830a7d",
794+
"receiptHandle": "AQEBwJnKyrHigUMZj6rYigCgxlaS3SLy0a...",
795+
"body": "Test message.",
796+
"attributes": {
797+
"ApproximateReceiveCount": "1",
798+
"SentTimestamp": "1545082649183",
799+
"SenderId": "AIDAIENQZJOLO23YVJ4VO",
800+
"ApproximateFirstReceiveTimestamp": "1545082649185"
801+
},
802+
"messageAttributes": {
803+
"testAttr": {
804+
"stringValue": "100",
805+
"binaryValue": "base64Str",
806+
"dataType": "Number"
807+
}
808+
},
809+
"md5OfBody": "e4e68fb7bd0e697a0ae8f1bb342846b3",
810+
"eventSource": "aws:sqs",
811+
"eventSourceARN": "arn:aws:sqs:us-east-2:123456789012:my-queue",
812+
"awsRegion": "us-east-2"
813+
}
814+
]
815+
}
816+
```
817+
770818
### Customizing AWS SDK configuration
771819

772820
The **`clientConfig`** and **`awsSdkV3Client`** parameters enable you to pass in custom configurations or your own [DynamoDBClient](https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/client/dynamodb/){target="_blank"} when constructing the persistence store.

0 commit comments

Comments
 (0)