You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/utilities/batch.md
+40-29
Original file line number
Diff line number
Diff line change
@@ -52,7 +52,7 @@ journey
52
52
Records expired: 1: Failure
53
53
```
54
54
55
-
This behavior changes when you enable Report Batch Item Failures feature in your Lambda function event source configuration:
55
+
This behavior changes when you enable [ReportBatchItemFailures feature](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting) in your Lambda function event source configuration:
56
56
57
57
<!-- markdownlint-disable MD013 -->
58
58
*[**SQS queues**](#sqs-standard). Only messages reported as failure will return to the queue for a retry, while successful ones will be deleted.
@@ -69,11 +69,11 @@ This behavior changes when you enable Report Batch Item Failures feature in your
69
69
70
70
For this feature to work, you need to **(1)** configure your Lambda function event source to use `ReportBatchItemFailures`, and **(2)** return [a specific response](https://docs.aws.amazon.com/lambda/latest/dg/with-sqs.html#services-sqs-batchfailurereporting){target="_blank" rel="nofollow"} to report which records failed to be processed.
71
71
72
-
You use your preferred deployment framework to set the correct configuration while this utility handles the correct response to be returned.
72
+
Use your preferred deployment framework to set the correct configuration while this utility handles the correct response to be returned.
73
73
74
74
### Required resources
75
75
76
-
The remaining sections of the documentation will rely on these samples. For completeness, this demonstrates IAM permissions and Dead Letter Queue where batch records will be sent after 2 retries were attempted.
76
+
The remaining sections of the documentation will rely on these samples. For completeness, this demonstrates IAM permissions and Dead Letter Queue where batch records will be sent after 2 retries.
77
77
78
78
!!! note "You do not need any additional IAM permissions to use this utility, except for what each event source requires."
79
79
@@ -137,14 +137,18 @@ Processing batches from SQS works in three stages:
137
137
#### FIFO queues
138
138
139
139
When using [SQS FIFO queues](https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/FIFO-queues.html){target="_blank"}, we will stop processing messages after the first failure, and return all failed and unprocessed messages in `batchItemFailures`.
140
-
This helps preserve the ordering of messages in your queue.
140
+
This helps preserve the ordering of messages in your queue.
1.**Step 1**. Creates a partial failure batch processor for SQS FIFO queues. See [partial failure mechanics for details](#partial-failure-mechanics)
147
147
148
+
!!! Note
149
+
Note that SqsFifoPartialProcessor is synchronous using `processPartialResponseSync`.
150
+
This is because we need to preserve the order of messages in the queue. See [Async or sync processing section](#async-or-sync-processing) for more details.
151
+
148
152
### Processing messages from Kinesis
149
153
150
154
Processing batches from Kinesis works in three stages:
@@ -225,7 +229,7 @@ By default, we catch any exception raised by your record handler function. This
225
229
--8<--
226
230
```
227
231
228
-
1. Any exception works here. See [extending BatchProcessor section, if you want to override this behavior.](#extending-batchprocessor)
232
+
1. Any exception works here. See [extending BatchProcessorSync section, if you want to override this behavior.](#extending-batchprocessor)
229
233
230
234
2. Exceptions raised in `record_handler` will propagate to `process_partial_response`. <br/><br/> We catch them and include each failed batch item identifier in the response dictionary (see `Sample response` tab).
231
235
@@ -356,21 +360,29 @@ sequenceDiagram
356
360
<i>Kinesis and DynamoDB streams mechanism with multiple batch item failures</i>
357
361
</center>
358
362
359
-
### Processing messages asynchronously
363
+
### Async or sync processing
360
364
361
-
You can use `AsyncBatchProcessor` class and `asyncProcessPartialResponse` function to process messages concurrently.
365
+
There are two processors you can use with this utility:
362
366
363
-
???+ question "When is this useful?"
364
-
Your use case might be able to process multiple records at the same time without conflicting with one another.
367
+
***`BatchProcessor`** and **`processPartialResponse`** – Processes messages asynchronously
368
+
***`BatchProcessorSync`** and **`processPartialResponseSync`** – Processes messages synchronously
369
+
370
+
In most cases your function will be `async` returning a `Promise`. Therefore, the `BatchProcessor` is the default processor handling your batch records asynchronously.
371
+
There are use cases where you need to process the batch records synchronously. For example, when you need to process multiple records at the same time without conflicting with one another.
372
+
For such cases we recommend to use the `BatchProcessorSync` and `processPartialResponseSync` functions.
373
+
374
+
!!! info "Note that you need match your processing function with the right batch processor"
375
+
* If your function is `async` returning a `Promise`, use `BatchProcessor` and `processPartialResponse`
376
+
* If your function is not `async`, use `BatchProcessorSync` and `processPartialResponseSync`
365
377
378
+
The difference between the two processors in implementation is that `BatchProcessor` uses `Promise.all()` while `BatchProcessorSync` loops through each record to preserve the order.
379
+
380
+
???+ question "When is this useful?"
381
+
366
382
For example, imagine you need to process multiple loyalty points and incrementally save in a database. While you await the database to confirm your records are saved, you could start processing another request concurrently.
367
383
368
384
The reason this is not the default behaviour is that not all use cases can handle concurrency safely (e.g., loyalty points must be updated in order).
369
385
370
-
```typescript hl_lines="1-5 14 28-30" title="High-concurrency with AsyncBatchProcessor"
@@ -391,7 +404,7 @@ Use the `BatchProcessor` directly in your function to access a list of all retur
391
404
392
405
Within your `recordHandler` function, you might need access to the Lambda context to determine how much time you have left before your function times out.
393
406
394
-
We can automatically inject the [Lambda context](https://docs.aws.amazon.com/lambda/latest/dg/typescript-context.html){target="_blank"} into your `recordHandler` as optional second argument if you register it when using `BatchProcessor` or the `processPartialResponse` function.
407
+
We can automatically inject the [Lambda context](https://docs.aws.amazon.com/lambda/latest/dg/typescript-context.html){target="_blank"} into your `recordHandler` as optional second argument if you register it when using `BatchProcessorSync` or the `processPartialResponseSync` function.
You can create your own partial batch processor from scratch by inheriting the `BasePartialProcessor` class, and implementing the `prepare()`, `clean()`, `processRecord()` and `asyncProcessRecord()` abstract methods.
431
+
You can create your own partial batch processor from scratch by inheriting the `BasePartialProcessor` class, and implementing the `prepare()`, `clean()`, `processRecord()` and `processRecordSync()` abstract methods.
<i>Visual representation to bring your own processor</i>
442
453
</center>
443
454
444
-
***`processRecord()`** – handles all processing logic for each individual message of a batch, including calling the `recordHandler` (`this.handler`)
445
455
***`prepare()`** – called once as part of the processor initialization
446
456
***`clean()`** – teardown logic called once after `processRecord` completes
447
-
***`asyncProcessRecord()`** – If you need to implement asynchronous logic, use this method, otherwise define it in your class with empty logic
448
-
449
-
You can then use this class as a context manager, or pass it to `processPartialResponse` to process the records in your Lambda handler function.
457
+
***`processRecord()`** – If you need to implement asynchronous logic, use this method, otherwise define it in your class with empty logic
458
+
***`processRecordSync()`** – handles all processing logic for each individual message of a batch, including calling the `recordHandler` (`this.handler`)
450
459
460
+
You can then use this class as a context manager, or pass it to `processPartialResponseSync` to process the records in your Lambda handler function.
@@ -456,7 +467,7 @@ You can then use this class as a context manager, or pass it to `processPartialR
456
467
457
468
You can use Tracer to create subsegments for each batch record processed. To do so, you can open a new subsegment for each record, and close it when you're done processing it. When adding annotations and metadata to the subsegment, you can do so directly without calling `tracer.setSegment(subsegment)`. This allows you to work with the subsegment directly and avoid having to either pass the parent subsegment around or have to restore the parent subsegment at the end of the record processing.
0 commit comments