Skip to content

fix(idempotency): types, docs, and makeIdempotent function wrapper #1579

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 4 commits into from
Jul 5, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 6 additions & 10 deletions packages/idempotency/README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
# Powertools for AWS Lambda (TypeScript) - Idempotency Utility <!-- omit in toc -->


| ⚠️ **WARNING: Do not use this utility in production just yet!** ⚠️ |
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| ⚠️ **WARNING: Do not use this utility in production just yet!** ⚠️ |
| :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| **This utility is currently released as beta developer preview** and is intended strictly for feedback and testing purposes **and not for production workloads**.. The version and all future versions tagged with the `-beta` suffix should be treated as not stable. Up until before the [General Availability release](https://github.com/aws-powertools/powertools-lambda-typescript/milestone/10) we might introduce significant breaking changes and improvements in response to customers feedback. | _ |


Expand All @@ -29,7 +29,7 @@ You can use the package in both TypeScript and JavaScript code bases.
## Intro

This package provides a utility to implement idempotency in your Lambda functions.
You can either use it to wrapp a function, or as Middy middleware to make your AWS Lambda handler idempotent.
You can either use it to wrap a function, or as Middy middleware to make your AWS Lambda handler idempotent.

The current implementation provides a persistence layer for Amazon DynamoDB, which offers a variety of configuration options. You can also bring your own persistence layer by extending the `BasePersistenceLayer` class.

Expand Down Expand Up @@ -59,7 +59,7 @@ The function wrapper takes a reference to the function to be made idempotent as

```ts
import { makeFunctionIdempotent } from '@aws-lambda-powertools/idempotency';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/persistence';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import type { Context, SQSEvent, SQSRecord } from 'aws-lambda';

const persistenceStore = new DynamoDBPersistenceLayer({
Expand All @@ -75,7 +75,7 @@ export const handler = async (
_context: Context
): Promise<void> => {
for (const record of event.Records) {
await makeFunctionIdempotent(proccessingFunction, {
await makeFunctionIdempotent(processingFunction, {
dataKeywordArgument: 'transactionId',
persistenceStore,
});
Expand All @@ -96,7 +96,7 @@ By default, the Idempotency utility will use the full event payload to create an
```ts
import { IdempotencyConfig } from '@aws-lambda-powertools/idempotency';
import { makeHandlerIdempotent } from '@aws-lambda-powertools/idempotency/middleware';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/persistence';
import { DynamoDBPersistenceLayer } from '@aws-lambda-powertools/idempotency/dynamodb';
import middy from '@middy/core';
import type { Context, APIGatewayProxyEvent } from 'aws-lambda';

Expand All @@ -111,10 +111,6 @@ const config = new IdempotencyConfig({
eventKeyJmesPath: 'headers.idempotency-key',
});

const processingFunction = async (payload: SQSRecord): Promise<void> => {
// your code goes here here
};

export const handler = middy(
async (event: APIGatewayProxyEvent, _context: Context): Promise<void> => {
// your code goes here here
Expand Down
6 changes: 1 addition & 5 deletions packages/idempotency/jest.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -14,11 +14,7 @@ module.exports = {
roots: ['<rootDir>/src', '<rootDir>/tests'],
testPathIgnorePatterns: ['/node_modules/'],
testEnvironment: 'node',
coveragePathIgnorePatterns: [
'/node_modules/',
'/types/',
'src/makeFunctionIdempotent.ts', // TODO: remove this once makeFunctionIdempotent is implemented
],
coveragePathIgnorePatterns: ['/node_modules/', '/types/'],
coverageThreshold: {
global: {
statements: 100,
Expand Down
103 changes: 66 additions & 37 deletions packages/idempotency/src/IdempotencyHandler.ts
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import type { AnyFunctionWithRecord, IdempotencyHandlerOptions } from './types';
import type { JSONValue } from '@aws-lambda-powertools/commons';
import type { AnyFunction, IdempotencyHandlerOptions } from './types';
import { IdempotencyRecordStatus } from './types';
import {
IdempotencyAlreadyInProgressError,
Expand All @@ -13,31 +14,57 @@ import { search } from 'jmespath';

/**
* @internal
*
* Class that handles the idempotency lifecycle.
*
* This class is used under the hood by the Idempotency utility
* and provides several methods that are called at different stages
* to orchestrate the idempotency logic.
*/
export class IdempotencyHandler<U> {
private readonly fullFunctionPayload: Record<string, unknown>;
private readonly functionPayloadToBeHashed: Record<string, unknown>;
private readonly functionToMakeIdempotent: AnyFunctionWithRecord<U>;
private readonly idempotencyConfig: IdempotencyConfig;
private readonly persistenceStore: BasePersistenceLayer;
export class IdempotencyHandler<Func extends AnyFunction> {
/**
* The arguments passed to the function.
*
* For example, if the function is `foo(a, b)`, then `functionArguments` will be `[a, b]`.
* We need to keep track of the arguments so that we can pass them to the function when we call it.
*/
readonly #functionArguments: unknown[];
/**
* The payload to be hashed.
*
* This is the argument that is used for the idempotency.
*/
readonly #functionPayloadToBeHashed: JSONValue;
/**
* Reference to the function to be made idempotent.
*/
readonly #functionToMakeIdempotent: AnyFunction;
/**
* Idempotency configuration options.
*/
readonly #idempotencyConfig: IdempotencyConfig;
/**
* Persistence layer used to store the idempotency records.
*/
readonly #persistenceStore: BasePersistenceLayer;

public constructor(options: IdempotencyHandlerOptions<U>) {
public constructor(options: IdempotencyHandlerOptions) {
const {
functionToMakeIdempotent,
functionPayloadToBeHashed,
idempotencyConfig,
fullFunctionPayload,
functionArguments,
persistenceStore,
} = options;
this.functionToMakeIdempotent = functionToMakeIdempotent;
this.functionPayloadToBeHashed = functionPayloadToBeHashed;
this.idempotencyConfig = idempotencyConfig;
this.fullFunctionPayload = fullFunctionPayload;
this.#functionToMakeIdempotent = functionToMakeIdempotent;
this.#functionPayloadToBeHashed = functionPayloadToBeHashed;
this.#idempotencyConfig = idempotencyConfig;
this.#functionArguments = functionArguments;

this.persistenceStore = persistenceStore;
this.#persistenceStore = persistenceStore;

this.persistenceStore.configure({
config: this.idempotencyConfig,
this.#persistenceStore.configure({
config: this.#idempotencyConfig,
});
}

Expand Down Expand Up @@ -69,14 +96,14 @@ export class IdempotencyHandler<U> {
return idempotencyRecord.getResponse();
}

public async getFunctionResult(): Promise<U> {
let result: U;
public async getFunctionResult(): Promise<ReturnType<Func>> {
let result;
try {
result = await this.functionToMakeIdempotent(this.fullFunctionPayload);
result = await this.#functionToMakeIdempotent(...this.#functionArguments);
} catch (e) {
try {
await this.persistenceStore.deleteRecord(
this.functionPayloadToBeHashed
await this.#persistenceStore.deleteRecord(
this.#functionPayloadToBeHashed
);
} catch (e) {
throw new IdempotencyPersistenceLayerError(
Expand All @@ -87,9 +114,9 @@ export class IdempotencyHandler<U> {
throw e;
}
try {
await this.persistenceStore.saveSuccess(
this.functionPayloadToBeHashed,
result as Record<string, unknown>
await this.#persistenceStore.saveSuccess(
this.#functionPayloadToBeHashed,
result
);
} catch (e) {
throw new IdempotencyPersistenceLayerError(
Expand All @@ -108,7 +135,7 @@ export class IdempotencyHandler<U> {
* window, we might get an `IdempotencyInconsistentStateError`. In such
* cases we can safely retry the handling a few times.
*/
public async handle(): Promise<U> {
public async handle(): Promise<ReturnType<Func>> {
let e;
for (let retryNo = 0; retryNo <= MAX_RETRIES; retryNo++) {
try {
Expand All @@ -129,34 +156,36 @@ export class IdempotencyHandler<U> {
throw e;
}

public async processIdempotency(): Promise<U> {
public async processIdempotency(): Promise<ReturnType<Func>> {
// early return if we should skip idempotency completely
if (
IdempotencyHandler.shouldSkipIdempotency(
this.idempotencyConfig.eventKeyJmesPath,
this.idempotencyConfig.throwOnNoIdempotencyKey,
this.fullFunctionPayload
this.#idempotencyConfig.eventKeyJmesPath,
this.#idempotencyConfig.throwOnNoIdempotencyKey,
this.#functionPayloadToBeHashed
)
) {
return await this.functionToMakeIdempotent(this.fullFunctionPayload);
return await this.#functionToMakeIdempotent(...this.#functionArguments);
}

try {
await this.persistenceStore.saveInProgress(
this.functionPayloadToBeHashed,
this.idempotencyConfig.lambdaContext?.getRemainingTimeInMillis()
await this.#persistenceStore.saveInProgress(
this.#functionPayloadToBeHashed,
this.#idempotencyConfig.lambdaContext?.getRemainingTimeInMillis()
);
} catch (e) {
if (e instanceof IdempotencyItemAlreadyExistsError) {
const idempotencyRecord: IdempotencyRecord =
await this.persistenceStore.getRecord(this.functionPayloadToBeHashed);
await this.#persistenceStore.getRecord(
this.#functionPayloadToBeHashed
);

return IdempotencyHandler.determineResultFromIdempotencyRecord(
idempotencyRecord
) as U;
) as ReturnType<Func>;
} else {
throw new IdempotencyPersistenceLayerError(
'Failed to save record in progress',
'Failed to save in progress record to idempotency store',
e as Error
);
}
Expand All @@ -177,7 +206,7 @@ export class IdempotencyHandler<U> {
public static shouldSkipIdempotency(
eventKeyJmesPath: string,
throwOnNoIdempotencyKey: boolean,
fullFunctionPayload: Record<string, unknown>
fullFunctionPayload: JSONValue
): boolean {
return (eventKeyJmesPath &&
!throwOnNoIdempotencyKey &&
Expand Down
6 changes: 2 additions & 4 deletions packages/idempotency/src/errors.ts
Original file line number Diff line number Diff line change
Expand Up @@ -34,10 +34,8 @@ class IdempotencyInconsistentStateError extends Error {}
class IdempotencyPersistenceLayerError extends Error {
public readonly cause: Error | undefined;

public constructor(message: string, cause?: Error) {
const errorMessage = cause
? `${message}. This error was caused by: ${cause.message}.`
: message;
public constructor(message: string, cause: Error) {
const errorMessage = `${message}. This error was caused by: ${cause.message}.`;
super(errorMessage);
this.cause = cause;
}
Expand Down
2 changes: 1 addition & 1 deletion packages/idempotency/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
export * from './errors';
export * from './IdempotencyConfig';
export * from './makeFunctionIdempotent';
export * from './makeIdempotent';
87 changes: 0 additions & 87 deletions packages/idempotency/src/makeFunctionIdempotent.ts

This file was deleted.

Loading