Skip to content

Latest commit

 

History

History
107 lines (93 loc) · 3.19 KB

index.md

File metadata and controls

107 lines (93 loc) · 3.19 KB
title tags categories weight description
Self-managed Kafka cluster
kafka
self-managed
11
Using Localstack lambda with self-managed Kafka cluster

Localstack does not currently support AWS MSK out of the box, but you can run your own self-managed Kafka cluster and integrate it with your own applications.

Running self-managed Kafka

You can find the example Docker Compose file which contains a single-noded ZooKeeper and a Kafka cluster and a simple LocalStack setup as well as Kowl, an Apache Kafka Web UI.

  1. Run Docker Compose:

{{< command >}} $ docker-compose up -d {{< / command >}}

  1. Create the Lambda function:

{{< command >}} $ awslocal lambda create-function
--function-name fun1
--handler lambda.handler
--runtime python3.8
--role r1
--zip-file fileb://lambda.zip { "FunctionName": "fun1", "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1", "Runtime": "python3.8", "Role": "r1", "Handler": "lambda.handler", "CodeSize": 294, "Description": "", "Timeout": 3, "LastModified": "2021-05-19T02:01:06.617+0000", "CodeSha256": "/GPsiNXaq4tBA4QpxPCwgpeVfP7j+1tTH6zdkJ3jiU4=", "Version": "$LATEST", "VpcConfig": {}, "TracingConfig": { "Mode": "PassThrough" }, "RevisionId": "d85469d2-8558-4d75-bc0e-5926f373e12c", "State": "Active", "LastUpdateStatus": "Successful", "PackageType": "Zip" } {{< / command >}}

  1. Create an example secret:

{{< command >}} $ awslocal secretsmanager create-secret --name localstack { "ARN": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI", "Name": "localstack", "VersionId": "32bbb8e2-46ee-4322-b3d5-b6459d54513b" } {{< / command >}}

  1. Create an example Kafka topic:

{{< command >}} $ docker exec -ti kafka kafka-topics --zookeeper zookeeper:2181 --create --replication-factor 1 --partitions 1 --topic t1 Created topic t1. {{< / command >}}

  1. Create the event source mapping to your local kafka cluster:

{{< command >}} $ awslocal lambda create-event-source-mapping
--topics t1
--source-access-configuration Type=SASL_SCRAM_512_AUTH,URI=arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI
--function-name arn:aws:lambda:us-east-1:000000000000:function:fun1
--self-managed-event-source '{"Endpoints":{"KAFKA_BOOTSTRAP_SERVERS":["localhost:9092"]}}' { "UUID": "4a2b0ea6-960c-4847-8684-465876dd6dbd", "BatchSize": 100, "FunctionArn": "arn:aws:lambda:us-east-1:000000000000:function:fun1", "LastModified": "2021-05-19T04:02:49+02:00", "LastProcessingResult": "OK", "State": "Enabled", "StateTransitionReason": "User action", "Topics": [ "t1" ], "SourceAccessConfigurations": [ { "Type": "SASL_SCRAM_512_AUTH", "URI": "arn:aws:secretsmanager:us-east-1:000000000000:secret:localstack-TDIuI" } ], "SelfManagedEventSource": { "Endpoints": { "KAFKA_BOOTSTRAP_SERVERS": [ "localhost:9092" ] } } } {{< / command >}}

  1. Aditionally visit http://localhost:8080 for Kowl's UI.