Merge pull request #1 from k-edge/DT-1179-localstack

DT-1171, DT-1179 localstack in docker-compose.yml and awslocal in req…

Author: SteffiAtFactoryPal

README.md

@@ -1,5 +1,47 @@
-* [] add option for transformation
-  * json -> parquet
-  * parqeut -> json
-* [] add option for moving data
-* [] add option for copying data
+# Datalake Batch Processor
+
+## Development
+
+### LocalStack
+
+> **_NOTE:_**
+> For easier development, we recommend using [awscli-local](https://github.com/localstack/awscli-local).
+> It will automatically recognize the LocalStack instance and use it as the default AWS endpoint.
+> Therefore, it has been added to the `requirements.txt`. The documentation below assumes the existence of the `awslocal` command.
+
+You will find a `docker-compose.yml` file in the `localstack` directory.
+This file will start a LocalStack instance with a configured S3 bucket.
+
+Before you run it, make sure the `localstack/init-aws.sh` script is executable:
+(see: [LocalStack Init Hooks](https://docs.localstack.cloud/references/init-hooks/))
+
+```bash
+chmod +x ./localstack/init-aws.sh
+```
+
+To test if LocalStack with the S3 instance is running, you can run the following command:
+
+```bash
+awslocal s3api list-buckets
+```
+
+which should print a list of all existing S3 buckets in your local environment as JSON:
+
+```json
+{
+    "Buckets": [
+        {
+            "Name": "my-bucket",
+            "CreationDate": "2025-02-17T13:06:48+00:00"
+        }
+    ]
+}
+```
+
+You should also be able to list all the files in the bucket:
+
+```bash
+aws --endpoint-url=http://localhost:4566 s3 ls s3://my-bucket/
+```
+
+which should contain an example parquet files.

docker-compose.yml

@@ -0,0 +1,14 @@
+services:
+  localstack:
+    container_name: "${LOCALSTACK_DOCKER_NAME:-localstack-main}"
+    image: localstack/localstack
+    ports:
+      - "127.0.0.1:4566:4566"            # LocalStack Gateway
+      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
+    environment:
+      - DEBUG=1
+    volumes:
+      - "./example.parquet:/etc/localstack/example.parquet"
+      - "./init-aws.sh:/etc/localstack/init/ready.d/init-aws.sh"  # ready hook
+      - "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
+      - "/var/run/docker.sock:/var/run/docker.sock"

example.parquet

@@ -0,0 +1,7 @@
+#!/bin/sh
+
+echo "Initializing localstack s3"
+awslocal s3 mb s3://my-bucket
+
+echo "Put example file"
+awslocal s3 cp /etc/localstack/example.parquet s3://my-bucket/

init-aws.sh

@@ -1 +1,2 @@
-pyspark
+pyspark
+awscli-local