Skip to content

Commit 7c49165

Browse files
alexrashedalexrashed
committed
initial commit
0 parents  commit 7c49165

19 files changed

+908
-0
lines changed

‎.env.sample

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
LOCALSTACK_API_KEY=...

‎.gitignore

Lines changed: 138 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,138 @@
1+
# secrets
2+
.env
3+
# localstack volume cache
4+
volume
5+
# build artifacts
6+
build
7+
8+
9+
10+
11+
# Created by https://www.toptal.com/developers/gitignore/api/terraform,pycharm+all
12+
# Edit at https://www.toptal.com/developers/gitignore?templates=terraform,pycharm+all
13+
14+
### PyCharm+all ###
15+
# Covers JetBrains IDEs: IntelliJ, RubyMine, PhpStorm, AppCode, PyCharm, CLion, Android Studio, WebStorm and Rider
16+
# Reference: https://intellij-support.jetbrains.com/hc/en-us/articles/206544839
17+
18+
# User-specific stuff
19+
.idea/**/workspace.xml
20+
.idea/**/tasks.xml
21+
.idea/**/usage.statistics.xml
22+
.idea/**/dictionaries
23+
.idea/**/shelf
24+
25+
# AWS User-specific
26+
.idea/**/aws.xml
27+
28+
# Generated files
29+
.idea/**/contentModel.xml
30+
31+
# Sensitive or high-churn files
32+
.idea/**/dataSources/
33+
.idea/**/dataSources.ids
34+
.idea/**/dataSources.local.xml
35+
.idea/**/sqlDataSources.xml
36+
.idea/**/dynamic.xml
37+
.idea/**/uiDesigner.xml
38+
.idea/**/dbnavigator.xml
39+
40+
# Gradle
41+
.idea/**/gradle.xml
42+
.idea/**/libraries
43+
44+
# Gradle and Maven with auto-import
45+
# When using Gradle or Maven with auto-import, you should exclude module files,
46+
# since they will be recreated, and may cause churn. Uncomment if using
47+
# auto-import.
48+
# .idea/artifacts
49+
# .idea/compiler.xml
50+
# .idea/jarRepositories.xml
51+
# .idea/modules.xml
52+
# .idea/*.iml
53+
# .idea/modules
54+
# *.iml
55+
# *.ipr
56+
57+
# CMake
58+
cmake-build-*/
59+
60+
# Mongo Explorer plugin
61+
.idea/**/mongoSettings.xml
62+
63+
# File-based project format
64+
*.iws
65+
66+
# IntelliJ
67+
out/
68+
69+
# mpeltonen/sbt-idea plugin
70+
.idea_modules/
71+
72+
# JIRA plugin
73+
atlassian-ide-plugin.xml
74+
75+
# Cursive Clojure plugin
76+
.idea/replstate.xml
77+
78+
# SonarLint plugin
79+
.idea/sonarlint/
80+
81+
# Crashlytics plugin (for Android Studio and IntelliJ)
82+
com_crashlytics_export_strings.xml
83+
crashlytics.properties
84+
crashlytics-build.properties
85+
fabric.properties
86+
87+
# Editor-based Rest Client
88+
.idea/httpRequests
89+
90+
# Android studio 3.1+ serialized cache file
91+
.idea/caches/build_file_checksums.ser
92+
93+
### PyCharm+all Patch ###
94+
# Ignore everything but code style settings and run configurations
95+
# that are supposed to be shared within teams.
96+
97+
.idea/*
98+
99+
!.idea/codeStyles
100+
!.idea/runConfigurations
101+
102+
### Terraform ###
103+
# Local .terraform directories
104+
**/.terraform/*
105+
106+
# .tfstate files
107+
*.tfstate
108+
*.tfstate.*
109+
110+
# Crash log files
111+
crash.log
112+
crash.*.log
113+
114+
# Exclude all .tfvars files, which are likely to contain sensitive data, such as
115+
# password, private keys, and other secrets. These should not be part of version
116+
# control as they are data points which are potentially sensitive and subject
117+
# to change depending on the environment.
118+
*.tfvars
119+
*.tfvars.json
120+
121+
# Ignore override files as they are usually used to override resources locally and so
122+
# are not checked in
123+
override.tf
124+
override.tf.json
125+
*_override.tf
126+
*_override.tf.json
127+
128+
# Include override files you do wish to add to version control using negated pattern
129+
# !example_override.tf
130+
131+
# Include tfplan files to ignore the plan output of command: terraform plan -out=tfplan
132+
# example: *tfplan*
133+
134+
# Ignore CLI configuration files
135+
.terraformrc
136+
terraform.rc
137+
138+
# End of https://www.toptal.com/developers/gitignore/api/terraform,pycharm+all

‎.terraform.lock.hcl

Lines changed: 61 additions & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎README.md

Lines changed: 45 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,45 @@
1+
# Fuzzy Movies
2+
## Lambda, Kinesis, Firehose, ElasticSearch, S3
3+
![Screenshot](./docs/screenshot.png)
4+
5+
This Hackathon project is an AWS app consisting of:
6+
- A data ingestion pipeline which allows adding movie data to an ElasticSearch index via:
7+
1. An AWS Lambda function, explosed via a fuction URL.
8+
2. The Lambda function sends the JSON payload to a Kinesis Data Stream.
9+
3. A Kinesis Firehose Delivery Stream forwards the data to an ElasticSearch domain.
10+
- A frontend / website which:
11+
- Has a simple search interface to search for movies in the database.
12+
- The HTML page uses a vanilla JS script to query data using a second Lambda function.
13+
- This Lambda function performs a fuzzy query on the movie index in the ElasticSearch cluster.
14+
15+
## System Overview
16+
![System Overview](./docs/overview.drawio.png)
17+
18+
## Setup
19+
1. Clone this repo and `cd` into its working directory
20+
2. Install the following tools:
21+
- [Terraform](https://www.terraform.io/downloads) (v1.4.5)
22+
- [tflocal](https://github.com/localstack/terraform-local)
23+
- [awslocal](https://github.com/localstack/awscli-local)
24+
3. Start LocalStack in the foreground so you can watch the logs:
25+
```
26+
docker compose up
27+
```
28+
4. Open another terminal window and `cd` into the same working directory
29+
5. Create the resource and trigger the invocation of the lambda:
30+
```
31+
./run.sh
32+
```
33+
34+
# TODO:
35+
- This sample does not yet run on AWS
36+
- Firehose -> ElasticSearch
37+
- Records are not properly delivered to ElasticSearch yet
38+
- Search Lambda -> ElasticSearch
39+
- Lambda needs to sign the HTTP requests to ElasticSearch
40+
- Simplify the S3 website URL in LocalStack
41+
- We need to use http://movie-search.s3.amazonaws.com:4566/index.html instead of the generated output: http://movie-search.s3-website-eu-west-1.amazonaws.com/
42+
- It works with http://movie-search.s3-website.localhost.localstack.cloud/
43+
- HTTPS?
44+
- Due to the function URLs having no proper certificate, we can only use the http version!
45+
- http://movie-search.s3-website.localhost.localstack.cloud:4566/

‎docker-compose.yml

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,20 @@
1+
version: "3.8"
2+
3+
services:
4+
localstack:
5+
container_name: "${LOCALSTACK_DOCKER_NAME-localstack_main}"
6+
image: localstack/localstack-pro # required for Pro
7+
ports:
8+
- "127.0.0.1:4566:4566" # LocalStack Gateway
9+
- "127.0.0.1:4510-4559:4510-4559" # external services port range
10+
- "127.0.0.1:53:53" # DNS config (required for Pro)
11+
- "127.0.0.1:53:53/udp" # DNS config (required for Pro)
12+
- "127.0.0.1:443:443" # LocalStack HTTPS Gateway (required for Pro)
13+
environment:
14+
- DEBUG=${DEBUG-}
15+
- PERSISTENCE=${PERSISTENCE-}
16+
- LOCALSTACK_API_KEY=${LOCALSTACK_API_KEY-} # required for Pro
17+
- DOCKER_HOST=unix:///var/run/docker.sock
18+
volumes:
19+
- "${LOCALSTACK_VOLUME_DIR:-./volume}:/var/lib/localstack"
20+
- "/var/run/docker.sock:/var/run/docker.sock"

‎docs/overview.drawio.png

178 KB
Loading

‎docs/screenshot.png

153 KB
Loading

‎ingest/lambda.py

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,23 @@
1+
import os
2+
import logging
3+
import boto3
4+
import json
5+
6+
logger = logging.getLogger(__name__)
7+
8+
9+
def handler(event, _):
10+
try:
11+
kinesis = boto3.client(
12+
"kinesis", region_name=os.environ["AWS_REGION"], verify=False
13+
)
14+
stream_name = os.environ['STREAM_NAME']
15+
16+
# just directly forward the JSON event body to the Kinesis stream
17+
kinesis.put_record(StreamName=stream_name, Data=json.dumps(event["body"]), PartitionKey="1")
18+
19+
logger.info("Put record in stream %s.", stream_name)
20+
except Exception:
21+
logger.exception("Sending record to kinesis failed.")
22+
23+
return {"body": "Thanks!"}

0 commit comments

Comments
 (0)