Skip to content

Commit 0a91151

Browse files
HarshCasperwhummer
HarshCasper
and
whummer
authored
add docs for the snowflake emulator (#1)
 Co-authored-by: Waldemar Hummer <[email protected]>
1 parent 470945f commit 0a91151

File tree

44 files changed

+1738
-168
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1738
-168
lines changed

‎assets/scss/_styles_project.scss

+8-8
Original file line numberDiff line numberDiff line change
@@ -405,14 +405,14 @@ table tfoot {
405405
visibility: hidden !important
406406
}
407407

408-
#m-tutorials-li {
409-
.ul-2 {
410-
display: none !important
411-
}
412-
label:before {
413-
display: none !important
414-
}
415-
}
408+
// #m-tutorials-li {
409+
// .ul-2 {
410+
// display: none !important
411+
// }
412+
// label:before {
413+
// display: none !important
414+
// }
415+
// }
416416

417417
#m-referencescoverage-li {
418418
.ul-3 {

‎content/en/_index.html

+1-1
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,6 @@
33
title: Documentation
44
layout: redirect
55
sitemap_exclude: true
6-
redirect: overview
6+
redirect: introduction
77
type: docs
88
---

‎content/en/getting-started/_index.html

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
---
2+
linktitle: Getting Started
3+
title: Getting Started
4+
layout: redirect
5+
sitemap_exclude: true
6+
weight: 2
7+
redirect: getting-started/installation
8+
cascade:
9+
type: docs
10+
---

‎content/en/getting-started/_index.md

-18
This file was deleted.

‎content/en/getting-started/installation/index.md

+82
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
---
2+
title: "Installation"
3+
linkTitle: "Installation"
4+
weight: 1
5+
description: Basic installation guide to get started with the LocalStack Snowflake emulator
6+
---
7+
8+
## Introduction
9+
10+
You can set up the LocalStack Snowflake emulator by utilizing LocalStack's Extension mechanism. There are two methods for installing the LocalStack Snowflake emulator:
11+
12+
1. Using the [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli)
13+
2. Using [Docker Compose](https://docs.docker.com/compose/install/)
14+
15+
This guide provides step-by-step instructions for installing the emulator using both methods.
16+
17+
{{<alert type="info">}}
18+
Before starting, ensure you have a valid `LOCALSTACK_AUTH_TOKEN` to access the LocalStack Snowflake emulator. Refer to the [Auth Token guide](https://docs.localstack.cloud/getting-started/auth-token/) to obtain your Auth Token and specify it in the `LOCALSTACK_AUTH_TOKEN` environment variable.
19+
{{</alert>}}
20+
21+
## `localstack` CLI
22+
23+
To install the LocalStack Snowflake emulator using the `localstack` CLI, execute the following command:
24+
25+
{{< command >}}
26+
$ localstack extensions install localstack-extension-snowflake
27+
{{< / command >}}
28+
29+
Upon successful installation, you should see the output listing the installed extension.
30+
31+
```bash
32+
[20:30:06] Extension successfully installed extensions.py:86
33+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
34+
┃ Name ┃ Summary ┃ Version ┃ Author ┃ Plugin name ┃
35+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
36+
│ localstack-extension-snowflake │ LocalStack Extension: Snowflake │ 0.1.22 │ LocalStack │ snowflake │
37+
└────────────────────────────────┴─────────────────────────────────┴─────────┴────────────┴─────────────┘
38+
```
39+
40+
## Docker Compose
41+
42+
To install the LocalStack Snowflake emulator using Docker Compose, use the `EXTENSION_AUTO_INSTALL` environment variable for automatic extension installation. Create a `docker-compose.yml` file with the specified content.
43+
44+
```yaml
45+
version: "3.8"
46+
47+
services:
48+
localstack:
49+
container_name: "localstack-main"
50+
image: localstack/localstack-pro
51+
ports:
52+
- "127.0.0.1:4566:4566"
53+
- "127.0.0.1:4510-4559:4510-4559"
54+
environment:
55+
- LOCALSTACK_AUTH_TOKEN=${LOCALSTACK_AUTH_TOKEN:?}
56+
- DEBUG=1
57+
- EXTENSION_AUTO_INSTALL=localstack-extension-snowflake
58+
volumes:
59+
- "./volume:/var/lib/localstack"
60+
- "/var/run/docker.sock:/var/run/docker.sock"
61+
```
62+
63+
Start the LocalStack Snowflake emulator with the following command:
64+
65+
{{< command >}}
66+
$ docker-compose up
67+
{{< / command >}}
68+
69+
## Updating
70+
71+
To update the LocalStack Snowflake emulator using the `localstack` CLI, uninstall and install the extension again:
72+
73+
{{< command >}}
74+
$ localstack extensions uninstall localstack-extension-snowflake
75+
$ localstack extensions install localstack-extension-snowflake
76+
{{< / command >}}
77+
78+
For Docker Compose, update the extension by restarting the LocalStack container.
79+
80+
## Next steps
81+
82+
Now that the LocalStack Snowflake emulator is installed, you can use it for developing and testing your Snowflake data pipelines. Refer to our [Quickstart]({{< ref "quickstart" >}}) guide to get started.

‎content/en/getting-started/quickstart/index.md

+140
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,140 @@
1+
---
2+
title: "Quickstart"
3+
linkTitle: "Quickstart"
4+
weight: 2
5+
description: Get started with the LocalStack Snowflake emulator in a few simple steps
6+
---
7+
8+
## Introduction
9+
10+
This guide explains how to set up the LocalStack Snowflake emulator and develop a Python program using the Snowflake Connector for Python (`snowflake-connector-python`) to interact with emulated Snowflake running on your local machine.
11+
12+
## Prerequisites
13+
14+
- [`localstack` CLI](https://docs.localstack.cloud/getting-started/installation/#localstack-cli)
15+
- [LocalStack Snowflake emulator]({{< ref "installation" >}})
16+
- Python 3.10 or later
17+
- [`snowflake-connector-python` library](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-install)
18+
19+
## Instructions
20+
21+
Before you begin, install the LocalStack Snowflake emulator and start the LocalStack container with the following commands:
22+
23+
{{< command >}}
24+
$ export LOCALSTACK_AUTH_TOKEN=<your_auth_token>
25+
$ localstack start
26+
{{< / command >}}
27+
28+
Check the emulator's availability by running:
29+
30+
{{< command >}}
31+
$ localstack extensions list
32+
<disable-copy>
33+
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━┳━━━━━━━━━━━━┳━━━━━━━━━━━━━┓
34+
┃ Name ┃ Summary ┃ Version ┃ Author ┃ Plugin name ┃
35+
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━╇━━━━━━━━━━━━╇━━━━━━━━━━━━━┩
36+
│ localstack-extension-snowflake │ LocalStack Extension: Snowflake │ 0.1.22 │ LocalStack │ snowflake │
37+
└────────────────────────────────┴─────────────────────────────────┴─────────┴────────────┴─────────────┘
38+
</disable-copy>
39+
{{< / command >}}
40+
41+
### Connect to LocalStack Snowflake emulator
42+
43+
Create a new Python file named `main.py` and use the following code to connect to the LocalStack Snowflake emulator:
44+
45+
```python
46+
import snowflake.connector as sf
47+
48+
sf_conn_obj = sf.connect(
49+
user="test",
50+
password="test",
51+
account="test",
52+
database="test",
53+
host="snowflake.localhost.localstack.cloud",
54+
)
55+
```
56+
57+
Specify the `host` parameter as `snowflake.localhost.localstack.cloud` and the other parameters as `test` to avoid connecting to the real Snowflake instance.
58+
59+
### Create and execute a query
60+
61+
Extend the Python program to insert rows from a list object into the emulated Snowflake table. Create a cursor object and execute the query:
62+
63+
```python
64+
print("1. Insert lot of rows from a list object to Snowflake table")
65+
print("2. Creating a cursor object")
66+
sf_cur_obj = sf_conn_obj.cursor()
67+
68+
print("3. Executing a query on cursor object")
69+
try:
70+
sf_cur_obj.execute(
71+
"create or replace table "
72+
"ability(name string, skill string )")
73+
74+
rows_to_insert = [('John', 'SQL'), ('Alex', 'Java'), ('Pete', 'Snowflake')]
75+
76+
sf_cur_obj.executemany(
77+
" insert into ability (name, skill) values (%s,%s) " ,rows_to_insert)
78+
79+
sf_cur_obj.execute("select name, skill from ability")
80+
81+
print("4. Fetching the results")
82+
result = sf_cur_obj.fetchall()
83+
print("Total # of rows :" , len(result))
84+
print("Row-1 =>",result[0])
85+
print("Row-2 =>",result[1])
86+
finally:
87+
sf_cur_obj.close()
88+
```
89+
90+
This program creates a table named `ability`, inserts rows, and fetches the results.
91+
92+
### Run the Python program
93+
94+
Execute the Python program with:
95+
96+
{{< command >}}
97+
$ python main.py
98+
{{< / command >}}
99+
100+
The output should be:
101+
102+
```bash
103+
Insert lot of rows from a list object to Snowflake table
104+
1. Insert lot of rows from a list object to Snowflake table
105+
2. Creating a cursor object
106+
3. Executing a query on cursor object
107+
4. Fetching the results
108+
Total # of rows : 3
109+
Row-1 => ('John', 'SQL')
110+
Row-2 => ('Alex', 'Java')
111+
```
112+
113+
Verify the results by navigating to the LocalStack logs:
114+
115+
```bash
116+
2024-02-22T06:03:13.627 INFO --- [ asgi_gw_0] localstack.request.http : POST /session/v1/login-request => 200
117+
2024-02-22T06:03:16.122 WARN --- [ asgi_gw_0] l.packages.core : postgresql will be installed as an OS package, even though install target is _not_ set to be static.
118+
2024-02-22T06:03:45.917 INFO --- [ asgi_gw_0] localstack.request.http : POST /queries/v1/query-request => 200
119+
2024-02-22T06:03:46.016 INFO --- [ asgi_gw_1] localstack.request.http : POST /queries/v1/query-request => 200
120+
2024-02-22T06:03:49.361 INFO --- [ asgi_gw_0] localstack.request.http : POST /queries/v1/query-request => 200
121+
2024-02-22T06:03:49.412 INFO --- [ asgi_gw_1] localstack.request.http : POST /session => 200
122+
```
123+
124+
### Destroy the local infrastructure
125+
126+
To stop LocalStack and remove locally created resources, use:
127+
128+
{{< command >}}
129+
$ localstack stop
130+
{{< / command >}}
131+
132+
LocalStack is ephemeral and doesn't persist data across restarts. It runs inside a Docker container, and once it’s stopped, all locally created resources are automatically removed. In a future release of the Snowflake emulator, we will provide proper persistence and integration with our [Cloud Pods](https://docs.localstack.cloud/user-guide/state-management/cloud-pods/) feature as well.
133+
134+
## Next steps
135+
136+
You can now explore the following resources to learn more about the LocalStack Snowflake emulator:
137+
138+
- [User Guide]({{< ref "user-guide" >}}): Learn about the LocalStack Snowflake emulator's features and how to use them.
139+
- [Tutorials]({{< ref "tutorials" >}}): Explore tutorials to use the LocalStack Snowflake emulator for local development and testing.
140+
- [References]({{< ref "references" >}}): Find information about the LocalStack Snowflake emulator's configuration, changelog, and function coverage.

‎content/en/introduction/_index.md

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: "Introduction"
3+
linkTitle: "Introduction"
4+
weight: 1
5+
description: >
6+
LocalStack Snowflake emulator allows you to develop and test your Snowflake data pipelines entirely on your local machine!
7+
cascade:
8+
type: docs
9+
aliases:
10+
- /get-started/
11+
hide_readingtime: true
12+
---
13+
14+
[LocalStack](https://localstack.cloud/) is a cloud service emulator that runs in a single container on your laptop or in your CI environment. LocalStack Snowflake emulator replicates the functionality of a real Snowflake instance, allowing you to perform operations without an internet connection or a Snowflake account. This is valuable for locally developing and testing Snowflake data pipelines without incurring costs.
15+
16+
LocalStack Snowflake emulator supports the following features:
17+
18+
* [**Basic operations** on warehouses, databases, schemas, and tables](https://docs.snowflake.com/en/developer-guide/python-connector/python-connector-example)
19+
* [**Storing files** in user/data/named **stages**](https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage)
20+
* [**Snowpark** libraries](https://docs.snowflake.com/en/developer-guide/snowpark/python/index)
21+
* [**Snowpipe** streaming with **Kafka connector**](https://docs.snowflake.com/en/user-guide/data-load-snowpipe-streaming-kafka)
22+
* [**JavaScript and Python UDFs**](https://docs.snowflake.com/en/developer-guide/udf/javascript/udf-javascript-introduction)
23+
* ... and more!
24+
25+
Integrating the LocalStack Snowflake emulator into your existing CI/CD pipeline allows you to run integration tests and identify issues early, reducing surprises during production deployment. Check our [Function Coverage]({{< ref "coverage" >}}) page for a comprehensive list of supported functions.

0 commit comments

Comments
 (0)