|
2 | 2 | "cells": [
|
3 | 3 | {
|
4 | 4 | "cell_type": "markdown",
|
5 |
| - "id": "homeless-registration", |
| 5 | + "id": "hairy-entrepreneur", |
6 | 6 | "metadata": {},
|
7 | 7 | "source": [
|
8 | 8 | "# Running on a cluster\n",
|
9 | 9 | "\n",
|
10 | 10 | "We'll use a Dask cluster in the cloud—in this case, using [Coiled](https://coiled.io/)—to use many machines to process the data in parallel. We can also run in a data center near* where the data is stored for better performance.\n",
|
11 | 11 | "\n",
|
12 |
| - "\\*_these Sentinel-2 COGs are in AWS's `us-west-2`; Coiled currently only offers `us-west-1`. But it's close enough._" |
| 12 | + "If you use Coiled (which is both easy to use, and currently free!), you can set `software=\"gjoseph92/stackstac\"` to get a software environment where the latest version of `stackstac` is already installed.\n", |
| 13 | + "\n", |
| 14 | + "_*these Sentinel-2 COGs are in AWS's `us-west-2`; Coiled currently only offers `us-west-1`. But it's close enough._" |
13 | 15 | ]
|
14 | 16 | },
|
15 | 17 | {
|
16 | 18 | "cell_type": "code",
|
17 | 19 | "execution_count": 1,
|
18 |
| - "id": "occupied-opening", |
| 20 | + "id": "clinical-pastor", |
19 | 21 | "metadata": {},
|
20 | 22 | "outputs": [
|
21 | 23 | {
|
|
73 | 75 | {
|
74 | 76 | "cell_type": "code",
|
75 | 77 | "execution_count": 2,
|
76 |
| - "id": "scheduled-batman", |
| 78 | + "id": "metallic-dodge", |
77 | 79 | "metadata": {},
|
78 | 80 | "outputs": [],
|
79 | 81 | "source": [
|
|
85 | 87 | },
|
86 | 88 | {
|
87 | 89 | "cell_type": "markdown",
|
88 |
| - "id": "bright-massachusetts", |
| 90 | + "id": "first-marshall", |
89 | 91 | "metadata": {},
|
90 | 92 | "source": [
|
91 | 93 | "### Search for a full year of Sentinel-2 data over Santa Fe, New Mexico.\n",
|
|
96 | 98 | {
|
97 | 99 | "cell_type": "code",
|
98 | 100 | "execution_count": 3,
|
99 |
| - "id": "rocky-exercise", |
| 101 | + "id": "opposite-weather", |
100 | 102 | "metadata": {},
|
101 | 103 | "outputs": [
|
102 | 104 | {
|
|
131 | 133 | },
|
132 | 134 | {
|
133 | 135 | "cell_type": "markdown",
|
134 |
| - "id": "detected-belief", |
| 136 | + "id": "noble-minneapolis", |
135 | 137 | "metadata": {},
|
136 | 138 | "source": [
|
137 | 139 | "### Set a coarser resolution to speed up the computation\n",
|
138 | 140 | "\n",
|
139 | 141 | "We'll have `stackstac` retrieve the data at 100m resolution, instead of its native 10m.\n",
|
140 |
| - "Since the data is stored in [Cloud-Optimized GeoTIFFs](https://www.cogeo.org/), with internal overviews, fetching lower-resolution data is very efficient and requires processing an order of magnitude less data.\n", |
| 142 | + "Since the data is stored in [Cloud-Optimized GeoTIFFs](https://www.cogeo.org/) with internal overviews, fetching lower-resolution data is very efficient and requires processing an order of magnitude less data.\n", |
141 | 143 | "\n",
|
142 | 144 | "(Internally, `stackstac` is just telling rasterio/GDAL to build a VRT at this resolution. GDAL then automatically figures out which overview level to fetch data from.)\n",
|
143 | 145 | "\n",
|
|
147 | 149 | {
|
148 | 150 | "cell_type": "code",
|
149 | 151 | "execution_count": 4,
|
150 |
| - "id": "elder-swaziland", |
| 152 | + "id": "scheduled-memphis", |
151 | 153 | "metadata": {},
|
152 | 154 | "outputs": [
|
153 | 155 | {
|
|
172 | 174 | {
|
173 | 175 | "cell_type": "code",
|
174 | 176 | "execution_count": 5,
|
175 |
| - "id": "stuck-organ", |
| 177 | + "id": "medieval-killing", |
176 | 178 | "metadata": {},
|
177 | 179 | "outputs": [
|
178 | 180 | {
|
|
1102 | 1104 | },
|
1103 | 1105 | {
|
1104 | 1106 | "cell_type": "markdown",
|
1105 |
| - "id": "registered-leonard", |
| 1107 | + "id": "fitting-apollo", |
1106 | 1108 | "metadata": {},
|
1107 | 1109 | "source": [
|
1108 | 1110 | "For comparison, this is how much data we'd be processing if we'd used full 10m resolution:"
|
|
1111 | 1113 | {
|
1112 | 1114 | "cell_type": "code",
|
1113 | 1115 | "execution_count": 6,
|
1114 |
| - "id": "duplicate-daniel", |
| 1116 | + "id": "about-stadium", |
1115 | 1117 | "metadata": {},
|
1116 | 1118 | "outputs": [
|
1117 | 1119 | {
|
|
1132 | 1134 | },
|
1133 | 1135 | {
|
1134 | 1136 | "cell_type": "markdown",
|
1135 |
| - "id": "pediatric-kazakhstan", |
| 1137 | + "id": "perfect-consideration", |
1136 | 1138 | "metadata": {},
|
1137 | 1139 | "source": [
|
1138 | 1140 | "### Prepare monthly RGB composites\n",
|
|
1143 | 1145 | {
|
1144 | 1146 | "cell_type": "code",
|
1145 | 1147 | "execution_count": 7,
|
1146 |
| - "id": "legendary-power", |
| 1148 | + "id": "tracked-festival", |
1147 | 1149 | "metadata": {},
|
1148 | 1150 | "outputs": [
|
1149 | 1151 | {
|
|
1640 | 1642 | },
|
1641 | 1643 | {
|
1642 | 1644 | "cell_type": "markdown",
|
1643 |
| - "id": "fatty-death", |
| 1645 | + "id": "dynamic-water", |
1644 | 1646 | "metadata": {},
|
1645 | 1647 | "source": [
|
1646 | 1648 | "### Compute in parallel on the cluster"
|
|
1649 | 1651 | {
|
1650 | 1652 | "cell_type": "code",
|
1651 | 1653 | "execution_count": 8,
|
1652 |
| - "id": "pending-celtic", |
| 1654 | + "id": "accomplished-insertion", |
1653 | 1655 | "metadata": {},
|
1654 | 1656 | "outputs": [
|
1655 | 1657 | {
|
|
1667 | 1669 | },
|
1668 | 1670 | {
|
1669 | 1671 | "cell_type": "markdown",
|
1670 |
| - "id": "blessed-bookmark", |
| 1672 | + "id": "revised-alberta", |
1671 | 1673 | "metadata": {},
|
1672 | 1674 | "source": [
|
1673 | 1675 | "Using 4 Coiled workers (4 CPU, 16 GiB memory each), we processed the ~10 GB of data into median composites in a couple minutes.\n",
|
|
1678 | 1680 | {
|
1679 | 1681 | "cell_type": "code",
|
1680 | 1682 | "execution_count": 9,
|
1681 |
| - "id": "dietary-cambodia", |
| 1683 | + "id": "guided-council", |
1682 | 1684 | "metadata": {},
|
1683 | 1685 | "outputs": [
|
1684 | 1686 | {
|
|
1710 | 1712 | },
|
1711 | 1713 | {
|
1712 | 1714 | "cell_type": "markdown",
|
1713 |
| - "id": "stone-forwarding", |
| 1715 | + "id": "polar-bridal", |
1714 | 1716 | "metadata": {},
|
1715 | 1717 | "source": [
|
1716 | 1718 | "According to the Coiled dashboard, this cost about 74 cents. Was this picture worth 74 cents to you?"
|
|
0 commit comments