docs: iteration 3 #492
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -13,6 +13,8 @@ TF_LOG_PROVIDER=INFO terraform apply | |
|
|
||
| ## Amazon Web Services | ||
|
|
||
| [Create an AWS account](https://aws.amazon.com/premiumsupport/knowledge-center/create-and-activate-aws-account/) if needed, and then set these environment variables: | ||
|
|
||
| - `AWS_ACCESS_KEY_ID` - Access key identifier. | ||
| - `AWS_SECRET_ACCESS_KEY` - Secret access key. | ||
| - `AWS_SESSION_TOKEN` - (Optional) Session token. | ||
|
|
@@ -29,6 +31,8 @@ export AWS_SECRET_ACCESS_KEY="$(terraform output --raw aws_secret_access_key)" | |
|
|
||
| ## Microsoft Azure | ||
|
|
||
| [Create an Azure account](https://docs.microsoft.com/en-us/learn/modules/create-an-azure-account/) if needed, and then set these environment variables: | ||
|
|
||
| - `AZURE_CLIENT_ID` - Client identifier. | ||
| - `AZURE_CLIENT_SECRET` - Client secret. | ||
| - `AZURE_SUBSCRIPTION_ID` - Subscription identifier. | ||
|
|
@@ -48,7 +52,10 @@ export AZURE_CLIENT_SECRET="$(terraform output --raw azure_client_secret)" | |
|
|
||
| ## Google Cloud Platform | ||
|
|
||
| [Create a GCP account](https://cloud.google.com/free) if needed, and then either one of the environment variables: | ||
|
|
||
| - `GOOGLE_APPLICATION_CREDENTIALS` - **Path** to a service account JSON key file. | ||
| - `GOOGLE_APPLICATION_CREDENTIALS_DATA` - Alternatively, **contents** of a service account JSON key file. | ||
|
|
||
| See the [GCP documentation](https://cloud.google.com/docs/authentication/getting-started#creating_a_service_account) to obtain these variables directly. | ||
|
|
||
|
|
||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -20,11 +20,11 @@ page_title: Getting Started | |
| sudo apt-get update && sudo apt-get install terraform | ||
| ``` | ||
|
|
||
| - Create an account with any supported cloud vendor and expose its [authentication credentials via environment variables][auth] | ||
|
|
||
| [auth]: https://registry.terraform.io/providers/iterative/iterative/latest/docs/guides/authentication | ||
|
Comment on lines
-23
to
+25
There was a problem hiding this comment. BTW I think the nav would be more logical with Getting Started on top, since it covers installation and then it links to the auth section. |
||
|
|
||
| ## Define a Task | ||
|
|
||
| In a project root directory, create a file named `main.tf` with the following contents: | ||
|
|
||
|
|
@@ -33,6 +33,7 @@ terraform { | |
| required_providers { iterative = { source = "iterative/iterative" } } | ||
| } | ||
| provider "iterative" {} | ||
|
|
||
| resource "iterative_task" "example" { | ||
| cloud = "aws" # or any of: gcp, az, k8s | ||
| machine = "m" # medium. Or any of: l, xl, m+k80, xl+v100, ... | ||
|
|
@@ -45,8 +46,18 @@ resource "iterative_task" "example" { | |
| } | ||
| script = <<-END | ||
| #!/bin/bash | ||
|
|
||
| # create output directory if needed | ||
| mkdir -p results | ||
| # read last result (in case of spot/preemptible instance recovery) | ||
| if test -f results/epoch.txt; then EPOCH="$(cat results/epoch.txt)"; fi | ||
| EPOCH=$${EPOCH:-1} # start from 1 if last result not found | ||
|
|
||
| echo "(re)starting training loop from $EPOCH up to 1337 epochs" | ||
| for epoch in $(seq $EPOCH 1337); do | ||
| sleep 1 | ||
| echo "$epoch" | tee results/epoch.txt | ||
| done | ||
| END | ||
| } | ||
| ``` | ||
|
|
@@ -61,7 +72,7 @@ The project layout should look similar to this: | |
| project/ | ||
| ├── main.tf | ||
| └── results/ | ||
| └── epoch.txt (created in the cloud and downloaded locally) | ||
| ``` | ||
|
|
||
| ## Initialise Terraform | ||
|
|
@@ -72,7 +83,7 @@ $ terraform init | |
|
|
||
| This command will check `main.tf` and download the required TPI plugin. | ||
|
|
||
| ~> **Warning:** None of the subsequent commands will work without first setting some [authentication environment variables][auth]. | ||
|
|
||
| ## Run Task | ||
|
|
||
|
|
@@ -82,14 +93,16 @@ $ TF_LOG_PROVIDER=INFO terraform apply | |
|
|
||
| This command will: | ||
|
|
||
| 1. Create all the required cloud resources (provisioning a `machine` with `disk_size` storage). | ||
| 2. Upload the working directory (`workdir`) to the cloud. | ||
| 3. Launch the task `script`. | ||
|
|
||
| With spot/preemptible instances (`spot >= 0`), auto-recovery logic and persistent (`disk_size`) storage will be used to relaunch interrupted tasks. | ||
|
|
||
| -> **Note:** A large `workdir` may take a long time to upload. | ||
|
|
||
| ~> **Warning:** To take full advantage of spot instance recovery, a `script` should start by cheching the disk for results (recovered from a previous interrupted run). | ||
|
|
||
| -> **Note:** The [`id`](https://registry.terraform.io/providers/iterative/iterative/latest/docs/resources/task#id) returned by `terraform apply` (i.e. `[id=tpi-···]`) can be used to locate the created cloud resources through the cloud's web console or command–line tool. | ||
|
|
||
| ## Query Status | ||
|
|
@@ -113,9 +126,9 @@ $ TF_LOG_PROVIDER=INFO terraform destroy | |
| This command will: | ||
|
|
||
| 1. Download the `output` directory from the cloud. | ||
| 2. Delete all the cloud resources created by `terraform apply` (terminating `machine` if it's still running and removing the persistent `disk_size` storage). | ||
|
|
||
| In this example, after running `terraform destroy`, the `results` directory should contain a file named `epoch.txt` with the text `1337`. | ||
|
|
||
| -> **Note:** A large `output` directory may take a long time to download. | ||
|
|
||
|
|
||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
made a lot of changes since this review, a couple outstanding things:
technically we are stopping tasks <- terminating instances <- destroying resources
implied by transparent spot auto-recovery, right?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes and no, you make the users to think or guess at least. There is a phase in design (title also of a praised book) "Dont make me think"
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No, technically Tasks are stopped by themselves, what I want to say with this is that the user might find the ambiguity of the stop concept in cloud. Where they can stop the machine and then at some point they can restart the machine again. Here after destroy there is no possible resume