Skip to content
This repository was archived by the owner on Apr 23, 2025. It is now read-only.

Document GitLab with private container registries #414

Merged
merged 6 commits into from
May 14, 2023
Merged

Document GitLab with private container registries #414

Changes from 5 commits
Commits
Show all changes
6 commits
Select commit Hold shift + click to select a range
91c5dbb
Document GitLab with private container registries
0x2b3bfa0 Dec 6, 2022
c8cd838
Apply suggestions from code review
0x2b3bfa0 Dec 6, 2022
eae46dc
Use tabs
0x2b3bfa0 Dec 6, 2022
4b8a672
Merge branch 'gitlab-ecr' of https://github.com/iterative/cml.dev int…
0x2b3bfa0 Dec 6, 2022
8d55704
Merge branch 'master' into gitlab-ecr
0x2b3bfa0 Dec 12, 2022
e8e7b7c
Merge branch 'master' into gitlab-ecr
0x2b3bfa0 Feb 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
96 changes: 95 additions & 1 deletion content/docs/self-hosted-runners.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -448,7 +448,101 @@ The same credentials can also be used for

</admon>

### On-premise (Local) Runners
## GitLab CI/CD and container images from private registries
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is GL specific? what about GH?

Copy link
Member Author

@0x2b3bfa0 0x2b3bfa0 Dec 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is GitLab-specific. On GitHub it's not possible.1

Footnotes

  1. Read as «not easy enough to be documented» ↩

Copy link
Member Author

@0x2b3bfa0 0x2b3bfa0 Dec 19, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The closest approximation I can think of is this (not working) example:

on: push
jobs:
  authentication:
    runs-on: ubuntu-latest
    outputs:
      username: ${{ steps.authenticate.outputs.username }}
      password: ${{ steps.authenticate.outputs.password }}
    steps:
      - id: authenticate
        run: |
          echo "username=AWS" >> $GITHUB_OUTPUT
          echo "password=$(aws ecr get-login-password --region us-east-1)" >> $GITHUB_OUTPUT
        env:
          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  example:
    needs: authentication
    runs-on: ubuntu-latest
    container:
      image: ACCOUNT.dkr.ecr.REGION.amazonaws.com/REPOSITORY:TAG
      credentials:
        username: ${{ needs.authentication.outputs.username }}
        password: ${{ needs.authentication.outputs.password }}
    steps:
      - run: true

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

...or https://agileek.github.io/software/aws/2020/10/28/using-an-ecr-image-in-github-actions

Copy link
Contributor

@casperdcl casperdcl Dec 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

kewl. I think this perhaps deserves a new page?
or perhaps https://cml.dev/doc/ref/runner#examples?

Copy link
Member Author

@0x2b3bfa0 0x2b3bfa0 Dec 20, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is about self-hosted runners, on GitLab, with private container registries. Definition of niche.


_See also the
[GitLab documentation](https://docs.gitlab.com/ee/ci/docker/using_docker_images.html#use-credential-helpers)
for more information._

<toggle>
<tab title="AWS">

### GitLab CI/CD Environment Variables

| Name | Value |
| ----------------------- | ----------------------------------------------------------------------------------------------------------- |
| `DOCKER_AUTH_CONFIG` | `{"credHelpers": {"ACCOUNT.dkr.ecr.REGION.amazonaws.com": "ecr-login"}}` |
| `AWS_ACCESS_KEY_ID` | [AWS access key identifier](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) |
| `AWS_SECRET_ACCESS_KEY` | [AWS secret access key](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) |
| `AWS_SESSION_TOKEN` | [AWS session token **_(optional)_**](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html) |
| `REPO_TOKEN` | [GitLab Personal Access Token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) |

### `.gitlab-ci.yml`

```yaml
runner:
when: always
image: iterativeai/cml
script:
- cml runner --labels=cml --cloud=aws

job:
tags: [cml]
needs: [runner]
image: ACCOUNT.dkr.ecr.REGION.amazonaws.com/REPOSITORY:TAG
script:
- echo succeeded
```

<admon type="tip">

Replace the `ACCOUNT`, `REGION`, `REPOSITORY` and `TAG` placeholders with
appropriate values.

</admon>

</tab>

<tab title="GCP">

### GitLab CI/CD Environment Variables

| Name | Value |
| ------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------- |
| `DOCKER_AUTH_CONFIG` | `{"credHelpers": {"LOCATION-docker.pkg.dev": "gcr"}}` |
| `GOOGLE_APPLICATION_CREDENTIALS_DATA` | [Contents of a Google Cloud service account JSON key file](https://cloud.google.com/iam/docs/creating-managing-service-account-keys#creating) |
| `REPO_TOKEN` | [GitLab Personal Access Token](https://docs.gitlab.com/ee/user/profile/personal_access_tokens.html) |

### `.gitlab-ci.yml`

```yaml
runner:
when: always
image: iterativeai/cml
script:
- cml runner --labels=cml --cloud=gcp --cloud-permission-set="$(printenv
GOOGLE_APPLICATION_CREDENTIALS_DATA | jq
.client_email),scopes=storage-ro,datastore"

job:
tags: [cml]
needs: [runner]
image: LOCATION-docker.pkg.dev/PROJECT/REPOSITORY/IMAGE:TAG
script:
- echo succeeded
```

<admon type="tip">

Replace the
[`LOCATION`](https://cloud.google.com/artifact-registry/docs/repositories/repo-locations),
`PROJECT`, `REPOSITORY`, `IMAGE` and `TAG` placeholders with appropriate values.

</admon>

<admon type="warn">

This example uses the new
[Artifact Registry](https://cloud.google.com/artifact-registry) (i.e. any of the
`LOCATION-docker.pkg.dev` domains) instead of the old Container Registry (i.e.
the `gcr.io` domain) but instructions are similar for both.

</admon>

</tab>
</toggle>

## On-premise (Local) Runners

The `cml runner` command can also be used to manually set up a local machine,
on-premise GPU cluster, or any other cloud compute resource as a self-hosted
Expand Down