Support http basic auth credentials for FROM repo.

[ebmeierj]

I'd like to switch over to this new task (from the docker-image-resource task). However the FROM image in my dockerfile requires you to specify username/password to pull it. I don't see any way to currently do this with the oci-builder-task. This seems like pretty important functionality especially if the community is trying to push people in the oci-build-task direction.

From Alex:
"It might be easy enough to add support for it; I think buildkit will look under $DOCKER_CONFIG/config.json for credentials, the task just doesn't support creating/configuring it yet."

Does anyone else know of a workaround?

[andymoe]

I think I'm running into this as well. Does this mean there is no way to build with a Dockerfile that relies on base images in a private repo?

[andy-paine]

I've tried dropping a config.json file at $DOCKER_CONFIG/config.json and at ~/.docker/config.json and running build from an intercepted container but I can't seem to get buildkit to read it. I even tried mangling the JSON to see if buildkit would throw some error but buildkit didn't say anything which makes me think it isn't even reading it 😞
See #14 (comment)

[RCM7]

I'm having the same problem. The FROM image is hosted in a private repo.
Was digging for a while in the builtkit project to try a workaround but with no luck.

[jgriff]

Having same issue.... But I'm wondering if the "Concourse Way" ™️ of doing this would be to fetch the base (FROM) image as a resource, and then the oci-build-task just loads it as a base image for the build.

This seems to be at least the motivation for #1

Something like...

resources:
  - name: some-base-image
    type: registry-image
    icon: docker
    source:
      repository: requires-auth/some-base-image
      tag: latest
      username: ((registry-username))
      password: ((registry-password))

  - name: my-image-src
    type: git
    icon: github
    source:
      uri: https://github.com/...

jobs:
  - name: build
    plan:
      - get: some-base-image
        trigger: true
        params: {format: oci}
      - get: my-image-src    
      - task: build
        privileged: true
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: vito/oci-build-task
          inputs:
          - name: some-base-image
          - name: my-image-src
          outputs:
          - name: image
          params:
            load_bases: some-base-image/image.tar
          run:
            dir: my-image-src
            path: build

This would offload all credential management and fetching responsibility to the resource, which is more closely tied to that responsibility. It also would let you track/trigger/pull that base image just once (atomically).

However, it too seems to be blocked (waiting on buildkit to support it).

Wonder if @vito has ideas on what we could do in the interim....

[andy-paine]

Turns out I was just being a bit dim and had formatted my JSON wrong. I wrote this task to run before my oci-build-task:

task: generate-docker-creds
config:
  platform: linux
  image_resource:
    type: registry-image
    source:
        repository: alpine
  outputs:
  - name: docker_creds
  run:
    path: ash
    args:
    - -c
    - |
      AUTH="$(echo -n '((docker_username)):((docker_password))' | base64)"
      cat > docker_creds/config.json <<EOF
      { "auths": { "https://index.docker.io/v1/": { "auth": "$AUTH", "email": "<yourdockeremail>" }}}
      EOF

Then just added docker_creds as an input to my build task and DOCKER_CONFIG: docker_creds to the params and it worked 🎉

[vito]

@jgriff Yep, you got it. That's my hope for the long term. Pretty bummed that it's still blocked, I would have thought other CI systems would want this sort of thing for controlling their build's inputs. 😕 I'll throw my hat into the ring on the upstream issue.

@andy-paine Nice, good to know that works! Until things get unblocked upstream I'd be happy to review a PR for that. Sounds a little tricky to configure though; you'd need to provide a registry URL, a username, and a password - possibly multiple if there are multiple FROM stanzas in the Dockerfile. 🤔

[geemanjs]

#14 (comment)

In case someone else stumbles upon:

WARNING: Error loading config file: docker_creds/config.json: invalid character '\n' in string literal

Adding -w 0 to the base64 resolved it for me ..

      AUTH="$(echo -n '((docker_username)):((docker_password))' | base64 -w 0)"
      cat > docker_creds/config.json <<EOF
      { "auths": { "https://index.docker.io/v1/": { "auth": "$AUTH", "email": "<yourdockeremail>" }}}
      EOF

[Petahhh]

For anyone who followed @andy-paine and @Geeman201 's code samples and run into this

base64: unrecognized option: w
BusyBox v1.31.1 () multi-call binary.

Usage: base64 [-d] [FILE]

Base64 encode or decode FILE to standard output
	-d	Decode data

the alpine image's base64 does not come with -w. I got unblocked by swapping it out with ubuntu

[jprelph]

We've had similar issues with ECR so I've created a fork with ECR support - https://github.com/jprelph/oci-build-task - image is at jprelph/oci-build-task in docker hub. I just stuck support for passing username/password in as well via env var, seems to work OK - nothing too elegant about it, it just creates a /root/.docker/config.json file for buildkit to pick up. Usage details in the README, but it basically looks like:

      params:
        LOGIN: PASS
        PASS: ((username)):((password))
        REG_URL: "https://quay.io"

for username/password use, or:

      params:
        LOGIN: ECR

to pickup from IAM role.

Happy to raise as a PR if felt worth doing, but I've thrown that together pretty quickly and at the moment only the ECR version supports multiple registries - I'm not sure how to handle that nicely with username/password.

[ndcotta-bc]

@vito we are trying to use private images (from google's container registry) as base images for builds. Could you please point us to where we can track the upstream issue?

[vito]

@ndcotta-bc here it is: moby/buildkit#706

I offered to help PR it but it doesn't look like the maintainers have had time to provide much guidance. 😕

[Krenair]

We just ran into this with DockerHub rate limiting, e.g.:

task: build
privileged: true
config:
  image_resource:
    type: registry-image
    source:
      repository: vito/oci-build-task
      username: ((dockerhubpull-concourse.username))
      password: ((dockerhubpull-concourse.password))
  inputs:
    - name: src
  outputs:
    - name: image
  params:
    CONTEXT: src/components/canary
  platform: linux
  run:
    path: build

can result in:

selected worker: gsp-concourse-worker-1
[snip]
#4 [internal] load metadata for docker.io/library/golang:latest
#4 ERROR: failed to copy: httpReaderSeeker: failed open: unexpected status code https://registry-1.docker.io/v2/library/golang/manifests/sha256:cf46c759511d0376c706a923f2800762948d4ea1a9290360720d5124a730ed63: 429 Too Many Requests - Server message: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
[snip]

At this point I'd try adding credentials to use to pull FROM images, but this resource doesn't make it easy. I'll try @andy-paine's hack, thanks for that. Also Andy I think your indentation is a bit broken :)

[andy-paine]

Thanks for the hint @Krenair - I think I've fixed it now. (Shameless plug) I've added a task and some tests to EngineerBetter's repo of reusable tasks for anyone who wants to just grab this task nice and succinctly https://github.com/EngineerBetter/concourse-tasks/tree/main/generate-docker-config-file

[Krenair]

Thanks for the hint @Krenair - I think I've fixed it now.

Thanks, looks better. I also put together a merged version that combines oci-build-task build usage with the authentication step in https://github.com/alphagov/gsp/pull/1210/files

[e-orz]

For anyone who followed @andy-paine and @Geeman201 's code samples and run into this

base64: unrecognized option: w
BusyBox v1.31.1 () multi-call binary.

Usage: base64 [-d] [FILE]

Base64 encode or decode FILE to standard output
	-d	Decode data

the alpine image's base64 does not come with -w. I got unblocked by swapping it out with ubuntu

My solution: simply change the line to:

AUTH="$(echo -n "$USERNAME:$PASSWORD" | base64 | tr -d "\n")"

No need for the full blown ubuntu for this.
BTW, in case of a private registry, don't forget to change https://index.docker.io/v1/ to your actual URL.

[vito]

I'm planning to close this with #52 which (finally!) adds support for loading pre-fetched image tarballs for use in FROM ....

The recommendation will be to fetch the image using the registry-image resource and then pass it along using IMAGE_ARG_*. That way the oci-build task doesn't have to duplicate effort by implementing TLS, auth, mirrors, etc. etc. and can fully control the image versions, allowing the build to be repeatable. This is the same approach mentioned by @jgriff in #14 (comment)

[CiucurDaniel]

Hello everybody!

I am facing the exact same issue described here. Regarding @vito comment with the recommended solution.

The recommendation will be to fetch the image using the registry-image resource and then pass it along using IMAGE_ARG_*. That way the oci-build task doesn't have to duplicate effort by implementing TLS, auth, mirrors, etc. etc. and can fully control the image versions, allowing the build to be repeatable. This is the same approach mentioned by @jgriff in #14 (comment)

Is there anybody who can provide an example configuration? Because I'm having some problems setting that up as I do not fully understand how IMAGE_ARG_* works. I did read the docs and the messages here but I'm still confused on how to do it exactly.

[suhlig]

@CiucurDaniel I found a working solution in #52 (comment)