Can't pull images from k8s.io in eu-west region over IPv6

[deric]

Is there an existing issue for this?

• I have searched the existing issues

What did you expect to happen?

containerd is failing to fetch any image from registry.k8s.io:

$ ctr image pull registry.k8s.io/pause:3.9
ctr: failed to resolve reference "registry.k8s.io/pause:3.9": pulling from host registry.k8s.io failed with status code [manifests 3.9]: 403 Forbidden

however curl work without any problems:

$ curl -I https://registry.k8s.io/v2
HTTP/2 200 
docker-distribution-api-version: registry/2.0
date: Thu, 05 Oct 2023 10:56:43 GMT
content-type: text/html
server: Google Frontend
via: 1.1 google, 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

I expect ctr image pull to pull the image (same as on any other machine in the same datacenter).

ctr image pull registry.k8s.io/pause:3.9
registry.k8s.io/pause:3.9:                                                        resolved       |++++++++++++++++++++++++++++++++++++++| 
index-sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097:    exists         |++++++++++++++++++++++++++++++++++++++| 
manifest-sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10: exists         |++++++++++++++++++++++++++++++++++++++| 
layer-sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007:    exists         |++++++++++++++++++++++++++++++++++++++| 
config-sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c:   exists         |++++++++++++++++++++++++++++++++++++++| 
elapsed: 0.4 s                                                                    total:   0.0 B (0.0 B/s)                                         
unpacking linux/amd64 sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097...
done: 9.945769ms

Debugging Information

From the same machine:

$ ctr --debug image pull --http-dump registry.k8s.io/pause:3.9
DEBU[0000] fetching                                      image="registry.k8s.io/pause:3.9"
DEBU[0000] resolving                                     host=registry.k8s.io
DEBU[0000] do request                                    host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.6.23 request.method=HEAD url="https://registry.k8s.io/v2/pause/manifests/3.9"
INFO[0000] HEAD /v2/pause/manifests/3.9 HTTP/1.1        
INFO[0000] Host: registry.k8s.io                        
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */* 
INFO[0000] User-Agent: containerd/v1.6.23               
INFO[0000]                                              
INFO[0000] HTTP/1.1 403 Forbidden                       
INFO[0000] Transfer-Encoding: chunked                   
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 
INFO[0000]                                              
INFO[0000]                                              
DEBU[0000] fetch response received                       host=registry.k8s.io response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.status="403 Forbidden" url="https://registry.k8s.io/v2/pause/manifests/3.9"
ctr: failed to resolve reference "registry.k8s.io/pause:3.9": pulling from host registry.k8s.io failed with status code [manifests 3.9]: 403 Forbidden

I would assume my IP is from some weird reason blocked, but curl works fine:

$ curl -LI https://registry.k8s.io/v2/pause/tags/list
HTTP/2 307 
content-type: text/html; charset=utf-8
location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/tags/list
x-cloud-trace-context: f873a252d498ee8c6d5593a94b50b786
date: Thu, 05 Oct 2023 10:36:24 GMT
server: Google Frontend
via: 1.1 google, 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

HTTP/2 405 
content-length: 1458
content-type: text/html; charset=utf-8
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-xss-protection: 0
date: Thu, 05 Oct 2023 10:36:24 GMT
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

The same issue with accessing URI https://dl.k8s.io/release/stable-1.txt:

$ kubeadm config images pull -v=3
I1005 10:37:47.958074 1142075 initconfiguration.go:116] detected and using CRI socket: unix:///var/run/containerd/containerd.sock
I1005 10:37:47.958437 1142075 kubelet.go:196] the value of KubeletConfiguration.cgroupDriver is empty; setting it to "systemd"
I1005 10:37:47.961639 1142075 version.go:187] fetching Kubernetes version from URL: https://dl.k8s.io/release/stable-1.txt
W1005 10:37:48.102118 1142075 version.go:104] could not fetch a Kubernetes version from the internet: unable to fetch file. URL: "https://dl.k8s.io/release/stable-1.txt", status: 403 Forbidden
W1005 10:37:48.102146 1142075 version.go:105] falling back to the local client version: v1.25.14

again curl is NOT blocked.

$ curl -LI https://dl.k8s.io/release/stable-1.txt
HTTP/2 302 
server: nginx
date: Thu, 05 Oct 2023 10:38:51 GMT
content-type: text/html
content-length: 138
location: https://cdn.dl.k8s.io/release/stable-1.txt
via: 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

HTTP/2 200 
expires: Sun, 24 Sep 2023 09:03:16 GMT
last-modified: Wed, 13 Sep 2023 17:49:44 GMT
etag: "916e3ef6ca132fdf822670211275122e"
content-type: text/plain
cache-control: private, no-store
accept-ranges: bytes
date: Thu, 05 Oct 2023 10:38:52 GMT
via: 1.1 varnish
age: 956135
x-served-by: cache-fra-etou8220067-FRA
x-cache: HIT
x-cache-hits: 4
access-control-allow-origin: *
content-length: 7

Using crane:

crane pull --verbose registry.k8s.io/pause:3.9 /dev/null
2023/10/05 10:52:12 --> GET https://registry.k8s.io/v2/
2023/10/05 10:52:12 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/0.16.1 go-containerregistry/0.16.1
Accept-Encoding: gzip


2023/10/05 10:52:12 <-- 403 https://registry.k8s.io/v2/ (147.948006ms)
2023/10/05 10:52:12 HTTP/2.0 403 Forbidden
Content-Length: 298
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=UTF-8
Referrer-Policy: no-referrer


<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>403 Forbidden</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/v2/</code> from this server.</h2>
<h2></h2>
</body></html>

Error: GET https://registry.k8s.io/v2/: unexpected status code 403 Forbidden: 
<html><head>
<meta http-equiv="content-type" content="text/html;charset=utf-8">
<title>403 Forbidden</title>
</head>
<body text=#000000 bgcolor=#ffffff>
<h1>Error: Forbidden</h1>
<h2>Your client does not have permission to get URL <code>/v2/</code> from this server.</h2>
<h2></h2>
</body></html>

curl pretending to be containerd:

$ curl -IL --user-agent "containerd/v1.6.23" -H "Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" https://registry.k8s.io/v2/pause/manifests/3.9
HTTP/2 307 
content-type: text/html; charset=utf-8
location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
x-cloud-trace-context: 626096157d0bb50b88eb394d0730c051
date: Thu, 05 Oct 2023 11:10:20 GMT
server: Google Frontend
via: 1.1 google, 1.1 google
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

HTTP/2 200 
content-length: 2405
content-type: application/vnd.docker.distribution.manifest.list.v2+json
docker-content-digest: sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097
docker-distribution-api-version: registry/2.0
date: Thu, 05 Oct 2023 11:10:20 GMT
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

with --http-trace:

$ ctr --debug image pull --http-dump --http-trace -k registry.k8s.io/pause:3.9DEBU[0000] fetching                                      image="registry.k8s.io/pause:3.9"
DEBU[0000] resolving                                     host=registry.k8s.io
DEBU[0000] do request                                    host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.6.23 request.method=HEAD url="https://registry.k8s.io/v2/pause/manifests/3.9"
DEBU[0000] DNS lookup                                    host=registry.k8s.io
INFO[0000] HEAD /v2/pause/manifests/3.9 HTTP/1.1        
INFO[0000] Host: registry.k8s.io                        
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */* 
INFO[0000] User-Agent: containerd/v1.6.23               
INFO[0000]                                              
DEBU[0000] DNS lookup complete                           coalesced=false result="2600:1901:0:bbc4::"
DEBU[0000] Connection successful                         remote_addr="[2600:1901:0:bbc4::]:443" reused=false
INFO[0000] HTTP/1.1 403 Forbidden                       
INFO[0000] Transfer-Encoding: chunked                   
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 
INFO[0000]                                              
INFO[0000]                                              
DEBU[0000] fetch response received                       host=registry.k8s.io response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.status="403 Forbidden" url="https://registry.k8s.io/v2/pause/manifests/3.9"
ctr: failed to resolve reference "registry.k8s.io/pause:3.9": pulling from host registry.k8s.io failed with status code [manifests 3.9]: 403 Forbidden

fetching manifest with curl:

$ curl -vI https://registry.k8s.io/v2/pause/manifests/3.9
*   Trying 34.96.108.209:443...
* Connected to registry.k8s.io (34.96.108.209) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*  CAfile: /etc/ssl/certs/ca-certificates.crt
*  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=registry.k8s.io
*  start date: Sep  6 01:59:50 2023 GMT
*  expire date: Dec  5 02:55:46 2023 GMT
*  subjectAltName: host "registry.k8s.io" matched cert's "registry.k8s.io"
*  issuer: C=US; O=Google Trust Services LLC; CN=GTS CA 1D4
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x563ef8b73990)
> HEAD /v2/pause/manifests/3.9 HTTP/2
> Host: registry.k8s.io
> user-agent: curl/7.74.0
> accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 100)!
< HTTP/2 307 
HTTP/2 307 
< content-type: text/html; charset=utf-8
content-type: text/html; charset=utf-8
< location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
< x-cloud-trace-context: 0d6c4da7a79ff93a7e8b4091997e00da
x-cloud-trace-context: 0d6c4da7a79ff93a7e8b4091997e00da
< date: Thu, 05 Oct 2023 11:23:00 GMT
date: Thu, 05 Oct 2023 11:23:00 GMT
< server: Google Frontend
server: Google Frontend
< via: 1.1 google, 1.1 google
via: 1.1 google, 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

< 
* Connection #0 to host registry.k8s.io left intact

Could this be an IPv6 issue on GCP? Or my IPv6 address is being blocked but not IPv4?

$ curl -I -6 https://registry.k8s.io/v2/pause/manifests/3.9
HTTP/2 403 
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

Anything else?

Expected output, from different IP, the same provider, the same DC:

$ ctr --debug image pull --http-dump registry.k8s.io/pause:3.9
DEBU[0000] fetching                                      image="registry.k8s.io/pause:3.9"
DEBU[0000] resolving                                     host=registry.k8s.io
DEBU[0000] do request                                    host=registry.k8s.io request.header.accept="application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */*" request.header.user-agent=containerd/v1.6.23 request.method=HEAD url="https://registry.k8s.io/v2/pause/manifests/3.9"
INFO[0000] HEAD /v2/pause/manifests/3.9 HTTP/1.1        
INFO[0000] Host: registry.k8s.io                        
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */* 
INFO[0000] User-Agent: containerd/v1.6.23               
INFO[0000]                                              
INFO[0000] HTTP/1.1 307 Temporary Redirect              
INFO[0000] Transfer-Encoding: chunked                   
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 
INFO[0000] Content-Type: text/html; charset=utf-8       
INFO[0000] Date: Thu, 05 Oct 2023 10:40:35 GMT          
INFO[0000] Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9 
INFO[0000] Server: Google Frontend                      
INFO[0000] Via: 1.1 google, 1.1 google                  
INFO[0000] X-Cloud-Trace-Context: b6c85b1c5e226abd140e1a99eef733f8 
INFO[0000]                                              
INFO[0000]                                              
INFO[0000] HEAD /v2/k8s-artifacts-prod/images/pause/manifests/3.9 HTTP/0.0 
INFO[0000] Host: europe-west4-docker.pkg.dev            
INFO[0000] Accept: application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, */* 
INFO[0000] Referer: https://registry.k8s.io/v2/pause/manifests/3.9 
INFO[0000] User-Agent: containerd/v1.6.23               
INFO[0000]                                              
INFO[0000] HTTP/1.1 200 OK                              
INFO[0000] Content-Length: 2405                         
INFO[0000] Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000 
INFO[0000] Content-Type: application/vnd.docker.distribution.manifest.list.v2+json 
INFO[0000] Date: Thu, 05 Oct 2023 10:40:35 GMT          
INFO[0000] Docker-Content-Digest: sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097 
INFO[0000] Docker-Distribution-Api-Version: registry/2.0 
DEBU[0000] fetch response received                       host=registry.k8s.io response.header.alt-svc="h3=\":443\"; ma=2592000,h3-29=\":443\"; ma=2592000" response.header.content-length=2405 response.header.content-type=application/vnd.docker.distribution.manifest.list.v2+json response.header.date="Thu, 05 Oct 2023 10:40:35 GMT" response.header.docker-content-digest="sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097" response.header.docker-distribution-api-version=registry/2.0 response.status="200 OK" url="https://registry.k8s.io/v2/pause/manifests/3.9"
DEBU[0000] resolved                                      desc.digest="sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097" host=registry.k8s.io
INFO[0000]                                              
DEBU[0000] fetch                                         digest="sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097" mediatype=application/vnd.docker.distribution.manifest.list.v2+json size=2405
DEBU[0000] fetch                                         digest="sha256:8d4106c88ec0bd28001e34c975d65175d994072d65341f62a8ab0754b0fafe10" mediatype=application/vnd.docker.distribution.manifest.v2+json size=526
DEBU[0000] fetch                                         digest="sha256:e6f1816883972d4be47bd48879a08919b96afcd344132622e4d444987919323c" mediatype=application/vnd.docker.container.image.v1+json size=973
DEBU[0000] fetch                                         digest="sha256:61fec91190a0bab34406027bbec43d562218df6e80d22d4735029756f23c7007" mediatype=application/vnd.docker.image.rootfs.diff.tar.gzip size=317616
DEBU[0000] unpacking                                     image="registry.k8s.io/pause:3.9"
unpacking linux/amd64 sha256:7031c1b283388d2c2e09b57badb803c05ebed362dc88d84b480cc47f72a21097...
done: 28.410402ms

Possibly related, but different issues:

• Getting 403 Forbidden when trying to pull an image #211 curl doesn't work, in my case curl in not blocked
• Random download failures - 403 errors [hetzner] #138 errors with curl over IPv4 (works for me)

Code of Conduct

• I agree to follow this project's Code of Conduct

[BenTheElder]

Unfortunately I can't comment about the specifics of cloud blocking :(

I can say that nothing about this appears to be coming from any settings that the kubernetes project operates, see the past issues [edit: that you linked] and the note at the top of the debugging doc :/

[zaro]

Same here, our VMs are in Germany (Hetzner) and this is what I get when trying to pull:

$ crane pull --verbose registry.k8s.io/pause:3.9 pause.tgz
2023/10/06 05:57:31 --> GET https://registry.k8s.io/v2/
2023/10/06 05:57:31 GET /v2/ HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/0.16.1 go-containerregistry/0.16.1
Accept-Encoding: gzip


2023/10/06 05:57:31 <-- 200 https://registry.k8s.io/v2/ (88.032603ms)
2023/10/06 05:57:31 HTTP/2.0 200 OK
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html
Date: Fri, 06 Oct 2023 05:57:31 GMT
Docker-Distribution-Api-Version: registry/2.0
Server: Google Frontend
Via: 1.1 google, 1.1 google
X-Cloud-Trace-Context: d9883ca923f3cfee550a81b7ae11bce6
Content-Length: 0


2023/10/06 05:57:31 --> GET https://registry.k8s.io/v2/pause/manifests/3.9
2023/10/06 05:57:31 GET /v2/pause/manifests/3.9 HTTP/1.1
Host: registry.k8s.io
User-Agent: crane/0.16.1 go-containerregistry/0.16.1
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Accept-Encoding: gzip


2023/10/06 05:57:31 <-- 307 https://registry.k8s.io/v2/pause/manifests/3.9 (23.311826ms)
2023/10/06 05:57:31 HTTP/2.0 307 Temporary Redirect
Content-Length: 120
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=utf-8
Date: Fri, 06 Oct 2023 05:57:31 GMT
Location: https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
Server: Google Frontend
Via: 1.1 google, 1.1 google
X-Cloud-Trace-Context: da7d60662abff0c54c5a87bfd58d3b6e

<a href="https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9">Temporary Redirect</a>.


2023/10/06 05:57:31 --> GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9
2023/10/06 05:57:31 GET /v2/k8s-artifacts-prod/images/pause/manifests/3.9 HTTP/1.1
Host: europe-west4-docker.pkg.dev
User-Agent: crane/0.16.1 go-containerregistry/0.16.1
Accept: application/vnd.docker.distribution.manifest.v1+json,application/vnd.docker.distribution.manifest.v1+prettyjws,application/vnd.docker.distribution.manifest.v2+json,application/vnd.oci.image.manifest.v1+json,application/vnd.docker.distribution.manifest.list.v2+json,application/vnd.oci.image.index.v1+json
Referer: https://registry.k8s.io/v2/pause/manifests/3.9
Accept-Encoding: gzip


2023/10/06 05:57:31 <-- 403 https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9 (183.983717ms)
2023/10/06 05:57:31 HTTP/2.0 403 Forbidden
Content-Length: 1627
Alt-Svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
Content-Type: text/html; charset=UTF-8
Date: Fri, 06 Oct 2023 05:57:31 GMT
Referrer-Policy: no-referrer

<!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 403 (Forbidden)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>403.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to get URL <code>/v2/k8s-artifacts-prod/images/pause/manifests/3.9</code> from this server.  <ins>That’s all we know.</ins>

Error: GET https://europe-west4-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9: unexpected status code 403 Forbidden: <!DOCTYPE html>
<html lang=en>
  <meta charset=utf-8>
  <meta name=viewport content="initial-scale=1, minimum-scale=1, width=device-width">
  <title>Error 403 (Forbidden)!!1</title>
  <style>
    *{margin:0;padding:0}html,code{font:15px/22px arial,sans-serif}html{background:#fff;color:#222;padding:15px}body{margin:7% auto 0;max-width:390px;min-height:180px;padding:30px 0 15px}* > body{background:url(//www.google.com/images/errors/robot.png) 100% 5px no-repeat;padding-right:205px}p{margin:11px 0 22px;overflow:hidden}ins{color:#777;text-decoration:none}a img{border:0}@media screen and (max-width:772px){body{background:none;margin-top:0;max-width:none;padding-right:0}}#logo{background:url(//www.google.com/images/branding/googlelogo/1x/googlelogo_color_150x54dp.png) no-repeat;margin-left:-5px}@media only screen and (min-resolution:192dpi){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat 0% 0%/100% 100%;-moz-border-image:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) 0}}@media only screen and (-webkit-min-device-pixel-ratio:2){#logo{background:url(//www.google.com/images/branding/googlelogo/2x/googlelogo_color_150x54dp.png) no-repeat;-webkit-background-size:100% 100%}}#logo{display:inline-block;height:54px;width:150px}
  </style>
  <a href=//www.google.com/><span id=logo aria-label=Google></span></a>
  <p><b>403.</b> <ins>That’s an error.</ins>
  <p>Your client does not have permission to get URL <code>/v2/k8s-artifacts-prod/images/pause/manifests/3.9</code> from this server.  <ins>That’s all we know.</ins>

[mrbobbytables]

@zaro This seems to occur more frequently with Hetzner. =/
Unfortunately, there isn't really anything we can do from the project's side of things. Hosting these images is incredibly expensive (to the tune of several million USD/year), and we've had to adopt this way of hosting to help us manage those expenses.

The best advise I can give is to reach out to Hetzner and in the mean time setup a local mirror and reconfigure your cluster to use that.

[zaro]

@mrbobbytables thnak you for the information.

Yes it seems it's some weird Hetzner problem because actually affects only one of their regions, we moved the VMs to another region and everything works just fine.

[simonoff]

@zaro change the default DNS from hetzner's to google's and all will be working great!

[Preisschild]

@zaro From which region to which did you switch? I'm on FSN1 and many servers are being blocked.

@simonoff Unfortunately that doesn't seem to make a difference at least for me.

[simonoff]

nbg1 works fine

[zaro]

@Preisschild I Couldnt get any working server in Falkestenstein. Nuremberg worked as @simonoff pointed out.

[deric]

I got reply from Hetzner support:

We're sorry you are having issues connecting to a service hosted by GCP.

We have had sporadic reports of issues with our IPs being blacklisted/banned by GCP. Unfortunately, it is virtually impossible to get any support from Google, so haven't been able to find out exactly what the issue is, or how to resolve it.

Through some third parties who are also Google clients, we have learned that some of the blocks are due to faulty GeoIP location data, and other blocks are due to network abuse. We are doing our best to investigate all reports and are continually attempting to get support from Google, but at this time we don't know which of our IPs are affected, and we don't know how to properly resolve the issue(s). As such, we cannot offer you a solution.

We apologize for this situation, and assure you that it is just as frustrating for us as it is for you (if not more so). We will continue to investigate these reports and attempt to get in contact with somebody at Google who can hopefully help us.

It might be worth trying to report all blocked IP addresses to Heztner. We have several working addresses in FSN1.

[mrbobbytables]

Might I suggest moving the convo to discuss.k8s.io??
We could continue here - but as a project there really isn't anything actionable for us to do =/