tilt up with podman fails #9162

Karthik-K-N · 2023-08-10T10:03:22Z

What steps did you take and what happened?

tilt with podman is faling

$ kind version
kind v0.20.0 go1.20.5 darwin/amd64

$ DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock tilt up

$ cat tilt-setting.yaml

default_registry: "gcr.io/you-project-name-here"
enable_providers:
  - kubeadm-bootstrap
  - kubeadm-control-plane
deploy_cert_manager: false

tilt error with webhook diabled

Loading Tiltfile at: /Users/karthikkn/k-workspace/cluster-api/Tiltfile
local: command -v kubectl || true
local: docker version || podman version
local: go env GOOS
 → darwin
local: go env GOARCH
 → amd64
local: kubectl get cm -n kube-public local-registry-hosting || true
local: make -B tilt-prepare && ./hack/tools/bin/tilt-prepare --tools kustomize,envsubst,clusterctl --tilt-settings-file ./tilt-settings.yaml
 → cd hack/tools; go build -tags=tools -o bin/tilt-prepare sigs.k8s.io/cluster-api/hack/tools/tilt-prepare
 → I0810 15:27:43.923934    5031 main.go:161] [main] started
 → I0810 15:27:43.926355    5031 main.go:423] [tools] task group started
 → I0810 15:27:43.926417    5031 main.go:475] [tools/envsubst] task started
 → I0810 15:27:43.926473    5031 main.go:475] [tools/kustomize] task started
 → I0810 15:27:43.926473    5031 main.go:475] [tools/clusterctl] task started
 → I0810 15:27:44.561459    5031 main.go:477] [tools/envsubst] task completed, elapsed: 634.998748ms
 → I0810 15:27:44.563695    5031 main.go:477] [tools/kustomize] task completed, elapsed: 637.137235ms
 → I0810 15:27:56.487972    5031 main.go:477] [tools/clusterctl] task completed, elapsed: 12.561213311s
 → I0810 15:27:56.488029    5031 main.go:425] [tools] task group completed, elapsed: 12.561394697s
 → I0810 15:27:56.488068    5031 main.go:423] [resources] task group started
 → I0810 15:27:56.488120    5031 main.go:475] [resources/clusterctl.crd] task started
 → I0810 15:27:56.488123    5031 main.go:475] [resources/kubeadm-bootstrap] task started
 → I0810 15:27:56.488727    5031 main.go:475] [resources/kubeadm-control-plane] task started
 → I0810 15:27:56.488739    5031 main.go:475] [resources/core] task started
 → I0810 15:27:56.513867    5031 main.go:751] [resources/clusterctl.crd] no changes in the generated yaml
 → I0810 15:27:56.513899    5031 main.go:477] [resources/clusterctl.crd] task completed, elapsed: 25.761284ms
 → I0810 15:27:57.390745    5031 main.go:751] [resources/kubeadm-bootstrap] no changes in the generated yaml
 → I0810 15:27:57.390761    5031 main.go:477] [resources/kubeadm-bootstrap] task completed, elapsed: 902.603905ms
 → I0810 15:27:57.391334    5031 main.go:751] [resources/kubeadm-control-plane] no changes in the generated yaml
 → I0810 15:27:57.391343    5031 main.go:477] [resources/kubeadm-control-plane] task completed, elapsed: 902.555328ms
 → I0810 15:27:57.499042    5031 main.go:751] [resources/core] no changes in the generated yaml
 → I0810 15:27:57.499059    5031 main.go:477] [resources/core] task completed, elapsed: 1.010263075s
 → I0810 15:27:57.499092    5031 main.go:425] [resources] task group completed, elapsed: 1.010994819s
 → I0810 15:27:57.499101    5031 main.go:187] [main] completed, elapsed: 13.574701223s
local: tee bootstrap/kubeadm/.tiltbuild/bin//Dockerfile
Traceback (most recent call last):
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:637:17: in <toplevel>
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:408:24: in enable_providers
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:348:23: in enable_provider
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:278:14: in build_docker_image
Error in local: command "tee bootstrap/kubeadm/.tiltbuild/bin//Dockerfile" failed.
error: exit status 1
stdout: "\n# Tilt image\nFROM golang:1.20.7 as tilt-helper\n# Install delve. Note this should be kept in step with the Go release minor version.\nRUN go install github.com/go-delve/delve/cmd/[email protected]\n# Support live reloading with Tilt\nRUN wget --output-document /restart.sh --quiet https://raw.githubusercontent.com/tilt-dev/rerun-process-wrapper/master/restart.sh  &&     wget --output-document /start.sh --quiet https://raw.githubusercontent.com/tilt-dev/rerun-process-wrapper/master/start.sh &&     chmod +x /start.sh && chmod +x /restart.sh && chmod +x /go/bin/dlv &&     touch /process.txt && chmod 0777 /process.txt `# pre-create PID file to allow even non-root users to run the image`\n\n\n\nFROM golang:1.20.7 as tilt\nWORKDIR /\nCOPY --from=tilt-helper /process.txt .\nCOPY --from=tilt-helper /start.sh .\nCOPY --from=tilt-helper /restart.sh .\nCOPY --from=tilt-helper /go/bin/dlv .\nCOPY $binary_name .\n\n"
stderr: "tee: bootstrap/kubeadm/.tiltbuild/bin//Dockerfile: No such file or directory\n"

tilt error with webhook enabled in tilt-setting.yaml

Loading Tiltfile at: /Users/karthikkn/k-workspace/cluster-api/Tiltfile
local: command -v kubectl || true
local: docker version || podman version
local: go env GOOS
 → darwin
local: go env GOARCH
 → amd64
local: kubectl get cm -n kube-public local-registry-hosting || true
local: make -B tilt-prepare && ./hack/tools/bin/tilt-prepare --tools kustomize,envsubst,clusterctl --tilt-settings-file ./tilt-settings.yaml
 → cd hack/tools; go build -tags=tools -o bin/tilt-prepare sigs.k8s.io/cluster-api/hack/tools/tilt-prepare
 → I0810 15:31:01.138415    5767 main.go:161] [main] started
 → I0810 15:31:01.141678    5767 main.go:423] [tools] task group started
 → I0810 15:31:01.141725    5767 main.go:475] [tools/kustomize] task started
 → I0810 15:31:01.141748    5767 main.go:475] [tools/envsubst] task started
 → I0810 15:31:01.141780    5767 main.go:475] [tools/clusterctl] task started
 → I0810 15:31:01.807255    5767 main.go:477] [tools/kustomize] task completed, elapsed: 665.470656ms
 → I0810 15:31:01.813225    5767 main.go:477] [tools/envsubst] task completed, elapsed: 671.414361ms
 → I0810 15:31:13.714953    5767 main.go:477] [tools/clusterctl] task completed, elapsed: 12.572886089s
 → I0810 15:31:13.715005    5767 main.go:425] [tools] task group completed, elapsed: 12.57305342s
 → I0810 15:31:13.717010    5767 main.go:423] [resources] task group started
 → I0810 15:31:13.717049    5767 main.go:475] [resources/cert-manager-controller] task started
 → I0810 15:31:13.717075    5767 main.go:475] [resources/cert-manager-cainjector] task started
 → I0810 15:31:13.717069    5767 main.go:475] [resources/core] task started
 → I0810 15:31:13.717117    5767 main.go:475] [resources/kubeadm-bootstrap] task started
 → I0810 15:31:13.717125    5767 main.go:475] [resources/cert-manager-webhook] task started
 → I0810 15:31:13.717172    5767 main.go:475] [resources/clusterctl.crd] task started
 → I0810 15:31:13.717161    5767 main.go:475] [resources/cert-manager] task started
 → I0810 15:31:13.717288    5767 main.go:475] [resources/kubeadm-control-plane] task started
 → No default config file available
 → I0810 15:31:13.720482    5767 main.go:477] [resources/cert-manager-webhook] task completed, elapsed: 3.246211ms
 → I0810 15:31:13.721545    5767 main.go:425] [resources] task group completed, elapsed: 4.37391ms
 → E0810 15:31:13.721751    5767 main.go:486] [resources/cert-manager-controller] Recovered from panic: send on closed channel
 → I0810 15:31:13.721806    5767 main.go:477] [resources/cert-manager-controller] task completed, elapsed: 4.747677ms
 → I0810 15:31:13.721817    5767 main.go:477] [resources/cert-manager-cainjector] task completed, elapsed: 4.646655ms
 → F0810 15:31:13.722398    5767 main.go:184] [main] failed to prepare tilt resources: [[resources/cert-manager-webhook] failed to pull quay.io/jetstack/cert-manager-webhook:v1.12.2: failure determining if the image exists in local cache: quay.io/jetstack/cert-manager-webhook:v1.12.2: failure listing container image: quay.io/jetstack/cert-manager-webhook:v1.12.2: Cannot connect to the Docker daemon at unix:///run/user/501/podman/podman.sock. Is the docker daemon running?, [resources/cert-manager-cainjector] failed to pull quay.io/jetstack/cert-manager-cainjector:v1.12.2: failure determining if the image exists in local cache: quay.io/jetstack/cert-manager-cainjector:v1.12.2: failure listing container image: quay.io/jetstack/cert-manager-cainjector:v1.12.2: Cannot connect to the Docker daemon at unix:///run/user/501/podman/podman.sock. Is the docker daemon running?]
Traceback (most recent call last):
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:631:12: in <toplevel>
  /Users/karthikkn/k-workspace/cluster-api/Tiltfile:487:10: in prepare_all
Error in local: command "make -B tilt-prepare && ./hack/tools/bin/tilt-prepare --tools kustomize,envsubst,clusterctl --tilt-settings-file ./tilt-settings.yaml" failed.
error: exit status 1

What did you expect to happen?

Successfully able to run tilt

Cluster API version

Latest main

$ git log --oneline
f9942a9 (HEAD -> main, origin/main, origin/HEAD) Merge pull request #9159 from kubernetes-sigs/dependabot/go_modules/hack/tools/google.golang.org/api-0.136.0
f1ef079 🌱 Bump google.golang.org/api in /hack/tools
d7229e5 Merge pull request #9119 from g-gaston/update-versions-1-6

Kubernetes version

$ kubectl version --short
Flag --short has been deprecated, and will be removed in the future. The --short output will become the default.
Client Version: v1.27.3
Kustomize Version: v5.0.1
Server Version: v1.27.3

Anything else you would like to add?

No response

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

The text was updated successfully, but these errors were encountered:

Karthik-K-N · 2023-08-10T10:04:01Z

@bengentil FYI

killianmuldoon · 2023-08-10T11:10:38Z

/triage accepted

Thanks for opening this! I don't think there's been any work to make the tilt setup work with podman as opposed to Docker to date. The tilt-prepare binary calls the docker client explicitly and I don't know if there is cross compatibility when running podman. It would be great to get it working with podman though!

Prajyot-Parab · 2023-08-10T11:21:45Z

After initial debug -

The required directories are not being created hence above error
How I fixed it manually -

mkdir -p ../cluster-api-provider-ibmcloud/.tiltbuild/bin/
mkdir -p bootstrap/kubeadm/.tiltbuild/bin/
mkdir -p controlplane/kubeadm/.tiltbuild/bin/
mkdir -p ./.tiltbuild/bin/
touch ../cluster-api-provider-ibmcloud/.tiltbuild/bin/Dockerfile
touch bootstrap/kubeadm/.tiltbuild/bin/Dockerfile
touch controlplane/kubeadm/.tiltbuild/bin/Dockerfile
touch ./.tiltbuild/bin/Dockerfile

It should probably be fixed via code similar to this - https://github.com/kubernetes-sigs/cluster-api/blob/main/Tiltfile#L251-L259
should replicate same inside this function - https://github.com/kubernetes-sigs/cluster-api/blob/main/Tiltfile#L261

Prajyot-Parab · 2023-08-10T11:24:46Z

As for the cert manager, kind does not support loading of images via podman (as per my understanding), not sure how can we deal with that, possibly will need some hacky way to move pass it.

Karthik-K-N · 2023-08-10T11:54:15Z

After initial debug -
1. The required directories are not being created hence above error
   How I fixed it manually -
mkdir -p ../cluster-api-provider-ibmcloud/.tiltbuild/bin/
mkdir -p bootstrap/kubeadm/.tiltbuild/bin/
mkdir -p controlplane/kubeadm/.tiltbuild/bin/
mkdir -p ./.tiltbuild/bin/
touch ../cluster-api-provider-ibmcloud/.tiltbuild/bin/Dockerfile
touch bootstrap/kubeadm/.tiltbuild/bin/Dockerfile
touch controlplane/kubeadm/.tiltbuild/bin/Dockerfile
touch ./.tiltbuild/bin/Dockerfile
It should probably be fixed via code similar to this - https://github.com/kubernetes-sigs/cluster-api/blob/main/Tiltfile#L251-L259 should replicate same inside this function - https://github.com/kubernetes-sigs/cluster-api/blob/main/Tiltfile#L261

I think those should be automatically created while building go binaries:

cluster-api/Tiltfile

Lines 251 to 259 in a3dd933

    
           local_resource( 
        
               label.lower() + "_binary", 
        
               cmd = "cd {context};mkdir -p .tiltbuild/bin;{build_cmd}".format( 
        
                   context = context, 
        
                   build_cmd = build_cmd, 
        
               ), 
        
               deps = live_reload_deps, 
        
               labels = [label, "ALL.binaries"], 
        
           )

Prajyot-Parab · 2023-08-10T12:09:32Z

@Karthik-K-N Yes, but seems like it directly goes to build images when podman is being used. (or at-least not executing the binaries related function)

bengentil · 2023-08-10T15:37:13Z

@killianmuldoon / @Prajyot-Parab podman support has been implemented here and it's working with kind (at least on Fedora 38)

@Karthik-K-N reached out as I am the author of the PR, I asked him to create this issue so I can investigate it further

I suspect an issue specific to macos or maybe the tilt version, as only local task seems to run

@Karthik-K-N I just saw you are using DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock can you confirm this is the correct podman socket? I'd expect a socket in the home directory or ssh on macos IIRC
can you post the output of podman system connection list and tilt version please?

I'll try to reproduce with my mac and let you know.

bengentil · 2023-08-10T15:44:50Z

tilt error with webhook enabled in tilt-setting.yaml

what do you mean by webhook enabled? here tilt-prepare gives, what I suspect to be the "correct" error:
Cannot connect to the Docker daemon at unix:///run/user/501/podman/podman.sock. Is the docker daemon running

Prajyot-Parab · 2023-08-10T16:15:56Z

@killianmuldoon / @Prajyot-Parab podman support has been implemented here and it's working with kind (at least on Fedora 38)

@Karthik-K-N reached out as I am the author of the PR, I asked him to create this issue so I can investigate it further

I suspect an issue specific to macos or maybe the tilt version, as only local task seems to run

@Karthik-K-N I just saw you are using DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock can you confirm this is the correct podman socket? I'd expect a socket in the home directory or ssh on macos IIRC can you post the output of podman system connection list and tilt version please?

I'll try to reproduce with my mac and let you know.

@bengentil I was able to reproduce the issues, below are my env details (Not on Mac + No Docker) -

[root@clammily1 ~]# kind version
kind v0.20.0 go1.20.4 linux/amd64

[root@clammily1 ~]# tilt version
v0.33.3, built 2023-07-25

[root@clammily1 ~]# uname -a
Linux clammily1.fyre.ibm.com 5.14.0-70.30.1.el9_0.x86_64 #1 SMP PREEMPT Fri Oct 14 10:30:46 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux

[root@clammily1 ~]# cat /etc/os-release 
NAME="Red Hat Enterprise Linux"
VERSION="9.2 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.2 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.2
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"

[root@clammily1 ~]# podman info
host:
  arch: amd64
  buildahVersion: 1.29.0
  cgroupControllers:
  - cpuset
  - cpu
  - io
  - memory
  - hugetlb
  - pids
  - rdma
  - misc
  cgroupManager: systemd
  cgroupVersion: v2
  conmon:
    package: conmon-2.1.7-1.el9_2.x86_64
    path: /usr/bin/conmon
    version: 'conmon version 2.1.7, commit: 606c693de21bcbab87e31002e46663c5f2dc8a9b'
  cpuUtilization:
    idlePercent: 82.86
    systemPercent: 6.39
    userPercent: 10.75
  cpus: 2
  distribution:
    distribution: '"rhel"'
    version: "9.2"
  eventLogger: journald
  hostname: clammily1.fyre.ibm.com
  idMappings:
    gidmap: null
    uidmap: null
  kernel: 5.14.0-70.30.1.el9_0.x86_64
  linkmode: dynamic
  logDriver: journald
  memFree: 79704064
  memTotal: 1861009408
  networkBackend: netavark
  ociRuntime:
    name: crun
    package: crun-1.8.4-1.el9_2.x86_64
    path: /usr/bin/crun
    version: |-
      crun version 1.8.4
      commit: 5a8fa99a5e41facba2eda4af12fa26313918805b
      rundir: /run/user/0/crun
      spec: 1.0.0
      +SYSTEMD +SELINUX +APPARMOR +CAP +SECCOMP +EBPF +CRIU +YAJL
  os: linux
  remoteSocket:
    path: /run/podman/podman.sock
  security:
    apparmorEnabled: false
    capabilities: CAP_CHOWN,CAP_DAC_OVERRIDE,CAP_FOWNER,CAP_FSETID,CAP_KILL,CAP_NET_BIND_SERVICE,CAP_SETFCAP,CAP_SETGID,CAP_SETPCAP,CAP_SETUID,CAP_SYS_CHROOT
    rootless: false
    seccompEnabled: true
    seccompProfilePath: /usr/share/containers/seccomp.json
    selinuxEnabled: false
  serviceIsRemote: false
  slirp4netns:
    executable: /usr/bin/slirp4netns
    package: slirp4netns-1.2.0-3.el9.x86_64
    version: |-
      slirp4netns version 1.2.0
      commit: 656041d45cfca7a4176f6b7eed9e4fe6c11e8383
      libslirp: 4.4.0
      SLIRP_CONFIG_VERSION_MAX: 3
      libseccomp: 2.5.2
  swapFree: 16986316800
  swapTotal: 17175670784
  uptime: 8h 6m 10.00s (Approximately 0.33 days)
plugins:
  authorization: null
  log:
  - k8s-file
  - none
  - passthrough
  - journald
  network:
  - bridge
  - macvlan
  volume:
  - local
registries:
  search:
  - registry.access.redhat.com
  - registry.redhat.io
  - docker.io
store:
  configFile: /etc/containers/storage.conf
  containerStore:
    number: 1
    paused: 0
    running: 1
    stopped: 0
  graphDriverName: overlay
  graphOptions:
    overlay.mountopt: nodev,metacopy=on
  graphRoot: /var/lib/containers/storage
  graphRootAllocated: 250059685888
  graphRootUsed: 23040110592
  graphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  imageCopyTmpDir: /var/tmp
  imageStore:
    number: 28
  runRoot: /run/containers/storage
  transientStore: false
  volumePath: /var/lib/containers/storage/volumes
version:
  APIVersion: 4.4.1
  Built: 1686828714
  BuiltTime: Thu Jun 15 04:31:54 2023
  GitCommit: ""
  GoVersion: go1.19.9
  Os: linux
  OsArch: linux/amd64
  Version: 4.4.1

[root@clammily1 ~]# podman system connection list
Name        URI         Identity    Default

Prajyot-Parab · 2023-08-10T16:17:32Z

I tackled the socket error with steps mentioned here - https://cluster-api.sigs.k8s.io/developer/tilt.html#using-podman

bengentil · 2023-08-10T16:23:59Z

@Prajyot-Parab did you reproduce after fixing the socket (starting the service and setting the appropriate DOCKER_HOST)?

Prajyot-Parab · 2023-08-10T16:47:16Z

@Prajyot-Parab did you reproduce after fixing the socket (starting the service and setting the appropriate DOCKER_HOST)?

These 2 steps fixed socket issue for me to be precise here

Enable the podman unix socket (eg. systemctl --user enable --now podman.socket on Fedora)
Define the env variable DOCKER_HOST to the right socket while running tilt (eg. DOCKER_HOST=unix:///run/user/$(id -u)/podman/podman.sock tilt up)

after fixing socket, we ran into kind load images related issue for cert_manager (so we disabled it to move forward)
but later faced issues related to Error in local: command "tee bootstrap/kubeadm/.tiltbuild/bin//Dockerfile" failed.
after manually creating those paths as mentioned in my previous comment I was able to move ahead, but then I had few other issues ahead which may or may not be generic.

bengentil · 2023-08-10T19:20:21Z

On a fresh setup on my mac I can reproduce the 2 issues:

1. error while loading cert-manager images

I think it is this issue: kubernetes-sigs/kind#2760

I had to create a docker symlink to podman in my PATH, then I pulled manually the images with podman pull to workaround this issue.

I think I don't have this issue on fedora because I have the podman-docker package which setup this symlink.

2. directory not created / build not done before generating the Dockerfile

build_go_binary is called but any local_resource is silently skipped,
I need to investigate this futher, the only things I know are:

if I add a simple local_resource with a echo command, it's not being run but the same command as local is executed 🤷
I tried to play with allow_parallel, auto_init or setting max_parallel_updates=1 without success

bengentil · 2023-08-10T22:30:08Z

With #9165, tilt is working on a fresh macos/podman install.

To summarise, you need:

the fix in the PR 🐛 tilt: ensure .tiltbuild/bin directory is created early enough, add tilt troubleshooting guide #9165
the correct socket exported as DOCKER_HOST (on macos it's the one displayed when you start the podman machine, in your home not the ssh one)
a symlink docker to your podman binary + manual pull of cert-manager images to workaround "kind load" tries to call docker though podman is used as runtime kind#2760 (only if you don't have a docker command in your PATH)
a working registry (gcr.io/you-project-name-here above in the tilt-setting.yaml is not a correct one), personally I use a local registry with kind

I hope this helps!

Karthik-K-N · 2023-08-11T12:35:24Z

With #9165, tilt is working on a fresh macos/podman install.

To summarise, you need:

* the fix in the PR [🐛 tilt: ensure Dockerfile directory for podman #9165](https://github.com/kubernetes-sigs/cluster-api/pull/9165)

* the correct socket exported as DOCKER_HOST (on macos it's the one displayed when you start the podman machine, in your home not the ssh one)

* a symlink docker to your podman binary + manual pull of cert-manager images to workaround ["kind load" tries to call docker though podman is used as runtime kind#2760](https://github.com/kubernetes-sigs/kind/issues/2760) (only if you don't have a docker command in your PATH)

* a working registry (gcr.io/you-project-name-here above in the `tilt-setting.yaml` is not a correct one), personally I use a [local registry with kind](https://kind.sigs.k8s.io/docs/user/local-registry/)

I hope this helps!

thanks for the quick help, I will check and update

KeerthanaAP · 2023-08-16T12:04:06Z

@bengentil tried the mentioned steps but the controllers were not running successfully and not able to pull or push the images locally as well.

Created a local registry with port 5001 and a kind cluster "capi-test"

 % kind version
kind v0.20.0 go1.20.4 darwin/arm64  

% tilt version
v0.33.3, built 2023-07-25

% cat tilt-settings.yaml 
default_registry: "localhost:5001"
build_engine: "podman"
provider_repos:
- ../cluster-api-provider-ibmcloud
enable_providers:
- ibmcloud
- kubeadm-bootstrap
- kubeadm-control-plane
kustomize_substitutions:
  IBMCLOUD_API_KEY: “***”
  PROVIDER_ID_FORMAT: "v2"
  EXP_CLUSTER_RESOURCE_SET: "true"
extra_args:
  ibmcloud:
    - '-v=5'  
    
Error:
 + podman push --format=docker localhost:5001/gcr.io_k8s-staging-cluster-api_cluster-api-controller:tilt-build-1692183577
Getting image source signatures
Copying blob sha256:735154a12616034ab862080eba4fc111d021b710fb8d08621c5c313892dbf50d
Copying blob sha256:cb3bf648496bc8eca4baaab2684011f6125fb1d47ecf69ce1a3467604582c486
Copying blob sha256:43b21807e78f9a417003c2c2530013f6ba012c005d5823a3003fedaf62191b2c
Copying blob sha256:099287c4a608101b214c0869657e9ac18b2124f16a92cd526817f200161b02d1
Copying blob sha256:af2e9ca2509082a61d746c98c75f877b7fac7316364b97889e5e913445914517
Copying blob sha256:959cb25b9021114160734dff36b8d1711806605403630b1d16e0ec7cd2d0466a
Copying blob sha256:d893c8de82e9ba4bfadeb7fed602d2829ad7c65cd0afc6fa25ce7d18ef7311b5
Copying blob sha256:9c3220d1b30771483290aff653affdaaffa7ce36c0e3b3431a16fea52e0586f8
Copying blob sha256:f2ce857878499e39ae59a6d5089d6658ea261779be5bda87cc898dcfc7506ab4
Copying blob sha256:d9a91f482428963ff6952fde02bcf7bde51116113b52487ca4885afe6741d8e6
Copying blob sha256:73686f14d60473e25bfc61ac534481031981390dbf7f3b382aa816a9d23c2411
Error: trying to reuse blob sha256:cb3bf648496bc8eca4baaab2684011f6125fb1d47ecf69ce1a3467604582c486 at destination: pinging container registry localhost:5001: Get "https://localhost:5001/v2/": http: server gave HTTP response to HTTPS client
sh -c "set -ex\npodman build -t $EXPECTED_REF --build-arg binary_name=manager --target tilt ./.tiltbuild/bin/\npodman push --format=docker $EXPECTED_REF" exited with exit code 125

Build Failed: Custom build "sh -c \"set -ex\\npodman build -t $EXPECTED_REF --build-arg binary_name=manager --target tilt ./.tiltbuild/bin/\\npodman push --format=docker $EXPECTED_REF\"" failed: exit status 125

bengentil · 2023-08-16T17:05:52Z

The error is Get "https://localhost:5001/v2/": http: server gave HTTP response to HTTPS client you need to tell podman the registry is unsecure with this configuration in a .conf file in /etc/containers/registries.conf.d:

[[registry]]
location = "localhost:5001"
insecure = true

On macos, the registry configuration is in the podman VM, as far as I know there is no way to manage this config from the host, you need to podman machine ssh <your_machine> and edit the configuration in the VM.

All of this is not needed if:

you run a local registry on :5000 (insecure by default, unfortunately the port is taken on macos)
you run a secure registry (either one with your own certificates or a public one like quay, ghcr...)

KeerthanaAP · 2023-08-17T05:12:21Z

/etc/containers/registries.conf.d:

The error is Get "https://localhost:5001/v2/": http: server gave HTTP response to HTTPS client you need to tell podman the registry is unsecure with this configuration in a .conf file in /etc/containers/registries.conf.d:
[[registry]]
location = "localhost:5001"
insecure = true
On macos, the registry configuration is in the podman VM, as far as I know there is no way to manage this config from the host, you need to podman machine ssh <your_machine> and edit the configuration in the VM.

All of this is not needed if:

you run a local registry on :5000 (insecure by default, unfortunately the port is taken on macos)

you run a secure registry (either one with your own certificates or a public one like quay, ghcr...)

Thank you. I will try this and update.

Karthik-K-N · 2023-08-23T16:04:32Z

@bengentil thanks for the quick fix we were able to verify it on macos with only podman

Create fresh podman machine by deleting old one

$ podman machine stop podman-machine-default
$ podman machine rm podman-machine-default
$ podman machine init 
$ podman machine start

Make podman to use insecure local registry and restart podman machine

$ podman machine ssh

$ sudo vi /etc/containers/registries.conf

# at the end add these lines

[[registry]]
location = "localhost:5001"
insecure = true


$ podman machine stop
$ podman machine start

Emulate docker cli with podman: https://podman-desktop.io/docs/migrating-from-docker/emulating-docker-cli-with-podman
Create a kind cluster with registry using script from here: https://kind.sigs.k8s.io/docs/user/local-registry/#create-a-cluster-and-registry

make sure update script to create two nodes

cat <<EOF | kind create cluster --config=-
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
- role: worker
containerdConfigPatches:
- |-
  [plugins."io.containerd.grpc.v1.cri".registry]
    config_path = "/etc/containerd/certs.d"
EOF

Set kind cluster name to use with tilt

$ export CAPI_KIND_CLUSTER_NAME=kind

$ cat tilt-setting.json in cluster-api repo

default_registry: "localhost:5001"
build_engine: "podman"
provider_repos:
  - ../cluster-api-provider-ibmcloud
enable_providers:
  - ibmcloud
  - kubeadm-bootstrap
  - kubeadm-control-plane
kustomize_substitutions:
  POWERVS_PROVIDER_ID_FORMAT: "v1"
  EXP_CLUSTER_RESOURCE_SET: "true"
  CLUSTER_TOPOLOGY: "true"

Start tilt

$ tilt up

k8s-ci-robot added kind/bug Categorizes issue or PR as related to a bug. needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 10, 2023

k8s-ci-robot added triage/accepted Indicates an issue or PR is ready to be actively worked on. and removed needs-triage Indicates an issue or PR lacks a `triage/foo` label and requires one. labels Aug 10, 2023

bengentil mentioned this issue Aug 10, 2023

🐛 tilt: ensure .tiltbuild/bin directory is created early enough, add tilt troubleshooting guide #9165

Merged

k8s-ci-robot closed this as completed in #9165 Aug 18, 2023

tilt up with podman fails #9162

tilt up with podman fails #9162

Comments

Karthik-K-N commented Aug 10, 2023

What steps did you take and what happened?

What did you expect to happen?

Cluster API version

Kubernetes version

Anything else you would like to add?

Label(s) to be applied

Karthik-K-N commented Aug 10, 2023

Uh oh!

killianmuldoon commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Prajyot-Parab commented Aug 10, 2023

Uh oh!

Prajyot-Parab commented Aug 10, 2023

Uh oh!

Karthik-K-N commented Aug 10, 2023

Uh oh!

Prajyot-Parab commented Aug 10, 2023

Uh oh!

bengentil commented Aug 10, 2023

Uh oh!

bengentil commented Aug 10, 2023

Uh oh!

Prajyot-Parab commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Prajyot-Parab commented Aug 10, 2023

Uh oh!

bengentil commented Aug 10, 2023

Uh oh!

Prajyot-Parab commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bengentil commented Aug 10, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

1. error while loading cert-manager images

2. directory not created / build not done before generating the Dockerfile

Uh oh!

bengentil commented Aug 10, 2023

Uh oh!

Karthik-K-N commented Aug 11, 2023

Uh oh!

KeerthanaAP commented Aug 16, 2023

Uh oh!

bengentil commented Aug 16, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

KeerthanaAP commented Aug 17, 2023

Uh oh!

Karthik-K-N commented Aug 23, 2023

Uh oh!

killianmuldoon commented Aug 10, 2023 •

edited

Loading

Prajyot-Parab commented Aug 10, 2023 •

edited

Loading

Prajyot-Parab commented Aug 10, 2023 •

edited

Loading

bengentil commented Aug 10, 2023 •

edited

Loading

bengentil commented Aug 16, 2023 •

edited

Loading