[OKD-SCOS v4.16] UPI install with agent installer or assisted installer fails for 4.16.0-0.okd-scos-2024-08-01-132038

[titou10titou10]

Context

Trying to install a SNO (Single Node) cluster:

• with OKD-SCOS v4.16.0-0.okd-scos-2024-08-01-132038 ("4-scos-next" version)
• on baremetal with UPI
• using agent based installer (ABI) or Assisted Installer (see below)
• on Proxmox (qemu) vm: 40G RAM, 100G SSD, 10 vCPUS

It is important to note that the install works perfectly well with the exact same agent and install config files for

• OKD-SCOS v4.15.0-0.okd-scos-2024-01-18-223523 ("stable")
• OKD-FCOS v4.15.0-0.okd-2024-03-10-010116 ("stable")

I also have tried for a multi-node cluster, it fails the same way

Summary

It fails with the following error from the "release-image-pivot" service:

okd-sno bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied

Details

install-config.yaml:

  apiVersion: v1
  baseDomain: denis.prive
  metadata:
    name: okd-sno
  controlPlane:
    architecture: amd64
    hyperthreading: Enabled   
    name: master 
    replicas: 1
  compute:
    - architecture: amd64
      hyperthreading: Enabled   
      name: worker
      replicas: 0 
  networking:
    networkType: OVNKubernetes
    clusterNetwork:
    - cidr: 10.132.0.0/16
      hostPrefix: 23 
    machineNetwork:
    - cidr: 192.168.5.0/24
    serviceNetwork: 
    - 172.31.0.0/16
  platform:
    none: {} 
  fips: false 
  sshKey: 'ssh-rsa AAAA{...}' 
  pullSecret: '{"auths": {"empty": {"auth": "aWQ6cGFzcwo="}}}'
  additionalTrustBundle: |
    -----BEGIN CERTIFICATE-----
    {...}
    -----END CERTIFICATE-----

agent-config.yaml

  apiVersion: v1beta1
  kind: AgentConfig
  metadata:
    name: okd-sno
  rendezvousIP: 192.168.5.71
  additionalNTPSources:
  - time.nrc.ca
  - time.google.com
  - time.cloudflare.com
  hosts: 
    - hostname: okd-sno
      role: master
      interfaces:
        - name: ens18
          macAddress: xx:xx:xx:xx:xx:71
      rootDeviceHints: 
        deviceName: /dev/sda
      networkConfig: 
        interfaces:
          - name: ens18
            type: ethernet
            state: up
            mac-address: xx:xx:xx:xx:xx:71
            ipv4:
              enabled: true
              dhcp: true
              #auto-dns: true
              auto-dns: false
              auto-gateway: true
              auto-routes: true
            ipv6:
              enabled: false
        dns-resolver:
          config:
            search:
              - denis.prive
            server:
              - 192.168.5.18
              - 8.8.8.8

The install fails after a few minutes the node boots, but the process fails, looping forever

ssh core@okd-sno
sudo su -

"kubelet" service:

journalctl -u kubelet

Aug 18 14:27:53 okd-sno systemd[1]: Starting kubelet.service - Kubernetes Kubelet...
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.367956979 +0000 UTC m=+0.181056791 container create 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.31278308 +0000 UTC m=+0.125882875 image pull 60a77a109240508f118aed8a08c7ffe822c6695f21b7761a9dbe8e008a9d54d2 quay.io/okd>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.593446922 +0000 UTC m=+0.406546764 container init 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (image>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.603769018 +0000 UTC m=+0.416868833 container start 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (imag>
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.606413875 +0000 UTC m=+0.419513627 container attach 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:54 okd-sno inspiring_raman[8518]: quay.io/okd/scos-content@sha256:8d99668e00b1c44a559b7869f660484e58071c581608669ef71c908499c82fc2
Aug 18 14:27:54 okd-sno podman[8503]: 2024-08-18 14:27:54.700681983 +0000 UTC m=+0.513781795 container died 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (image>
Aug 18 14:27:54 okd-sno podman[8532]: 2024-08-18 14:27:54.879290798 +0000 UTC m=+0.151817502 container remove 8a29c9b5a59fb19ee25166335a70204b250f2afd844cfc5e2564aa3a23b42bfd (ima>
Aug 18 14:27:55 okd-sno kubelet.sh[8566]: /usr/local/bin/kubelet.sh: line 7: /usr/bin/kubelet: No such file or directory
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Main process exited, code=exited, status=127/n/a
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Failed with result 'exit-code'.
Aug 18 14:27:55 okd-sno systemd[1]: Failed to start kubelet.service - Kubernetes Kubelet.
Aug 18 14:27:55 okd-sno systemd[1]: kubelet.service: Consumed 1.945s CPU time.
Aug 18 14:28:05 okd-sno systemd[1]: kubelet.service: Scheduled restart job, restart counter is at 3.
Aug 18 14:28:08 okd-sno systemd[1]: Starting kubelet.service - Kubernetes Kubelet...

systemctl status  kubelet

? kubelet.service - Kubernetes Kubelet
    Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; preset: disabled)
   Drop-In: /usr/lib/systemd/system/service.d
            +-10-timeout-abort.conf
    Active: activating (auto-restart) (Result: exit-code) since Sun 2024-08-18 14:33:32 UTC; 3s ago
   Process: 27054 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests (code=exited, status=0/SUCCESS)
   Process: 27055 ExecStartPre=/bin/mkdir --parents /etc/kubernetes/kubelet-plugins/volume/exec (code=exited, status=0/SUCCESS)
   Process: 27058 ExecStartPre=/usr/local/bin/kubelet-pause-image.sh (code=exited, status=0/SUCCESS)
   Process: 27135 ExecStart=/usr/local/bin/kubelet.sh (code=exited, status=127)
  Main PID: 27135 (code=exited, status=127)
       CPU: 1.587s

"release-image-pivot" service:

journalctl -u  release-image-pivot

Aug 18 14:31:57 okd-sno systemd[1]: Starting release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image...
Aug 18 14:31:57 okd-sno bootstrap-pivot.sh[22255]: touch: cannot touch '/usr/.test': Read-only file system
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.315765956 +0000 UTC m=+0.156637380 container create afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (im>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.26111923 +0000 UTC m=+0.101990587 image pull 60a77a109240508f118aed8a08c7ffe822c6695f21b7761a9dbe8e008a9d54d2 quay.io/ok>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.506675673 +0000 UTC m=+0.347546962 container init afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.510429857 +0000 UTC m=+0.351301098 container start afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (ima>
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.511434609 +0000 UTC m=+0.352305922 container attach afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (im>
Aug 18 14:31:58 okd-sno gracious_ride[22350]: quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b
Aug 18 14:31:58 okd-sno podman[22315]: 2024-08-18 14:31:58.614285575 +0000 UTC m=+0.455156810 container died afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno podman[22388]: 2024-08-18 14:31:58.7502221 +0000 UTC m=+0.120921124 container remove afe47be8c1ead2929c3d50f79c88dc21c90a3328dd15df084b04a9ade9ff545d (imag>
Aug 18 14:31:58 okd-sno bootstrap-pivot.sh[22249]: Pulling quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b...
Aug 18 14:31:59 okd-sno bootstrap-pivot.sh[22412]: 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c
Aug 18 14:31:59 okd-sno podman[22412]: 2024-08-18 14:31:59.897322251 +0000 UTC m=+0.930918869 image pull 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c quay.io/o>
Aug 18 14:32:00 okd-sno bootstrap-pivot.sh[22459]: error: Remounting /sysroot read-write: Permission denied
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
Aug 18 14:32:00 okd-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
Aug 18 14:32:00 okd-sno systemd[1]: release-image-pivot.service: Consumed 2.767s CPU time.
Aug 18 14:32:03 okd-sno systemd[1]: Starting release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image...

systemctl status release-image-pivot

× release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image
     Loaded: loaded (/etc/systemd/system/release-image-pivot.service; static)
    Drop-In: /usr/lib/systemd/system/service.d
             +-10-timeout-abort.conf
     Active: failed (Result: exit-code) since Sun 2024-08-18 14:33:06 UTC; 2s ago
    Process: 25649 ExecStart=/usr/local/bin/bootstrap-pivot.sh (code=exited, status=1/FAILURE)
   Main PID: 25649 (code=exited, status=1/FAILURE)
        CPU: 2.237s

 Aug 18 14:33:04 okd-sno podman[25675]: 2024-08-18 14:33:04.192809038 +0000 UTC m=+0.384116252 container died ed9e60ddea89f2f661318693f279a1de1278f60e62af95feb7d1e3d9b44b6f02 (imag>
 Aug 18 14:33:04 okd-sno podman[25707]: 2024-08-18 14:33:04.415531285 +0000 UTC m=+0.184837394 container remove ed9e60ddea89f2f661318693f279a1de1278f60e62af95feb7d1e3d9b44b6f02 (im>
 Aug 18 14:33:04 okd-sno bootstrap-pivot.sh[25649]: Pulling quay.io/okd/scos-content@sha256:94db3cd97915d6016b603aaaedeafcf94a7af3134640de07ae9763d6a0222e5b...
 Aug 18 14:33:05 okd-sno bootstrap-pivot.sh[25740]: 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c
 Aug 18 14:33:05 okd-sno podman[25740]: 2024-08-18 14:33:05.418204678 +0000 UTC m=+0.805573537 image pull 24ce814c87b1268edf974d1cf52b0cd7e811b602dc8ddc3fb1b63d716dea593c quay.io/o>
 Aug 18 14:33:05 okd-sno bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Main process exited, code=exited, status=1/FAILURE
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Failed with result 'exit-code'.
 Aug 18 14:33:06 okd-sno systemd[1]: Failed to start release-image-pivot.service - Pivot bootstrap to the OpenShift Release Image.
 Aug 18 14:33:06 okd-sno systemd[1]: release-image-pivot.service: Consumed 2.237s CPU time.

So the "release-image-pivot" fails to start because this problem?:

bootstrap-pivot.sh[25771]: error: Remounting /sysroot read-write: Permission denied

Other (pertinent?) info:

approve-csr.service

systemctl status approve-csr.service
● approve-csr.service - Approve CSRs during bootstrap phase
     Loaded: loaded (/etc/systemd/system/approve-csr.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sun 2024-08-18 14:27:27 UTC; 48min ago
   Main PID: 7048 (bash)
      Tasks: 2 (limit: 47999)
     Memory: 652.0K
        CPU: 4.558s
     CGroup: /system.slice/approve-csr.service
             ├─  7048 bash /usr/local/bin/approve-csr.sh /opt/openshift/auth/kubeconfig-loopback
             └─162821 sleep 20

Aug 18 15:12:30 okd-sno approve-csr.sh[152879]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:12:50 okd-sno approve-csr.sh[153717]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:10 okd-sno approve-csr.sh[154652]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:30 okd-sno approve-csr.sh[155904]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:13:50 okd-sno approve-csr.sh[156695]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:10 okd-sno approve-csr.sh[158162]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:30 okd-sno approve-csr.sh[159034]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:14:50 okd-sno approve-csr.sh[160435]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:15:10 okd-sno approve-csr.sh[161571]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found
Aug 18 15:15:30 okd-sno approve-csr.sh[162817]: /usr/local/bin/approve-csr.sh: line 11: oc: command not found

podman

podman ps
CONTAINER ID  IMAGE                                                                                             COMMAND               CREATED         STATUS         PORTS       NAMES
7f0dce9bf83f  localhost/podman-pause:4.7.0-1695838680                                                                                 45 minutes ago  Up 45 minutes              691c75a59612-infra
4553e7249821  quay.io/okd/scos-content@sha256:9277ace8253e579f8652eedb149d665287ae5dde4c3ba129c4ff70774bee4ac6  /bin/bash start_d...  44 minutes ago  Up 44 minutes              assisted-db
3279197317a9  quay.io/okd/scos-content@sha256:9277ace8253e579f8652eedb149d665287ae5dde4c3ba129c4ff70774bee4ac6  /assisted-service     44 minutes ago  Up 44 minutes              service
4d7f336db99a  quay.io/okd/scos-content@sha256:32818d2c26e3a11dc7acc313f8d277a3d11602675461af627502d32f919f93a1  next_step_runner ...  44 minutes ago  Up 44 minutes              next-step-runner
25ec9c2bee7a  quay.io/okd/scos-content@sha256:f78e6f0722a5a6a89423046e1abc68bd237551a9d93dfeac0e92e4ea34cb50bb  --role bootstrap ...  41 minutes ago  Up 41 minutes              assisted-installer

podman logs 4d7f336db99a
WARN[0000] Agent Authentication Token not set

Message from the installer:

  ./openshift-install --dir install agent wait-for bootstrap-complete --log-level=debug

  DEBUG OpenShift Installer 4.16.0-0.okd-scos-2024-08-01-132038
  DEBUG Built from commit ade29f4fbb06f3e9059e635dab4409304d70b38d
  DEBUG asset directory: install
  {...}
  INFO Unable to retrieve cluster metadata from Agent Rest API: no clusterID known for the cluster
  INFO Cluster is not ready for install. Check validations
  INFO Registered infra env
  INFO Updated image information (Image type is "full-iso", SSH public key is set)
  DEBUG Cluster validation: All hosts in the cluster are ready to install.
  WARNING Cluster validation: Single-node clusters must have a single control plane node and no workers.
  DEBUG Cluster validation: API virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: API virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: The Cluster Network CIDR is defined.
  DEBUG Cluster validation: The base domain is defined.
  DEBUG Cluster validation: Ingress virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: Ingress virtual IPs are not required: User Managed Networking
  DEBUG Cluster validation: The Machine Network CIDR is defined.
  DEBUG Cluster validation: The Cluster Machine CIDR is not required: User Managed Networking
  DEBUG Cluster validation: The Cluster Network prefix is valid.
  DEBUG Cluster validation: The cluster has a valid network type
  DEBUG Cluster validation: Same address families for all networks.
  DEBUG Cluster validation: No CIDRS are overlapping.
  DEBUG Cluster validation: No ntp problems found
  DEBUG Cluster validation: The Service Network CIDR is defined.
  DEBUG Cluster validation: cnv is disabled
  DEBUG Cluster validation: lso is disabled
  DEBUG Cluster validation: lvm is disabled
  DEBUG Cluster validation: mce is disabled
  DEBUG Cluster validation: odf is disabled
  DEBUG Cluster validation: Platform requirements satisfied
  DEBUG Cluster validation: The pull secret is set.
  INFO Host 33f1ba18-536d-4d4c-bd58-f669b5f90b97: Successfully registered
  DEBUG Host okd-sno validation: Valid inventory exists for the host
  DEBUG Host okd-sno validation: Sufficient CPU cores
  DEBUG Host okd-sno validation: Sufficient minimum RAM
  DEBUG Host okd-sno validation: Sufficient disk capacity
  DEBUG Host okd-sno validation: Sufficient CPU cores for role master
  DEBUG Host okd-sno validation: Sufficient RAM for role master
  DEBUG Host okd-sno validation: Hostname okd-sno is unique in cluster
  DEBUG Host okd-sno validation: Hostname okd-sno is allowed
  DEBUG Host okd-sno validation: Speed of installation disk has not yet been measured
  DEBUG Host okd-sno validation: Host is compatible with cluster platform none
  DEBUG Host okd-sno validation: Host agent is compatible with the service
  DEBUG Host okd-sno validation: No request to skip formatting of the installation disk
  DEBUG Host okd-sno validation: All disks that have skipped formatting are present in the host inventory
  DEBUG Host okd-sno validation: Host is connected
  DEBUG Host okd-sno validation: Media device is connected
  DEBUG Host okd-sno validation: No Machine Network CIDR needed: User Managed Networking
  DEBUG Host okd-sno validation: Host belongs to all machine network CIDRs
  DEBUG Host okd-sno validation: Host has connectivity to the majority of hosts in the cluster
  DEBUG Host okd-sno validation: Platform Standard PC (i440FX + PIIX, 1996) is allowed
  WARNING Host okd-sno validation: Host couldn't synchronize with any NTP server
  DEBUG Host okd-sno validation: Host clock is synchronized with service
  DEBUG Host okd-sno validation: All required container images were either pulled successfully or no attempt was made to pull them
  DEBUG Host okd-sno validation: Network latency requirement has been satisfied.
  DEBUG Host okd-sno validation: Packet loss requirement has been satisfied.
  DEBUG Host okd-sno validation: Host has been configured with at least one default route.
  DEBUG Host okd-sno validation: Domain name resolution for the api.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Domain name resolution for the api-int.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Domain name resolution for the *.apps.okd-sno.denis.prive domain was successful or not required
  DEBUG Host okd-sno validation: Host subnets are not overlapping
  DEBUG Host okd-sno validation: No IP collisions were detected by host 33f1ba18-536d-4d4c-bd58-f669b5f90b97
  DEBUG Host okd-sno validation: cnv is disabled
  DEBUG Host okd-sno validation: lso is disabled
  DEBUG Host okd-sno validation: lvm is disabled
  DEBUG Host okd-sno validation: mce is disabled
  DEBUG Host okd-sno validation: odf is disabled
  WARNING Host okd-sno: updated status from discovering to insufficient (Host cannot be installed due to following failing validation(s): Host couldn't synchronize with any NTP server)
  WARNING Cluster validation: The cluster has hosts that are not ready to install.
  INFO Cluster validation: The cluster has the exact amount of dedicated control plane nodes.
  INFO Host okd-sno: updated status from insufficient to known (Host is ready to be installed)
  INFO Cluster is ready for install
  INFO Cluster validation: All hosts in the cluster are ready to install.
  INFO Preparing cluster for installation
  INFO Host okd-sno: updated status from known to preparing-for-installation (Host finished successfully to prepare for installation)
  INFO Host okd-sno validation: Host NTP is synced
  INFO Host okd-sno: validation 'ntp-synced' is now fixed
  INFO Host okd-sno: New image status quay.io/okd/scos-content@sha256:f78e6f0722a5a6a89423046e1abc68bd237551a9d93dfeac0e92e4ea34cb50bb. result: success. time: 5.11 seconds; size: 332.66 Megabytes; download rate: 68.29 MBps
  INFO Host okd-sno: updated status from preparing-for-installation to preparing-successful (Host finished successfully to prepare for installation)
  INFO Cluster installation in progress
  INFO Host okd-sno: updated status from preparing-successful to installing (Installation is in progress)
  INFO Host: okd-sno, reached installation stage Installing: bootstrap
  INFO Host: okd-sno, reached installation stage Waiting for bootkube

[madpearl]

I've just checked that, agent-based installer is not a part of docs.okd.io, you can find it on docs.openshift.com. May be this method is not supported by OKD!

[titou10titou10]

Follow-up

Also fails with the Assisted Installer:

The installation fails with the Assisted Installer in the same way. Not a big surprise as the assisted installer uses the agent installer under the hood
I use the technique described here with

RELEASE_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038","version":"4.16.0-0.okd-scos-2024-08-01-132038","default":true,"support_level":"beta"}]'

Workaround for a success with Assisted Installer:

The installation works when replacing the embedded FCOS bootstrap OS by RHCOS:

OS_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","url":"https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/4.16.3/rhcos-4.16.3-x86_64-live.x86_64.iso","version":"416.94.202406251923-0"}]'
RELEASE_IMAGES: '[{"openshift_version":"4.16","cpu_architecture":"x86_64","cpu_architectures":["x86_64"],"url":"quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038","version":"4.16.0-0.okd-scos-2024-08-01-132038","default":true,"support_level":"beta"}]'

Workaround for a success with Agent Installer (ABI):

Overriding the bootstrap OS image with a RHCOS image make the installation a success also when installing OKD via ABI by using the following when building the install image:

export OPENSHIFT_INSTALL_OS_IMAGE_OVERRIDE=https://mirror.openshift.com/pub/openshift-v4/x86_64/dependencies/rhcos/4.16/4.16.3/rhcos-4.16.3-x86_64-live.x86_64.iso
oc adm release extract --command=openshift-install quay.io/okd/scos-release:4.16.0-0.okd-scos-2024-08-01-132038
./openshift-install agent create image --dir install --log-level=debug

I did not choose a random OS bootstrap OS image, this is the one for v4.16 specified for an OCP installation via the ABI as specified here: https://github.com/openshift/assisted-service/blob/d3324b06a7c7772f4619c3ab13dd8c0706e55fd9/deploy/podman/configmap.yml#L25

Q:

• is it planned for the next version of OKD-SCOS to include an SCOS/RHCOS bootstrap image instead of the FCOS OS image?

[arichard42]

Hi,

I have encountered the exact same issue.

Overriding the bootstrap OS image with a RHCOS image as proposed is not working for me as my lab hardware is using an LSI hardware (Dell Perc H310 using an IT firmware) that is not recognize by the RHCOS image (the mpt3sas driver on RHCOS has disabled the support for this driver).

Also the other alternative to use openshift-install-linux-4.15.0-0.okd-scos-2024-01-18-223523 do not work for me : the rendezvous host fails to start the bootstrap with an error 'pull secret for new cluster is invalid: pull secret must contain auth for "registry.ci.openshift.org"'.

@titou10titou10, shouldn't you open an issue about this problem ?

Best Regards,

Alain

[akiselev1]

I reproduced this problem with 4.16.0-0.okd-scos-2024-09-27-110344 image on Dell PowerEdge R740.
The error message looks slightly different now:
okd-sno bootstrap-pivot.sh[2073771]: error: Read-only file system
But the rest is exactly the same.

I also verified the proposed RHCOS workaround. That worked fine for me.

[Prashanth684]

We have OKD releases now based on SCOS bootimages, so this should not be a problem anymore. @titou10titou10 could you give 4.19.0-okd-scos.ec.5 a try - for SNO as well as assisted/agent? Thanks!

[titou10titou10]

@titou10titou10 could you give 4.19.0-okd-scos.ec.5 a try - for SNO as well as assisted/agent? Thanks!

Sure

[titou10titou10]

Test install of a SNO with ABI of v4.19.0-okd-scos.ec.5 without overriding the bootstrap image with a n RH image:

• The "error: Remounting /sysroot read-write: Permission denied" error is gone: thanks
• During the creation of the boot image, got this warning this seems not to be a problem:
  WARNING base ISO version mismatch in release payload. Expected version 9.0.20250222-0 but found 9.6.20250121-0
  The bootstrap used v9.6.20250121-0 and then reboot (downgrade?) tov9.0.20250222-0

  ./openshift-install coreos print-stream-json | grep x86_64 | grep live
                "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/c9s/builds/9.0.20250222-0/x86_64/scos-9.0.20250222-0-live-iso.x86_64.iso",
                "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/c9s/builds/9.0.20250222-0/x86_64/scos-9.0.20250222-0-live-kernel.x86_64",
                "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/c9s/builds/9.0.20250222-0/x86_64/scos-9.0.20250222-0-live-initramfs.x86_64.img",
                "location": "https://rhcos.mirror.openshift.com/art/storage/prod/streams/c9s/builds/9.0.20250222-0/x86_64/scos-9.0.20250222-0-live-rootfs.x86_64.img",

  rpm-ostree status
  State: idle
  Deployments:
  ● ostree-unverified-registry:quay.io/okd/scos-content@sha256:cfedd417112c2d03756013a87b72ccf7ff94dea04dbc1a40fef1909f14d2703a
               Digest: sha256:65a8744c8542b17fad9de33854d9e92220fbf8b96b2b8d25124858c136401188
               Version: 9.0.20250318-0 (2025-03-19T15:10:12Z)
  

• Theassisted-service-dbservice still fails and has to be patched as described here: Please document in a "known issues" section that installing OKD v4.16/v4.17 in UPI+ABI/AI on baremetal fails okd-web#47 (comment)
• Theoperator-controller-controller-managerfails to start with: /etc/docker is not a directory. ssh to the node then sudo make /etc/dockersolved the problem:

oc events -n openshift-operator-controller operator-controller-controller-manager-cc5cdb967-7qgh2
LAST SEEN              TYPE      REASON              OBJECT                                                        MESSAGE
25m                    Normal    SuccessfulCreate    ReplicaSet/operator-controller-controller-manager-cc5cdb967   Created pod: operator-controller-controller-manager-cc5cdb967-vdb56
25m                    Normal    ScalingReplicaSet   Deployment/operator-controller-controller-manager             Scaled up replica set operator-controller-controller-manager-cc5cdb967 from 0 to 1
25m                    Normal    Scheduled           Pod/operator-controller-controller-manager-cc5cdb967-vdb56    Successfully assigned openshift-operator-controller/operator-controller-controller-manager-cc5cdb967-vdb56 to okd-sno
23m (x9 over 25m)      Warning   FailedMount         Pod/operator-controller-controller-manager-cc5cdb967-vdb56    MountVolume.SetUp failed for volume "ca-certs" : configmap references non-existent config key: ca-bundle.crt
21m (x10 over 25m)     Warning   FailedMount         Pod/operator-controller-controller-manager-cc5cdb967-vdb56    MountVolume.SetUp failed for volume "etc-docker" : hostPath type check failed: /etc/docker is not a directory
2m19s (x15 over 16m)   Warning   FailedMount         Pod/operator-controller-controller-manager-cc5cdb967-vdb56    MountVolume.SetUp failed for volume "etc-docker" : hostPath type check failed: /etc/docker is not a directory
24s                    Normal    SuccessfulCreate    ReplicaSet/operator-controller-controller-manager-cc5cdb967   Created pod: operator-controller-controller-manager-cc5cdb967-7qgh2
24s                    Normal    Scheduled           Pod/operator-controller-controller-manager-cc5cdb967-7qgh2    Successfully assigned openshift-operator-controller/operator-controller-controller-manager-cc5cdb967-7qgh2 to okd-sno
8s (x6 over 24s)       Warning   FailedMount         Pod/operator-controller-controller-manager-cc5cdb967-7qgh2    MountVolume.SetUp failed for volume "etc-docker" : hostPath type check failed: /etc/docker is not a directory

Last remark. I have to re-verify this, but during the first boot, the boot process stops, with a screen asking to configure the network with 2 choices, either "quit" or "configure".
In my case the network is fully configured inagent-config.yaml. So I clicked onquit. When using the RH bootstrap image, if I remember well there is a 15s timeout, and after this timeout the install continues. I do not thing the timeout is there with the SCOS image, forcing the user for a manual intervention. QUite bad. As said. to be re-verified

... and I like the new 4.19 console visual and new features (favorites, helm,...)...but that's another story lol

[titou10titou10]

Last remark. I have to re-verify this, but during the first boot, the boot process stops, with a screen asking to configure the network with 2 choices, either "quit" or "configure". In my case the network is fully configured inagent-config.yaml. So I clicked onquit. When using the RH bootstrap image, if I remember well there is a 15s timeout, and after this timeout the install continues. I do not thing the timeout is there with the SCOS image, forcing the user for a manual intervention. Quite bad. As said. to be re-verified

I retest it, I was wrong. There is no visible countdown as for the RH isos, but after a while (15s as before ?), the installation goes on. The screen in question:

[Prashanth684]

Thanks for the quick feedback:

WARNING base ISO version mismatch in release payload. Expected version 9.0.20250222-0 but found 9.6.20250121-0

This is because the "machine-os-content" image was still pulling in the rhcos iso, this has been fixed recently. On my next promotion of an ec image, this should be addressed. Meanwhile you can use the latest image in the 4.19.0-0.okd-scos stream, but you need a pull secret for that.

Theoperator-controller-controller-managerfails to start with: /etc/docker is not a directory

Seems like other components are hitting this issue as well. See: https://issues.redhat.com//browse/OCPBUGS-54175. There is an ongoing fix for this in the MCO.

For the issue with the assisted-service-db, could you please collect the journal logs for that unit so we can take a look ?

[titou10titou10]

Theoperator-controller-controller-managerfails to start with: /etc/docker is not a directory

Seems like other components are hitting this issue as well. See: https://issues.redhat.com//browse/OCPBUGS-54175. There is an ongoing fix for this in the MCO.

I forgot to says that I had to do this on 1 of my 6 nodes when I upgraded from v4.18.0-okd-scos.4 to v4.18.0-okd-scos.5...

For the issue with the assisted-service-db, could you please collect the journal logs for that unit so we can take a look ?

I have documented the problem in a few issues/discussions since months: #2068 (comment), #2035 (comment) ...

Basically at the very beginning of the install, the systemd serviceassisted-service-dbfails to start and loops again and again. It's role is to start an instance of PostgreSQL for the assisted installer. TheExecStartcommand of the systemd service is

ExecStart=/usr/bin/podman run --net host --user=postgres --cidfile=%t/%n.ctr-id --cgroups=no-conmon --log-driver=journald --rm --pod-id-file=%t/assisted-service-pod.pod-id --sdnotify=conmon --replace -d --name=assisted-db --env-file=/usr/local/share/assisted-service/assisted-db.env $SERVICE_IMAGE /bin/bash start_db.sh

With/bin/bash start_db.sh:

#!/usr/bin/env bash
set -o nounset
set -euo pipefail
set -o errexit
/usr/bin/pg_ctl -D /tmp/postgres/data/ -l /tmp/postgres/logfile start -w
createuser -s admin -h localhost
createdb installer -h localhost
/usr/bin/pg_ctl -D /tmp/postgres/data/ -l /tmp/postgres/logfile stop -w
exec postgres -D /tmp/postgres/data/

When you run the first command by hand (logs copied from my past tests. but it's the same with the latest SCOS iso) . It's going too fast to capture the logs of the pod started by podman to capture the logs...

/usr/bin/pg_ctl -D /tmp/postgres/data/ -l /tmp/postgres/logfile start -w
more log/postgresql-Sat.log
2024-09-28 14:44:21.024 UTC [6] LOG:  starting PostgreSQL 13.16 on x86_64-redhat-linux-gnu, compiled by gcc (GCC) 11.5.0 20240719 (Red Hat 11.5.0-2), 64-bit
2024-09-28 14:44:21.027 UTC [6] LOG:  listening on IPv6 address "::1", port 5432
2024-09-28 14:44:21.027 UTC [6] LOG:  listening on IPv4 address "127.0.0.1", port 5432
2024-09-28 14:44:21.028 UTC [6] FATAL:  could not create lock file "/var/run/postgresql/.s.PGSQL.5432.lock": No such file or directory
2024-09-28 14:44:21.033 UTC [6] LOG:  database system is shut down

The workaround I proposed and use each time I install OKD-SCOS described here okd-project/okd-web#47 (comment), is to add the-o "-k /tmp"parameter to the pg_ctlcommand to specify the place for the lock file. I embed the content of theExecStartcommand with the extra parameter in the systemd service as I do not have access of the container image where the script is

The real fix should be to change the/bin/bash start_db.shscript like this:

#!/usr/bin/env bash
set -o nounset
set -euo pipefail
set -o errexit
/usr/bin/pg_ctl -D /tmp/postgres/data/ -l /tmp/postgres/logfile start -w -o "-k /tmp"
createuser -s admin -h localhost
createdb installer -h localhost
/usr/bin/pg_ctl -D /tmp/postgres/data/ -l /tmp/postgres/logfile stop -w -o "-k /tmp"
exec postgres -D /tmp/postgres/data/ -k /tmp

I thing the script is here: https://github.com/openshift/assisted-service/blob/master/hack/agent_installer/start_db.sh

[Prashanth684]

OK. That makes sense . A similar fix is being discussed to add --tmpfs /var/run/postgresql:rw to the installer service

[titou10titou10]

A similar fix is being discussed to add --tmpfs /var/run/postgresql:rw to the installer service

Yes that's another way to fix the problem, I didn't think about this possibility. My "fix" would be simpler in the meantime...

I tested it with this in the systemdassisted-service-dbunit, and it works as expected

ExecStart=/usr/bin/podman run --tmpfs /var/run/postgresql:rw --net host --user=postgres ...

[Prashanth684]

opened a fix in the agent script : openshift/assisted-service#7458 - the same as your suggested fix.

[titou10titou10]

@Prashanth684 I f I understand well what happened. the just merged fix will be in v4.20.0+, right? Will it be backported to v4.19 or v4.18?

[Prashanth684]

The latest release nightly: 4.19.0-0.okd-scos-2025-03-28-055146 has the fix. I am going to promote this to an ec. Would appreciate if you could quickly test it though.

[titou10titou10]

Test install of a SNO with ABI of v4.19.0-okd-scos.ec.6 without overriding the bootstrap image with an RH image:

• The "error: Remounting /sysroot read-write: Permission denied" error is gone as before
• No more warning for ISO version mismatch when building the iso. Bootstrap and final node uses version 9.0.20250324-0
• There is a new screen (see below) to configure network. The second one appears 15secs after the first one (with silent countdown) and wait again for 15 secs (with a visible countdown..) for a total of 30 secs
• The problem with the systemd"assisted-service-db"unit is indeed solved. No manual patch required
• The node now reboots twice during install compared to one previously
• Theoperator-controller-controller-managerstill fails to start with "/etc/docker is not a directory...". Creating the directory solved the problem
• And something not related to ABI install itself. There is this alert poping up and the corresponding message in theopenshift-console:console pod:

Alert: 100% of the console/console targets in openshift-console namespace have been unreachable for more than 15 minutes. This may be a symptom of network connectivity issues, down nodes, or failures within these components. Assess the health of the infrastructure and nodes running these targets and then contact support.

Message: E0330 13:32:38.406673 1 middleware.go:51] TOKEN_REVIEW: 'GET /metrics' unauthorized, invalid user token, [invalid bearer token, token lookup failed]

Experience is quite smooth now. Thanks

[titou10titou10]

Done a test for a full 3 +3 nodes cluster with v4.19.0-okd-scos.ec.6. Works well.
I added this manifest along theagent-config.yanlandinstall-config.yamlfiles to address the problem with the missing"/etc/docker"directory.
My ABI install in now again fully scripted:

apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
  name: 98-mkdir-docker
  labels:
    machineconfiguration.openshift.io/role: master
spec:
  config:
    ignition:
      version: 3.5.0
    storage:
      directories:
        - path: /etc/docker
          mode: 0755