Skip to content

OKD-152: Add layered build for OKD-SCOS #46

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 4 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
apiVersion: security.openshift.io/v1
kind: SecurityContextConstraints
metadata:
name: scc-admin
allowHostDirVolumePlugin: false
allowHostIPC: false
allowHostNetwork: false
allowHostPID: false
allowHostPorts: false
allowPrivilegeEscalation: true
allowPrivilegedContainer: true
allowedCapabilities: null
defaultAddCapabilities: null
fsGroup:
type: RunAsAny
groups: []
priority: null
readOnlyRootFilesystem: false
requiredDropCapabilities:
- KILL
- MKNOD
- SYS_CHROOT
runAsUser:
type: RunAsAny
seLinuxContext:
type: RunAsAny
supplementalGroups:
type: RunAsAny
users:
- system:serviceaccount:okd-coreos:pipeline
volumes:
- awsElasticBlockStore
- azureDisk
- azureFile
- cephFS
- cinder
- configMap
- csi
- downwardAPI
- emptyDir
- ephemeral
- fc
- flexVolume
- flocker
- gcePersistentDisk
- gitRepo
- glusterfs
- iscsi
- nfs
- persistentVolumeClaim
- photonPersistentDisk
- portworxVolume
- projected
- quobyte
- rbd
- scaleIO
- secret
- storageOS
- vsphere
37 changes: 33 additions & 4 deletions base/tekton.dev/pipelines/okd-coreos.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -26,10 +26,13 @@ spec:
default: "quay.io/okd"
type: string
- name: baseos-container-image-name
default: "centos-stream-coreos-9"
default: "stream-coreos"
type: string
- name: layered-os-container-image-name
default: "stream-coreos-okd"
type: string
- name: extensions-container-image-name
default: "centos-stream-coreos-9-extensions"
default: "stream-coreos-extensions"
type: string
- name: upload-container-images
default: "false"
Expand Down Expand Up @@ -111,14 +114,23 @@ spec:
- name: ws
workspace: shared-workspace
- name: cosa-test
taskRef:
kind: Task
name: cosa-test
params:
- name: variant
value: $(params.variant)
runAfter:
- cosa-build-baseos
workspaces:
- name: ws
workspace: shared-workspace
- name: build-layer
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe we should run the build-layer task after cosa-test and cosa-upload-baseos in order to pull the freshly baseos built during cosa-build-baseos task of the current pipeline. Otherwise, we would pull the baseos image from the latest pipeline run

runAfter:
- cosa-test
taskRef:
kind: Task
name: cosa-test
name: build-layer
workspaces:
- name: ws
workspace: shared-workspace
Expand All @@ -136,7 +148,7 @@ spec:
workspace: shared-workspace
- name: cosa-buildextend
runAfter:
- cosa-test
- cosa-build-extensions
taskRef:
kind: Task
name: cosa-buildextend
Expand Down Expand Up @@ -255,6 +267,23 @@ spec:
workspace: s3-credentials
- name: ws
workspace: shared-workspace
- name: cosa-upload-scosokd
taskRef:
kind: Task
name: cosa-push-layered-container
runAfter:
- build-layer
- cosa-generate-release-meta
workspaces:
- name: ws
workspace: shared-workspace
when:
- input: $(params.upload-container-images)
operator: in
values: ["true"]
- input: $(tasks.build-layer.results.output)
operator: in
values: ["OK"]
finally:
- name: notify-matrix
when:
Expand Down
63 changes: 63 additions & 0 deletions base/tekton.dev/tasks/build-layer.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: build-layer
spec:
steps:
- image: quay.io/podman/stable:latest
name: build-layer
resources: {}
securityContext:
privileged: true
script: |
#!/usr/bin/env bash
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could this Task be separated in future and run in a separate pipeline (like the 'future' payload one, as we can do for okd/fcos) and maintain the coreos pipeline to only build the base os?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's definitely the plan to buid base and layer(s) separately!
I'd like to be able to have one pipeline that can be flexibly configured either way, but nothing will stop you from creating a completely separate pipeline with that task, or even run it as a task run.

set -euxo pipefail

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: whitespaces

dnf install -y jq

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: whitespaces

cd /srv/coreos

OSTREE_VERSION=$(jq -r '."ostree-version"' < builds/latest/$(uname -m)/meta.json)
OSTREE_TARFILE=$(jq -r '.images.ostree.path' < builds/latest/$(uname -m)/meta.json)

# the c9s.repo file is meant to be consumed by a COSA container
# where the RPM GPG keys are shipped in /usr/share/distribution-gpg-keys/centos
# SCOS however ships the keys in the /etc/pki/rpm-gpg
sed -i 's;/usr/share/distribution-gpg-keys/centos;/etc/pki/rpm-gpg;g' src/config/c9s.repo
sed -i 's/RPM-GPG-KEY-CentOS-Official/RPM-GPG-KEY-centosofficial/g' src/config/c9s.repo

cat <<EOF >> src/config/Dockerfile
FROM quay.io/okd/centos-stream-coreos-9:4.14-x86_64
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be nice as a parameter

COPY rpms/ /srv/coreos/rpms/
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be COPY rpms/ /workspace/coreos/rpms/ since this commit 067ce9d
Otherwise we are hitting the issue below :
error: Updating rpm-md repo 'artifacts': /workspace/coreos/rpms/ was not found during the step STEP 5/6: RUN cat /etc/os-release && rpm-ostree --version && ostree --version && rpm-ostree ex rebuild && rpm-ostree cleanup -m && rm -rf /var/lib/unbound /srv/coreos/rpms /etc/rpm-ostree/origin.d /etc/yum.repos.d/c9s.repo && ostree container commit

COPY src/config/c9s.repo /etc/yum.repos.d/c9s.repo
COPY src/config/packages-openshift.yaml /etc/rpm-ostree/origin.d/packages-openshift.yaml
RUN cat /etc/os-release \
&& rpm-ostree --version \
&& ostree --version \
&& rpm-ostree ex rebuild \
&& rpm-ostree cleanup -m \
&& rm -rf /var/lib/unbound /srv/coreos/rpms /etc/rpm-ostree/origin.d /etc/yum.repos.d/c9s.repo \
&& ostree container commit

LABEL io.openshift.release.operator=true \
io.openshift.build.version-display-names="machine-os=CentOS Stream CoreOS" \
io.openshift.build.versions="machine-os=${OSTREE_VERSION}"

EOF

ls /srv/coreos/builds/latest/$(uname -m)/${OSTREE_TARFILE}
podman build \
-f src/config/Dockerfile \
-t ostree-okd \
.
# mkdir /srv/ostree-okd
podman save --format oci-dir ostree-okd:latest -o /srv/ostree-okd
# podman tag localhost/ostree-okd quay.io/okd/stream-coreos-okd
# podman push quay.io/okd/stream-coreos-okd
# TODO Allow for pulling previously built base image, and for pushing the layered image
printf "%s" "OK" > $(results.output.path)
results:
- name: output
workspaces:
- mountPath: /srv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For consistency with 067ce9d we should use /workspace

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And, replace all the references of /srv by /workspace in the script

name: ws
11 changes: 9 additions & 2 deletions base/tekton.dev/tasks/cosa-init.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -56,9 +56,16 @@ spec:
fi

# TODO: Upstream to openshift/os
# Get oc and hypershift from the yum repo in the artifacts image,
# instead of the rhel ose repo.
# Remove OpenShift package layer from SCOS base manifest
sed -i '/- packages-openshift.yaml/d' $(readlink -f src/config/manifest-$(params.variant).yaml)

# Don't inject CVO layers in SCOS base compose
sed -i 's/ostree-container-inject-openshift-cvo-labels: true/ostree-container-inject-openshift-cvo-labels: false/' $(readlink -f src/config/image-$(params.variant).yaml)

# Get oc and hyperkube rpms frpm the artifacts image instead of rhel ose repo
sed -i 's/rhel-9.*-server-ose.*/artifacts/' $(readlink -f src/config/manifest-$(params.variant).yaml)

# Add repo for RPMs from artifacts container image
cat <<EOF >> src/config/c9s.repo

[artifacts]
Expand Down
46 changes: 46 additions & 0 deletions base/tekton.dev/tasks/cosa-push-layered-container.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
apiVersion: tekton.dev/v1beta1
kind: Task
metadata:
name: cosa-push-layered-container
spec:
params:
- name: target-registry
default: quay.io/okd
type: string
- name: container-image-name
default: stream-coreos-okd
type: string
- name: image
default: localhost/ostree-okd
type: string
- name: tag
default: "4.14"
type: string
- name: tag-latest
default: "false"
type: string
steps:
- image: 'quay.io/podman/stable:latest'
name: upload-image
resources: {}
securityContext:
privileged: true
script: |
#!/usr/bin/env bash
set -euxo pipefail

cd /srv/coreos
podman images
podman load -i /srv/ostree-okd
podman tag ostree-okd:latest $(params.target-registry)/$(params.container-image-name):$(params.tag)-$(uname -m)
podman push $(params.target-registry)/$(params.container-image-name):$(params.tag)-$(uname -m) --authfile=$(workspaces.regcred.path)/.dockerconfigjson
# cosa push-container --authfile=$(workspaces.regcred.path)/.dockerconfigjson --image=$(params.image) $(params.target-registry)/$(params.container-image-name)
# if [[ "$(params.tag-latest)" == "true" ]]; then
# # Add stable tag for import into Prow
# cosa push-container --authfile=$(workspaces.regcred.path)/.dockerconfigjson --image=$(params.image) $(params.target-registry)/$(params.container-image-name):$(params.tag)-$(uname -m)
# fi

workspaces:
- name: regcred
- mountPath: /srv
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

name: ws
2 changes: 2 additions & 0 deletions base/tekton.dev/tasks/kustomization.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,10 @@ resources:
- cosa-buildextend.yaml
- cosa-init.yaml
- cosa-push-container.yaml
- cosa-push-layered-container.yaml
- cosa-test.yaml
- cosa-upload-s3.yaml
- cosa-generate-release-meta.yaml
- rpm-artifacts-copy.yaml
- notify-matrix.yaml
- build-layer.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -19,9 +19,11 @@ spec:
- name: target-registry
default: "quay.io/okd"
- name: baseos-container-image-name
default: "centos-stream-coreos-9"
default: "stream-coreos"
- name: layered-os-container-image-name
default: "stream-coreos-okd"
- name: extensions-container-image-name
default: "centos-stream-coreos-9-extensions"
default: "stream-coreos-extensions"
- name: upload-container-images
default: "true"
- name: tag-latest
Expand Down Expand Up @@ -71,6 +73,8 @@ spec:
value: $(tt.params.target-registry)
- name: baseos-container-image-name
value: $(tt.params.baseos-container-image-name)
- name: layered-os-container-image-name
value: $(tt.params.layered-os-container-image-name)
- name: extensions-container-image-name
value: $(tt.params.extensions-container-image-name)
- name: upload-container-images
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -19,14 +19,20 @@ spec:
value: "4.13"
- name: release-stream
value: "stable"
- name: s3-bucket-name
value: "okd-scos"
- name: s3-endpoint-url
value: "https://okd-scos.s3.amazonaws.com/"
- name: rpm-artifacts-image
value: "registry.ci.openshift.org/origin/4.13:artifacts"
- name: target-registry
value: "quay.io/okd"
- name: baseos-container-image-name
value: "centos-stream-coreos-9"
value: "stream-coreos"
- name: layered-os-container-image-name
value: "stream-coreos-okd"
- name: extensions-container-image-name
value: "centos-stream-coreos-9-extensions"
value: "stream-coreos-extensions"
- name: s3-bucket-name
value: "okd-scos"
- name: s3-endpoint-url
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ metadata:
operator.tekton.dev/prune.keep: "1"
operator.tekton.dev/prune.strategy: "keep"
spec:
serviceAccountName: pipeline
params:
- name: repo
value: "https://github.com/openshift/os.git"
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ metadata:
operator.tekton.dev/prune.keep: "1"
operator.tekton.dev/prune.strategy: "keep"
spec:
serviceAccountName: pipeline
params:
- name: repo
value: "https://github.com/openshift/os.git"
Expand All @@ -22,9 +23,11 @@ spec:
- name: target-registry
value: "quay.io/okd"
- name: baseos-container-image-name
value: "centos-stream-coreos-9"
value: "stream-coreos"
- name: layered-os-container-image-name
value: "stream-coreos-okd"
- name: extensions-container-image-name
value: "centos-stream-coreos-9-extensions"
value: "stream-coreos-extensions"
- name: s3-bucket-name
value: "okd-scos"
- name: s3-endpoint-url
Expand All @@ -45,7 +48,7 @@ spec:
value: pipeline-scos-4.13-pvc
podTemplate:
nodeSelector:
kubernetes.io/hostname: host-192-168-111-83
kubernetes.io/hostname: host-192-168-111-83
pipelineRef:
name: okd-coreos
timeouts:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ metadata:
operator.tekton.dev/prune.keep: "1"
operator.tekton.dev/prune.strategy: "keep"
spec:
serviceAccountName: pipeline
params:
- name: repo
value: "https://github.com/openshift/os.git"
Expand All @@ -22,9 +23,11 @@ spec:
- name: target-registry
value: "quay.io/okd"
- name: baseos-container-image-name
value: "centos-stream-coreos-9"
value: "stream-coreos"
- name: layered-os-container-image-name
value: "stream-coreos-okd"
- name: extensions-container-image-name
value: "centos-stream-coreos-9-extensions"
value: "stream-coreos-extensions"
- name: s3-bucket-name
value: "okd-scos"
- name: s3-endpoint-url
Expand Down
Loading