Skip to content

Commit d9a83c2

Browse files
committed
deploy: allow overriding image registry and tags, use fixed images
They can be overridden for all images or per image by setting env variables, for example: IMAGE_REGISTRY=localhost:9000 \ IMAGE_TAG=canary \ CSI_PROVISIONER_TAG=v1.0.2 \ ./deploy/deploy.sh This is useful for users in air-gapped clusters that host their images in a local repository, for developers and for CI testing. We switched to a release model where tags always refer to the same image, therefore we don't need `imagePullPolicy: Always` anymore. We also don't want it for CI testing when using `canary` images, because then images can be loaded once onto a cluster before a test and are guaranteed to not change in the middle of the test. When using kind with locally build images, `imagePullPolicy: Always` breaks deployment because the side-loaded image gets ignored.
1 parent c07eae1 commit d9a83c2

6 files changed

+127
-23
lines changed

README.md

Lines changed: 16 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -18,29 +18,43 @@ You should see an output similar to the following printed on the terminal showin
1818

1919
```shell
2020
applying RBAC rules
21+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/v1.0.1/deploy/kubernetes/rbac.yaml
2122
serviceaccount/csi-provisioner created
2223
clusterrole.rbac.authorization.k8s.io/external-provisioner-runner created
2324
clusterrolebinding.rbac.authorization.k8s.io/csi-provisioner-role created
2425
role.rbac.authorization.k8s.io/external-provisioner-cfg created
2526
rolebinding.rbac.authorization.k8s.io/csi-provisioner-role-cfg created
27+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/v1.0.1/deploy/kubernetes/rbac.yaml
2628
serviceaccount/csi-attacher created
2729
clusterrole.rbac.authorization.k8s.io/external-attacher-runner created
2830
clusterrolebinding.rbac.authorization.k8s.io/csi-attacher-role created
2931
role.rbac.authorization.k8s.io/external-attacher-cfg created
3032
rolebinding.rbac.authorization.k8s.io/csi-attacher-role-cfg created
33+
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/v1.0.1/deploy/kubernetes/rbac.yaml
3134
serviceaccount/csi-snapshotter created
3235
clusterrole.rbac.authorization.k8s.io/external-snapshotter-runner created
3336
clusterrolebinding.rbac.authorization.k8s.io/csi-snapshotter-role created
3437
deploying hostpath components
38+
deploy/hostpath/csi-hostpath-attacher.yaml
39+
using image: quay.io/k8scsi/csi-attacher:v1.0.1
3540
service/csi-hostpath-attacher created
3641
statefulset.apps/csi-hostpath-attacher created
42+
deploy/hostpath/csi-hostpath-plugin.yaml
43+
using image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
44+
using image: quay.io/k8scsi/hostpathplugin:v1.0.1
45+
using image: quay.io/k8scsi/livenessprobe:v1.0.2
46+
service/csi-hostpathplugin created
3747
statefulset.apps/csi-hostpathplugin created
48+
deploy/hostpath/csi-hostpath-provisioner.yaml
49+
using image: quay.io/k8scsi/csi-provisioner:v1.0.1
3850
service/csi-hostpath-provisioner created
3951
statefulset.apps/csi-hostpath-provisioner created
40-
deploying snapshotter
41-
volumesnapshotclass.snapshot.storage.k8s.io/csi-hostpath-snapclass created
52+
deploy/hostpath/csi-hostpath-snapshotter.yaml
53+
using image: quay.io/k8scsi/csi-snapshotter:v1.0.1
4254
service/csi-hostpath-snapshotter created
4355
statefulset.apps/csi-hostpath-snapshotter created
56+
deploying snapshotclass
57+
volumesnapshotclass.snapshot.storage.k8s.io/csi-hostpath-snapclass created
4458
```
4559

4660
The script can also install CRDs that are needed for alpha features,

deploy/deploy-hostpath.sh

Lines changed: 111 additions & 17 deletions
Original file line numberDiff line numberDiff line change
@@ -11,39 +11,133 @@
1111
set -e
1212
set -o pipefail
1313

14-
function image_version () {
14+
BASE_DIR=$(dirname "$0")
15+
K8S_RELEASE=${K8S_RELEASE:-"release-1.13"}
16+
17+
# If set, the following env variables override image registry and/or tag for each of the images.
18+
# They are named after the image name, with hyphen replaced by underscore and in upper case.
19+
#
20+
# - CSI_ATTACHER_REGISTRY
21+
# - CSI_ATTACHER_TAG
22+
# - CSI_NODE_DRIVER_REGISTRAR_REGISTRY
23+
# - CSI_NODE_DRIVER_REGISTRAR_TAG
24+
# - CSI_PROVISIONER_REGISTRY
25+
# - CSI_PROVISIONER_TAG
26+
# - CSI_SNAPSHOTTER_REGISTRY
27+
# - CSI_SNAPSHOTTER_TAG
28+
# - HOSTPATHPLUGIN_REGISTRY
29+
# - HOSTPATHPLUGIN_TAG
30+
#
31+
# Alternatively, it is possible to override all registries or tags with:
32+
# - IMAGE_REGISTRY
33+
# - IMAGE_TAG
34+
# These are used as fallback when the more specific variables are unset or empty.
35+
#
36+
# Beware that the .yaml files do not have "imagePullPolicy: Always". That means that
37+
# also the "canary" images will only be pulled once. This is good for testing
38+
# (starting a pod multiple times will always run with the same canary image), but
39+
# implies that refreshing that image has to be done manually.
40+
#
41+
# As a special case, 'none' as registry removes the registry name.
42+
43+
# The default is to use the RBAC rules that match the image that is
44+
# being used, also in the case that the image gets overridden. This
45+
# way if there are breaking changes in the RBAC rules, the deployment
46+
# will continue to work.
47+
#
48+
# However, such breaking changes should be rare and only occur when updating
49+
# to a new major version of a sidecar. Nonetheless, to allow testing the scenario
50+
# where the image gets overridden but not the RBAC rules, updating the RBAC
51+
# rules can be disabled.
52+
: ${UPDATE_RBAC_RULES:=true}
53+
function rbac_version () {
1554
yaml="$1"
1655
image="$2"
56+
update_rbac="$3"
57+
58+
# get version from `image: quay.io/k8scsi/csi-attacher:v1.0.1`, ignoring comments
59+
version="$(sed -e 's/ *#.*$//' "$yaml" | grep "image:.*$image" | sed -e 's/ *#.*//' -e 's/.*://')"
60+
61+
if $update_rbac; then
62+
# apply overrides
63+
varname=$(echo $image | tr - _ | tr a-z A-Z)
64+
eval version=\${${varname}_TAG:-\${IMAGE_TAG:-\$version}}
65+
fi
66+
67+
# When using canary images, we have to assume that the
68+
# canary images were built from the corresponding branch.
69+
case "$version" in canary) version=master;;
70+
*-canary) version="$(echo "$version" | sed -e 's/\(.*\)-canary/release-\1/')";;
71+
esac
1772

18-
# get version from `image: quay.io/k8scsi/csi-attacher:v1.0.1`
19-
grep "image:.*$image" "$yaml" | sed -e 's/.*:v/v/'
73+
echo "$version"
2074
}
2175

22-
BASE_DIR=$(dirname "$0")
23-
K8S_RELEASE=${K8S_RELEASE:-"release-1.13"}
24-
PROVISIONER_RELEASE=${PROVISIONER_RELEASE:-$(image_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner)}
25-
ATTACHER_RELEASE=${ATTACHER_RELEASE:-$(image_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher)}
26-
SNAPSHOTTER_RELEASE=${SNAPSHOTTER_RELEASE:-$(image_version "${BASE_DIR}/snapshotter/csi-hostpath-snapshotter.yaml" csi-snapshotter)}
76+
# In addition, the RBAC rules can be overridden separately.
77+
CSI_PROVISIONER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner false)/deploy/kubernetes/rbac.yaml"
78+
: ${CSI_PROVISIONER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-provisioner.yaml" csi-provisioner "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
79+
CSI_ATTACHER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher false)/deploy/kubernetes/rbac.yaml"
80+
: ${CSI_ATTACHER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-attacher/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-attacher.yaml" csi-attacher "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
81+
CSI_SNAPSHOTTER_RBAC_YAML="https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter false)/deploy/kubernetes/rbac.yaml"
82+
: ${CSI_SNAPSHOTTER_RBAC:=https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/$(rbac_version "${BASE_DIR}/hostpath/csi-hostpath-snapshotter.yaml" csi-snapshotter "${UPDATE_RBAC_RULES}")/deploy/kubernetes/rbac.yaml}
83+
2784
INSTALL_CRD=${INSTALL_CRD:-"false"}
2885

86+
run () {
87+
echo "$@" >&2
88+
"$@"
89+
}
90+
2991
# apply CSIDriver and CSINodeInfo API objects
3092
if [[ "${INSTALL_CRD}" =~ ^(y|Y|yes|true)$ ]] ; then
3193
echo "installing CRDs"
32-
kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csidriver.yaml --validate=false
33-
kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csinodeinfo.yaml --validate=false
94+
run kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csidriver.yaml --validate=false
95+
run kubectl apply -f https://raw.githubusercontent.com/kubernetes/csi-api/${K8S_RELEASE}/pkg/crd/manifests/csinodeinfo.yaml --validate=false
3496
fi
3597

3698
# rbac rules
3799
echo "applying RBAC rules"
38-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-provisioner/${PROVISIONER_RELEASE}/deploy/kubernetes/rbac.yaml
39-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-attacher/${ATTACHER_RELEASE}/deploy/kubernetes/rbac.yaml
40-
kubectl apply -f https://raw.githubusercontent.com/kubernetes-csi/external-snapshotter/${SNAPSHOTTER_RELEASE}/deploy/kubernetes/rbac.yaml
100+
for component in CSI_PROVISIONER CSI_ATTACHER CSI_SNAPSHOTTER; do
101+
eval current="\${${component}_RBAC}"
102+
eval original="\${${component}_RBAC_YAML}"
103+
if [ "$current" != "$original" ]; then
104+
echo "Using non-default RBAC rules for $component. Changes from $original to $current are:"
105+
diff -c <(wget --quiet -O - "$original") <(if [[ "$current" =~ ^http ]]; then wget --quiet -O - "$current"; else cat "$current"; fi) || true
106+
fi
107+
run kubectl apply -f "${current}"
108+
done
41109

42110
# deploy hostpath plugin and registrar sidecar
43111
echo "deploying hostpath components"
44-
kubectl apply -f ${BASE_DIR}/hostpath
112+
for i in $(ls ${BASE_DIR}/hostpath/*.yaml | sort); do
113+
echo " $i"
114+
cat "$i" | while IFS= read -r line; do
115+
nocomments="$(echo "$line" | sed -e 's/ *#.*$//')"
116+
if echo "$nocomments" | grep -q '^\s*image:\s*'; then
117+
# Split 'image: quay.io/k8scsi/csi-attacher:v1.0.1'
118+
# into image (quay.io/k8scsi/csi-attacher:v1.0.1),
119+
# registry (quay.io/k8scsi),
120+
# name (csi-attacher),
121+
# tag (v1.0.1).
122+
image=$(echo "$nocomments" | sed -e 's;.*image:\s*;;')
123+
registry=$(echo "$image" | sed -e 's;\(.*\)/.*;\1;')
124+
name=$(echo "$image" | sed -e 's;.*/\([^:]*\).*;\1;')
125+
tag=$(echo "$image" | sed -e 's;.*:;;')
126+
127+
# Variables are with underscores and upper case.
128+
varname=$(echo $name | tr - _ | tr a-z A-Z)
129+
130+
# Now replace registry and/or tag, if set as env variables.
131+
# If not set, the replacement is the same as the original value.
132+
prefix=$(eval echo \${${varname}_REGISTRY:-${IMAGE_REGISTRY:-${registry}}}/ | sed -e 's;none/;;')
133+
suffix=$(eval echo :\${${varname}_TAG:-${IMAGE_TAG:-${tag}}})
134+
line="$(echo "$nocomments" | sed -e "s;$image;${prefix}${name}${suffix};")"
135+
echo " using $line" >&2
136+
fi
137+
echo "$line"
138+
done | kubectl apply -f -
139+
done
45140

46-
# deploy snapshotter and snapshotclass
47-
echo "deploying snapshotter and snapshotclass"
48-
kubectl create -f ${BASE_DIR}/snapshotter/csi-hostpath-snapshotter.yaml
141+
# deploy snapshotclass
142+
echo "deploying snapshotclass"
49143
kubectl create -f ${BASE_DIR}/snapshotter/csi-hostpath-snapshotclass.yaml

deploy/hostpath/csi-hostpath-attacher.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -41,7 +41,6 @@ spec:
4141
containers:
4242
- name: csi-attacher
4343
image: quay.io/k8scsi/csi-attacher:v1.0.1
44-
imagePullPolicy: Always
4544
args:
4645
- --v=5
4746
- --csi-address=$(ADDRESS)

deploy/hostpath/csi-hostpath-plugin.yaml

Lines changed: 0 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -38,7 +38,6 @@ spec:
3838
containers:
3939
- name: node-driver-registrar
4040
image: quay.io/k8scsi/csi-node-driver-registrar:v1.0.2
41-
imagePullPolicy: Always
4241
lifecycle:
4342
preStop:
4443
exec:
@@ -77,7 +76,6 @@ spec:
7776
fieldRef:
7877
apiVersion: v1
7978
fieldPath: spec.nodeName
80-
imagePullPolicy: Always
8179
securityContext:
8280
privileged: true
8381
ports:

deploy/hostpath/csi-hostpath-provisioner.yaml

Lines changed: 0 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -48,7 +48,6 @@ spec:
4848
env:
4949
- name: ADDRESS
5050
value: /csi/csi.sock
51-
imagePullPolicy: Always
5251
volumeMounts:
5352
- mountPath: /csi
5453
name: socket-dir

0 commit comments

Comments
 (0)