Skip to content

Commit 259bd33

Browse files
openshift-merge-robotopenshift-merge-robot
authored
Merge pull request #16571 from smarterclayton/bootstrap_config
Automatic merge from submit-queue. Allow bootstrap configuration to be configured and reentrant Make bootstrapping a real production node possible. 1. Simplify and streamline the process whereby the bootstrap config is looked up. 1. `--bootstrap-config-name` can be used to customize which config is looked up (one per node group) 2. Any failure in fetching node config results in termination of the pass - no client side defaulting 3. Handle 0.0.0.0 in dnsIP as a special case which results in the local node IP lookup 2. Backport the one remaining cert rotation patch and make client and server node side cert rotation mandatory when bootstrapping 1. Fix a number of small issues upstream where node bootstrapping is not reentrant 2. pass cert-dir to the kubelet instead of passing individual key and cert, which was preventing rotation from working 3. Make `openshift start network` work podified 1. Allow kubeconfig to be specified on the CLI and override the node-config 2. The proxy healthz was not starting due to missing config - this has been corrected 3. Provide a daemonset example that correctly starts OpenShift SDN in a pod 4. Fix a few minor bugs in openshift-sdn There is still one bug outstanding upstream that can be fixed separately - the kubelet client rotation can fail due to the cert expiring and be unable to get new certs, so it never exits. Tested the following scenario extensively (requires a new openshift/node image tagged as v3.7.0-alpha.1): 1. Update master-config with cert signing on on 10m rotation 2. create a node-config `oc create configmap -n openshift-node node-config --from-file=node-config.yaml=contrib/kubernetes/default-node-config.yaml` 3. start node in bootstrapping mode `openshift start node --bootstrap-config-name=node-config --config=/etc/origin/node/node-config.yaml --enable=kubelet --loglevel=3` (which has it run only the kubelet) 4. run a background `oc observe csr -- oc adm certificate approve` to approve both csr 5. run `oc create -f contrib/kubernetes/static/network-policy.yaml` 6. run `oc create -f contrib/kubernetes/static/network-daemonset.yaml` 7. verify the daemonset starts correctly and that it passes health checks 8. launch a new pod and verify it has dns `oc run --restart=Never --attach -it --image=centos:7 -- /bin/bash` and then `yum install bind-utils -y && dig +search kubernetes.default.svc` 9. launch two pods in two namespaces and verify multi tenant SDN works Follow up for the daemonset - openshift-sdn expects to have access to the dockershim.sock which this doesn't bind mount in.
2 parents 14566cf + ae05ccd commit 259bd33

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

61 files changed

+1668
-597
lines changed

‎contrib/completions/bash/openshift

+4-2
Original file line numberDiff line numberDiff line change
@@ -33708,6 +33708,8 @@ _openshift_start_network()
3370833708
local_nonpersistent_flags+=("--kubernetes=")
3370933709
flags+=("--latest-images")
3371033710
local_nonpersistent_flags+=("--latest-images")
33711+
flags+=("--listen=")
33712+
local_nonpersistent_flags+=("--listen=")
3371133713
flags+=("--network-plugin=")
3371233714
local_nonpersistent_flags+=("--network-plugin=")
3371333715
flags+=("--recursive-resolv-conf=")
@@ -33731,8 +33733,8 @@ _openshift_start_node()
3373133733
flags_with_completion=()
3373233734
flags_completion=()
3373333735

33734-
flags+=("--bootstrap")
33735-
local_nonpersistent_flags+=("--bootstrap")
33736+
flags+=("--bootstrap-config-name=")
33737+
local_nonpersistent_flags+=("--bootstrap-config-name=")
3373633738
flags+=("--config=")
3373733739
flags_with_completion+=("--config")
3373833740
flags_completion+=("__handle_filename_extension_flag yaml|yml")

‎contrib/completions/zsh/openshift

+4-2
Original file line numberDiff line numberDiff line change
@@ -33857,6 +33857,8 @@ _openshift_start_network()
3385733857
local_nonpersistent_flags+=("--kubernetes=")
3385833858
flags+=("--latest-images")
3385933859
local_nonpersistent_flags+=("--latest-images")
33860+
flags+=("--listen=")
33861+
local_nonpersistent_flags+=("--listen=")
3386033862
flags+=("--network-plugin=")
3386133863
local_nonpersistent_flags+=("--network-plugin=")
3386233864
flags+=("--recursive-resolv-conf=")
@@ -33880,8 +33882,8 @@ _openshift_start_node()
3388033882
flags_with_completion=()
3388133883
flags_completion=()
3388233884

33883-
flags+=("--bootstrap")
33884-
local_nonpersistent_flags+=("--bootstrap")
33885+
flags+=("--bootstrap-config-name=")
33886+
local_nonpersistent_flags+=("--bootstrap-config-name=")
3388533887
flags+=("--config=")
3388633888
flags_with_completion+=("--config")
3388733889
flags_completion+=("__handle_filename_extension_flag yaml|yml")

‎contrib/kubernetes/controllers.yaml

-26
This file was deleted.

‎contrib/kubernetes/default-node-config.yaml

+51
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,51 @@
1+
allowDisabledDocker: false
2+
apiVersion: v1
3+
authConfig:
4+
authenticationCacheSize: 1000
5+
authenticationCacheTTL: 5m
6+
authorizationCacheSize: 1000
7+
authorizationCacheTTL: 5m
8+
dnsDomain: cluster.local
9+
dnsIP: 0.0.0.0
10+
dnsBindAddress: 0.0.0.0:53
11+
dnsRecursiveResolvConf: ""
12+
dockerConfig:
13+
dockerShimRootDirectory: /var/lib/dockershim
14+
dockerShimSocket: /var/run/kubernetes/dockershim.sock
15+
execHandlerName: native
16+
enableUnidling: true
17+
imageConfig:
18+
format: openshift/origin-${component}:${version}
19+
latest: false
20+
iptablesSyncPeriod: 30s
21+
kind: NodeConfig
22+
kubeletArguments:
23+
cert-dir:
24+
- ./certificates
25+
feature-gates:
26+
- RotateKubeletClientCertificate=true,RotateKubeletServerCertificate=true
27+
masterClientConnectionOverrides:
28+
acceptContentTypes: application/vnd.kubernetes.protobuf,application/json
29+
burst: 40
30+
contentType: application/vnd.kubernetes.protobuf
31+
qps: 20
32+
masterKubeConfig: node.kubeconfig
33+
networkConfig:
34+
mtu: 1450
35+
networkPluginName: redhat/openshift-ovs-multitenant
36+
nodeIP: ""
37+
proxyArguments:
38+
healthz-bind-address:
39+
- 0.0.0.0
40+
healthz-port:
41+
- "10256"
42+
metrics-bind-address:
43+
- 0.0.0.0:10257
44+
servingInfo:
45+
bindAddress: 0.0.0.0:10250
46+
bindNetwork: tcp4
47+
namedCertificates: null
48+
volumeConfig:
49+
localQuota:
50+
perFSGroup: null
51+
volumeDirectory: /var/lib/origin/volumes

‎contrib/kubernetes/static/controllers-pod.yaml

+1-1
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ metadata:
55
spec:
66
containers:
77
- name: controllers
8-
image: openshift/origin:v3.6.0-rc.0
8+
image: openshift/origin:v3.6.0
99
command: ["/usr/bin/openshift", "start", "master", "controllers"]
1010
args:
1111
- "--config=/etc/origin/master/master-config.yaml"

‎contrib/kubernetes/static/network-daemonset.yaml

+157
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,157 @@
1+
kind: DaemonSet
2+
apiVersion: extensions/v1beta1
3+
metadata:
4+
name: sdn
5+
annotations:
6+
kubernetes.io/description: |
7+
This daemon set launches the OpenShift networking components (kube-proxy, DNS, and openshift-sdn).
8+
It expects that OVS is running on the node.
9+
spec:
10+
updateStrategy:
11+
type: RollingUpdate
12+
template:
13+
metadata:
14+
labels:
15+
component: network
16+
type: infra
17+
openshift.io/role: network
18+
annotations:
19+
scheduler.alpha.kubernetes.io/critical-pod: ''
20+
spec:
21+
# Requires fairly broad permissions - ability to read all services and network functions as well
22+
# as all pods.
23+
serviceAccountName: sdn
24+
hostNetwork: true
25+
hostPID: true
26+
containers:
27+
- name: network
28+
image: openshift/node:v3.7.0-alpha.1
29+
command:
30+
- /bin/bash
31+
- -c
32+
- |
33+
#!/bin/sh
34+
set -o errexit
35+
# Take over network functions on the node
36+
rm -Rf /etc/cni/net.d/*
37+
rm -Rf /host/opt/cni/bin/*
38+
cp -Rf /opt/cni/bin/* /host/opt/cni/bin/
39+
# Use whichever node-config exists
40+
cfg=/etc/openshift/node
41+
if [[ ! -f "${cfg}/node-config.yaml" ]]; then
42+
cfg=/etc/origin/node
43+
fi
44+
# Use the same config as the node, but with the service account token
45+
openshift cli config "--config=${cfg}/node.kubeconfig" view --flatten > /tmp/kubeconfig
46+
openshift cli config --config=/tmp/kubeconfig set-credentials sa "--token=$( cat /var/run/secrets/kubernetes.io/serviceaccount/token )"
47+
openshift cli config --config=/tmp/kubeconfig set-context "$( openshift cli config current-context)" --user=sa
48+
# Launch the network process
49+
exec openshift start network "--config=${cfg}/node-config.yaml" --kubeconfig=/tmp/kubeconfig --loglevel=5
50+
51+
securityContext:
52+
runAsUser: 0
53+
# Permission could be reduced by selecting an appropriate SELinux policy
54+
privileged: true
55+
# TODO: debugging only
56+
imagePullPolicy: Never
57+
volumeMounts:
58+
# Directory which contains the host configuration. We look at both locations
59+
# to simplify setup.
60+
- mountPath: /etc/origin/node/
61+
name: host-config
62+
readOnly: true
63+
- mountPath: /etc/openshift/node/
64+
name: host-config-alt
65+
readOnly: true
66+
# Run directories where we need to be able to access sockets
67+
- mountPath: /var/run/dbus/
68+
name: host-var-run-dbus
69+
readOnly: true
70+
- mountPath: /var/run/openvswitch/
71+
name: host-var-run-ovs
72+
readOnly: true
73+
- mountPath: /var/run/kubernetes/
74+
name: host-var-run-kubernetes
75+
readOnly: true
76+
# We mount our socket here
77+
- mountPath: /var/run/openshift-sdn
78+
name: host-var-run-openshift-sdn
79+
# CNI related mounts which we take over
80+
- mountPath: /host/opt/cni/bin
81+
name: host-opt-cni-bin
82+
- mountPath: /etc/cni/net.d
83+
name: host-etc-cni-netd
84+
- mountPath: /var/lib/cni/networks/openshift-sdn
85+
name: host-var-lib-cni-networks-openshift-sdn
86+
87+
resources:
88+
requests:
89+
cpu: 100m
90+
memory: 200Mi
91+
env:
92+
- name: OPENSHIFT_DNS_DOMAIN
93+
value: cluster.local
94+
ports:
95+
- name: healthz
96+
containerPort: 10256
97+
livenessProbe:
98+
initialDelaySeconds: 10
99+
httpGet:
100+
path: /healthz
101+
port: 10256
102+
scheme: HTTP
103+
lifecycle:
104+
# postStart:
105+
# exec:
106+
# command:
107+
# - /usr/bin/dbus-send
108+
# - --system
109+
# - --dest=uk.org.thekelleys.dnsmasq
110+
# - /uk/org/thekelleys/dnsmasq
111+
# - uk.org.thekelleys.SetDomainServers
112+
# - array:string:/in-addr.arpa/127.0.0.1,/$(OPENSHIFT_DNS_DOMAIN)/127.0.0.1
113+
# preStop:
114+
# exec:
115+
# command:
116+
# - /usr/bin/dbus-send
117+
# - --system
118+
# - --dest=uk.org.thekelleys.dnsmasq
119+
# - /uk/org/thekelleys/dnsmasq
120+
# - uk.org.thekelleys.SetDomainServers
121+
# - "array:string:"
122+
123+
volumes:
124+
# In bootstrap mode, the host config contains information not easily available
125+
# from other locations.
126+
- name: host-config
127+
hostPath:
128+
path: /etc/origin/node
129+
- name: host-config-alt
130+
hostPath:
131+
path: /etc/openshift/node
132+
- name: host-modules
133+
hostPath:
134+
path: /lib/modules
135+
136+
- name: host-var-run-ovs
137+
hostPath:
138+
path: /var/run/openvswitch
139+
- name: host-var-run-kubernetes
140+
hostPath:
141+
path: /var/run/kubernetes
142+
- name: host-var-run-dbus
143+
hostPath:
144+
path: /var/run/dbus
145+
- name: host-var-run-openshift-sdn
146+
hostPath:
147+
path: /var/run/openshift-sdn
148+
149+
- name: host-opt-cni-bin
150+
hostPath:
151+
path: /opt/cni/bin
152+
- name: host-etc-cni-netd
153+
hostPath:
154+
path: /etc/cni/net.d
155+
- name: host-var-lib-cni-networks-openshift-sdn
156+
hostPath:
157+
path: /var/lib/cni/networks/openshift-sdn

‎contrib/kubernetes/static/network-ovs.yaml

+61
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
kind: DaemonSet
2+
apiVersion: extensions/v1beta1
3+
metadata:
4+
name: ovs
5+
annotations:
6+
kubernetes.io/description: |
7+
This daemon set launches the openvswitch daemon.
8+
spec:
9+
updateStrategy:
10+
type: RollingUpdate
11+
template:
12+
metadata:
13+
labels:
14+
component: network
15+
type: infra
16+
openshift.io/role: network
17+
annotations:
18+
scheduler.alpha.kubernetes.io/critical-pod: ''
19+
spec:
20+
# Requires fairly broad permissions - ability to read all services and network functions as well
21+
# as all pods.
22+
serviceAccountName: sdn
23+
hostNetwork: true
24+
containers:
25+
- name: openvswitch
26+
image: openshift/openvswitch:v3.7.0-alpha.1
27+
securityContext:
28+
runAsUser: 0
29+
privileged: true
30+
volumeMounts:
31+
- mountPath: /lib/modules
32+
name: host-modules
33+
readOnly: true
34+
- mountPath: /run/openvswitch
35+
name: host-run-ovs
36+
- mountPath: /sys
37+
name: host-sys
38+
readOnly: true
39+
- mountPath: /etc/openvswitch
40+
name: host-config-openvswitch
41+
resources:
42+
requests:
43+
cpu: 100m
44+
memory: 200Mi
45+
limits:
46+
cpu: 200m
47+
memory: 300Mi
48+
49+
volumes:
50+
- name: host-modules
51+
hostPath:
52+
path: /lib/modules
53+
- name: host-run-ovs
54+
hostPath:
55+
path: /run/openvswitch
56+
- name: host-sys
57+
hostPath:
58+
path: /sys
59+
- name: host-config-openvswitch
60+
hostPath:
61+
path: /etc/origin/openvswitch

‎contrib/kubernetes/static/network-policy.yaml

+29
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,29 @@
1+
kind: List
2+
apiVersion: v1
3+
items:
4+
- kind: ServiceAccount
5+
apiVersion: v1
6+
metadata:
7+
name: sdn
8+
namespace: openshift-node
9+
- apiVersion: authorization.openshift.io/v1
10+
kind: ClusterRoleBinding
11+
metadata:
12+
name: sdn-cluster-reader
13+
roleRef:
14+
name: cluster-reader
15+
subjects:
16+
- kind: ServiceAccount
17+
name: sdn
18+
namespace: openshift-node
19+
- apiVersion: authorization.openshift.io/v1
20+
kind: ClusterRoleBinding
21+
metadata:
22+
name: sdn-reader
23+
roleRef:
24+
name: system:sdn-reader
25+
subjects:
26+
- kind: ServiceAccount
27+
name: sdn
28+
namespace: openshift-node
29+
# TODO: PSP binding

0 commit comments

Comments
 (0)