Skip to content

Commit 4f1e8c8

Browse files
author
Andi Li
committed
Add boiler plate for webhook build and deployment. Snapshot endpoint coming soon.
1 parent d24d2b9 commit 4f1e8c8

16 files changed

+1300
-2
lines changed

Makefile

+2-2
Original file line numberDiff line numberDiff line change
@@ -12,8 +12,8 @@
1212
# See the License for the specific language governing permissions and
1313
# limitations under the License.
1414

15-
.PHONY: all snapshot-controller csi-snapshotter clean test
15+
.PHONY: all snapshot-controller csi-snapshotter webhook clean test
1616

17-
CMDS=snapshot-controller csi-snapshotter
17+
CMDS=snapshot-controller csi-snapshotter webhook
1818
all: build
1919
include release-tools/build.make

cmd/webhook/Dockerfile

+7
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,7 @@
1+
FROM gcr.io/distroless/base:latest
2+
LABEL maintainers="Kubernetes Authors"
3+
LABEL description="Snapshot Webhook"
4+
ARG binary=./bin/webhook
5+
6+
COPY ${binary} webhook
7+
ENTRYPOINT ["/webhook"]

cmd/webhook/main.go

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
package main
2+
3+
import (
4+
"flag"
5+
6+
webhook "github.com/kubernetes-csi/external-snapshotter/v2/pkg/webhook"
7+
"k8s.io/klog"
8+
)
9+
10+
func main() {
11+
rootCmd := webhook.CmdWebhook
12+
13+
loggingFlags := &flag.FlagSet{}
14+
klog.InitFlags(loggingFlags)
15+
rootCmd.PersistentFlags().AddGoFlagSet(loggingFlags)
16+
rootCmd.Execute()
17+
}
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
# How to deploy the webhook
2+
3+
The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.
4+
5+
## Example in-cluster deployment using Kubernetes Secrets
6+
7+
Please note this is not considered to be a production ready method to deploy the certificates and is only provided for demo purposes. This is only one of many ways to deploy the certificates, it is your responsibility to ensure the security of your cluster. TLS certificates and private keys should be handled with care and you may not want to keep them in plain Kubernetes secrets. These commands should be run from the top level directory.
8+
9+
This method was heavily adapted from [banzai cloud](https://banzaicloud.com/blog/k8s-admission-webhooks/).
10+
11+
### Method
12+
13+
Run the `create-cert.sh` script. Note using the default namespace will allow anyone with access to that namespace to read your secret. It is recommended to change the namespace in all the file and the command.
14+
15+
This script will create a TLS certificate signed by the [cluster](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/). It will place the public and private key into a secret on the cluster.
16+
17+
```bash
18+
./deploy/kubernetes/webhook-example/create-cert.sh --service snapshot-validation-service --namespace default --secret snapshot-validation-secret
19+
```
20+
21+
Patch the `ValidatingWebhookConfiguration` file from the template, filling in the CA bundle field.
22+
s
23+
```bash
24+
cat ./deploy/kubernetes/webhook-example/admission-configuration-template | ./deploy/kubernetes/webhook-example/patch-ca-bundle.sh > ./deploy/kubernetes/webhook-example/admission-configuration.yaml
25+
```
26+
27+
Create the deployment, service and admission configuration objects on the cluster.
28+
29+
```bash
30+
kubectl apply -f ./deploy/kubernetes/webhook-example
31+
```
32+
33+
### Verify the webhook works
34+
35+
Try to create a snapshot object. Since the configuration is set to always deny, the snapshot creation should fail.
36+
37+
```bash
38+
kubectl create -f ./examples/kubernetes/snapshot.yaml
39+
```
40+
41+
## Other methods to deply the webhook server
42+
43+
Look into [cert-manager](https://cert-manager.io/) to handle the certificates, and this kube-builder [tutorial](https://book.kubebuilder.io/cronjob-tutorial/cert-manager.html) on how to deploy a webhook.
44+
45+
### Important
46+
47+
Please see the deployment [yaml](./webhook.yaml) for the arguments expected by the webhook server. The snapshot validation webhook is served at the path `/snapshot`.
48+
49+
## How to build the webhook
50+
51+
Build the binary
52+
53+
```bash make ```
54+
55+
Build the docker image
56+
57+
```bash docker build -t gcr.io/your-project-name/webhook:latest -f ./cmd/webhook/Dockerfile .```
58+
59+
Push the docker image to the remote registry
60+
61+
```bash docker push gcr.io/your-project-name/webhook:latest```
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
apiVersion: admissionregistration.k8s.io/v1
2+
kind: ValidatingWebhookConfiguration
3+
metadata:
4+
name: "webhook-validation.storage.sigs.k8s.io"
5+
webhooks:
6+
- name: "snapshot.webhook-validation.storage.sigs.k8s.io"
7+
rules:
8+
- apiGroups: ["snapshot.storage.k8s.io"]
9+
apiVersions: ["v1beta1"]
10+
operations: ["CREATE", "UPDATE"]
11+
resources: ["volumesnapshots", "volumesnapshotcontents"]
12+
scope: "*"
13+
clientConfig:
14+
service:
15+
namespace: "default"
16+
name: "snapshot-validation-service"
17+
path: "/snapshot"
18+
caBundle: ${CA_BUNDLE}
19+
admissionReviewVersions: ["v1", "v1beta1"]
20+
sideEffects: None
21+
failurePolicy: Fail # We recommend switching to Fail only after successful installation of the server and webhook.
22+
timeoutSeconds: 10 # This will affect the latency and performance. Finetune this value based on your application's tolerance.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,125 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
usage() {
6+
cat <<EOF
7+
Generate certificate suitable for use with an sidecar-injector webhook service.
8+
This script uses k8s' CertificateSigningRequest API to a generate a
9+
certificate signed by k8s CA suitable for use with sidecar-injector webhook
10+
services. This requires permissions to create and approve CSR. See
11+
https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
12+
detailed explantion and additional instructions.
13+
The server key/cert k8s CA cert are stored in a k8s secret.
14+
usage: ${0} [OPTIONS]
15+
The following flags are required.
16+
--service Service name of webhook.
17+
--namespace Namespace where webhook service and secret reside.
18+
--secret Secret name for CA certificate and server certificate/key pair.
19+
EOF
20+
exit 1
21+
}
22+
23+
while [[ $# -gt 0 ]]; do
24+
case ${1} in
25+
--service)
26+
service="$2"
27+
shift
28+
;;
29+
--secret)
30+
secret="$2"
31+
shift
32+
;;
33+
--namespace)
34+
namespace="$2"
35+
shift
36+
;;
37+
*)
38+
usage
39+
;;
40+
esac
41+
shift
42+
done
43+
44+
[ -z ${service} ] && service=admission-webhook-example-svc
45+
[ -z ${secret} ] && secret=admission-webhook-example-certs
46+
[ -z ${namespace} ] && namespace=default
47+
48+
if [ ! -x "$(command -v openssl)" ]; then
49+
echo "openssl not found"
50+
exit 1
51+
fi
52+
53+
csrName=${service}.${namespace}
54+
tmpdir=$(mktemp -d)
55+
echo "creating certs in tmpdir ${tmpdir} "
56+
57+
cat <<EOF >> ${tmpdir}/csr.conf
58+
[req]
59+
req_extensions = v3_req
60+
distinguished_name = req_distinguished_name
61+
[req_distinguished_name]
62+
[ v3_req ]
63+
basicConstraints = CA:FALSE
64+
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
65+
extendedKeyUsage = serverAuth
66+
subjectAltName = @alt_names
67+
[alt_names]
68+
DNS.1 = ${service}
69+
DNS.2 = ${service}.${namespace}
70+
DNS.3 = ${service}.${namespace}.svc
71+
EOF
72+
73+
openssl genrsa -out ${tmpdir}/server-key.pem 2048
74+
openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
75+
76+
# clean-up any previously created CSR for our service. Ignore errors if not present.
77+
kubectl delete csr ${csrName} 2>/dev/null || true
78+
79+
# create server cert/key CSR and send to k8s API
80+
cat <<EOF | kubectl create -f -
81+
apiVersion: certificates.k8s.io/v1beta1
82+
kind: CertificateSigningRequest
83+
metadata:
84+
name: ${csrName}
85+
spec:
86+
groups:
87+
- system:authenticated
88+
request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
89+
usages:
90+
- digital signature
91+
- key encipherment
92+
- server auth
93+
EOF
94+
95+
# verify CSR has been created
96+
while true; do
97+
kubectl get csr ${csrName}
98+
if [ "$?" -eq 0 ]; then
99+
break
100+
fi
101+
done
102+
103+
# approve and fetch the signed certificate
104+
kubectl certificate approve ${csrName}
105+
# verify certificate has been signed
106+
for x in $(seq 10); do
107+
serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
108+
if [[ ${serverCert} != '' ]]; then
109+
break
110+
fi
111+
sleep 1
112+
done
113+
if [[ ${serverCert} == '' ]]; then
114+
echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
115+
exit 1
116+
fi
117+
echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
118+
119+
120+
# create the secret with CA cert and server cert/key
121+
kubectl create secret generic ${secret} \
122+
--from-file=key.pem=${tmpdir}/server-key.pem \
123+
--from-file=cert.pem=${tmpdir}/server-cert.pem \
124+
--dry-run -o yaml |
125+
kubectl -n ${namespace} apply -f -
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
#!/bin/bash
2+
3+
ROOT=$(cd $(dirname $0)/../../; pwd)
4+
5+
set -o errexit
6+
set -o nounset
7+
set -o pipefail
8+
9+
export CA_BUNDLE=$(kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"')
10+
11+
if command -v envsubst >/dev/null 2>&1; then
12+
envsubst
13+
else
14+
sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
15+
fi
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,44 @@
1+
apiVersion: apps/v1
2+
kind: Deployment
3+
metadata:
4+
name: snapshot-validation-deployment
5+
namespace: default # TODO: change the namespace
6+
labels:
7+
app: snapshot-validation
8+
spec:
9+
replicas: 3
10+
selector:
11+
matchLabels:
12+
app: snapshot-validation
13+
template:
14+
metadata:
15+
labels:
16+
app: snapshot-validation
17+
spec:
18+
containers:
19+
- name: snapshot-validation
20+
image: gcr.io/your-project-name/webhook:latest # change the image if you wish to use your own custom validation server image
21+
args: ['--tls-cert-file=/etc/webhook/certs/cert.pem', '--tls-private-key-file=/etc/webhook/certs/key.pem']
22+
ports:
23+
- containerPort: 443 # change the port as needed
24+
volumeMounts:
25+
- name: webhook-certs
26+
mountPath: /etc/webhook/certs
27+
readOnly: true
28+
volumes:
29+
- name: webhook-certs
30+
secret:
31+
secretName: snapshot-validation-secret
32+
---
33+
apiVersion: v1
34+
kind: Service
35+
metadata:
36+
name: snapshot-validation-service
37+
namespace: default # TODO: change the namespace
38+
spec:
39+
selector:
40+
app: snapshot-validation
41+
ports:
42+
- protocol: TCP
43+
port: 443 # Change if needed
44+
targetPort: 443 # Change if the webserver image expects a different port

go.mod

+6
Original file line numberDiff line numberDiff line change
@@ -4,21 +4,27 @@ go 1.12
44

55
require (
66
github.com/container-storage-interface/spec v1.2.0
7+
github.com/evanphx/json-patch v4.5.0+incompatible
78
github.com/golang/mock v1.4.3
89
github.com/golang/protobuf v1.4.2
10+
github.com/google/gofuzz v1.1.0
911
github.com/imdario/mergo v0.3.9 // indirect
1012
github.com/kubernetes-csi/csi-lib-utils v0.7.0
1113
github.com/kubernetes-csi/csi-test v2.0.0+incompatible
1214
github.com/prometheus/client_golang v1.7.1
1315
github.com/prometheus/client_model v0.2.0
1416
github.com/prometheus/common v0.10.0
17+
github.com/spf13/cobra v1.0.0
1518
google.golang.org/grpc v1.28.0
1619
k8s.io/api v0.19.0-rc.2
20+
k8s.io/apiextensions-apiserver v0.0.0
1721
k8s.io/apimachinery v0.19.0-rc.2
1822
k8s.io/client-go v0.19.0-rc.2
1923
k8s.io/code-generator v0.19.0-rc.2
2024
k8s.io/component-base v0.19.0-rc.2
25+
k8s.io/klog v1.0.0
2126
k8s.io/klog/v2 v2.2.0
27+
k8s.io/kubernetes v1.18.6
2228
)
2329

2430
replace (

0 commit comments

Comments
 (0)