Add snapshot webhook build and deployment. Modify controller to label invalid objects.

Makefile

@@ -12,8 +12,8 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-.PHONY: all snapshot-controller csi-snapshotter clean test
+.PHONY: all snapshot-controller csi-snapshotter validation-webhook clean test
 
-CMDS=snapshot-controller csi-snapshotter
+CMDS=snapshot-controller csi-snapshotter validation-webhook
 all: build
 include release-tools/build.make

README.md

@@ -6,6 +6,8 @@ The CSI snapshotter is part of Kubernetes implementation of [Container Storage I
 
 The volume snapshot feature supports CSI v1.0 and higher. It was introduced as an Alpha feature in Kubernetes v1.12 and has been promoted to an Beta feature in Kubernetes 1.17.
 
+> :warning: **WARNING**: There is a new validating webhook server which provides tightened validation on snapshot objects. This SHOULD be installed by all users of this feature. More details [below](#validating-webhook).
+
 
 ## Overview
 
@@ -79,6 +81,22 @@ Install CSI Driver:
   * kubectl create -f deploy/kubernetes/csi-snapshotter
   * https://github.com/kubernetes-csi/external-snapshotter/tree/master/deploy/kubernetes/csi-snapshotter
 
+### Validating Webhook
+
+The snapshot validating webhook is an HTTP callback which responds to [admission requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). It is part of a larger [plan](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md) to tighten validation for volume snapshot objects. This webhook introduces the [ratcheting validation](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md#backwards-compatibility) mechanism targeting the tighter validation. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs.
+
+> :warning: **WARNING**: Cluster admins choosing not to install the webhook server and participate in the phased release process can cause future problems when upgrading from `v1beta1` to `v1` volumesnapshot API, if there are currently persisted objects which fail the new stricter validation. Potential impacts include being unable to delete invalid snapshot objects.
+
+Read more about how to install the example webhook [here](deploy/kubernetes/webhook-example/README.md).
+
+####  Validating Webhook Command Line Options
+
+* `--tls-cert-file`: File containing the x509 Certificate for HTTPS. (CA cert, if any, concatenated after server cert). Required.
+
+* `--tls-private-key-file`: File containing the x509 private key matching --tls-cert-file. Required.
+
+* `--port`: Secure port that the webhook listens on (default 443)
+
 ### Snapshot controller command line options
 
 #### Important optional arguments that are highly recommended to be used

cmd/validation-webhook/Dockerfile

@@ -0,0 +1,7 @@
+FROM gcr.io/distroless/base:latest
+LABEL maintainers="Kubernetes Authors"
+LABEL description="Snapshot Validation Webhook"
+ARG binary=./bin/validation-webhook
+
+COPY ${binary} validation-webhook
+ENTRYPOINT ["/validation-webhook"]

cmd/validation-webhook/main.go

@@ -0,0 +1,33 @@
+/*
+Copyright 2020 The Kubernetes Authors.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+*/
+
+package main
+
+import (
+	"flag"
+
+	webhook "github.com/kubernetes-csi/external-snapshotter/v2/pkg/validation-webhook"
+	"k8s.io/klog"
+)
+
+func main() {
+	rootCmd := webhook.CmdWebhook
+
+	loggingFlags := &flag.FlagSet{}
+	klog.InitFlags(loggingFlags)
+	rootCmd.PersistentFlags().AddGoFlagSet(loggingFlags)
+	rootCmd.Execute()
+}

deploy/kubernetes/webhook-example/README.md

@@ -0,0 +1,81 @@
+# Validating Webhook
+
+The snapshot validating webhook is an HTTP callback which responds to [admission requests](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/). It is part of a larger [plan](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md) to tighten validation for volume snapshot objects. This webhook introduces the [ratcheting validation](https://github.com/kubernetes/enhancements/blob/master/keps/sig-storage/177-volume-snapshot/tighten-validation-webhook-crd.md#backwards-compatibility) mechanism targeting the tighter validation. The cluster admin or Kubernetes distribution admin should install the webhook alongside the snapshot controllers and CRDs.
+
+> :warning: **WARNING**: Cluster admins choosing not to install the webhook server and participate in the phased release process can cause future problems when upgrading from `v1beta1` to `v1` volumesnapshot API, if there are currently persisted objects which fail the new stricter validation. Potential impacts include being unable to delete invalid snapshot objects.
+
+## Prerequisites
+
+The following are prerequisites to use this validating webhook:
+
+- K8s version 1.17+ (v1.9+ to use `admissionregistration.k8s.io/v1beta1`, v1.16+ to use `admissionregistration.k8s.io/v1`, v1.17+ to use  `snapshot.storage.k8s.io/v1beta1`)
+- ValidatingAdmissionWebhook is [enabled](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/#prerequisites). (in v1.18+ it will be enabled by default)
+- API `admissionregistration.k8s.io/v1beta1` or `admissionregistration.k8s.io/v1` is enabled.
+
+## How to build the webhook
+
+Build the binary
+
+```bash
+make 
+```
+
+Build the docker image
+
+```bash
+docker build -t validation-webhook:latest -f ./cmd/validation-webhook/Dockerfile .
+```
+
+## How to deploy the webhook
+
+The webhook server is provided as an image which can be built from this repository. It can be deployed anywhere, as long as the api server is able to reach it over HTTPS. It is recommended to deploy the webhook server in the cluster as snapshotting is latency sensitive. A `ValidatingWebhookConfiguration` object is needed to configure the api server to contact the webhook server. Please see the [documentation](https://kubernetes.io/docs/reference/access-authn-authz/extensible-admission-controllers/) for more details. The webhook server code is adapted from the [webhook server](https://github.com/kubernetes/kubernetes/tree/v1.18.6/test/images/agnhost/webhook) used in the kubernetes/kubernetes end to end testing code.
+
+### Example in-cluster deployment using Kubernetes Secrets
+
+Please note this is not considered to be a production ready method to deploy the certificates and is only provided for demo purposes. This is only one of many ways to deploy the certificates, it is your responsibility to ensure the security of your cluster. TLS certificates and private keys should be handled with care and you may not want to keep them in plain Kubernetes secrets.
+
+This method was heavily adapted from [banzai cloud](https://banzaicloud.com/blog/k8s-admission-webhooks/).
+
+#### Method
+
+These commands should be run from the top level directory.
+
+1. Run the `create-cert.sh` script. Note using the default namespace will allow anyone with access to that namespace to read your secret. It is recommended to change the namespace in all the files and the commands given below.
+
+
+    ```bash
+    # This script will create a TLS certificate signed by the [cluster](https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/). It will place the public and private key into a secret on the cluster.
+    ./deploy/kubernetes/webhook-example/create-cert.sh --service snapshot-validation-service --secret snapshot-validation-secret --namespace default # Make sure to use a different namespace
+    ```
+
+2. Patch the `ValidatingWebhookConfiguration` file from the template, filling in the CA bundle field.
+
+    ```bash
+    cat ./deploy/kubernetes/webhook-example/admission-configuration-template | ./deploy/kubernetes/webhook-example/patch-ca-bundle.sh > ./deploy/kubernetes/webhook-example/admission-configuration.yaml
+    ```
+
+3. Change the namespace in the generated `admission-configuration.yaml` file. Change the namespace in the service and deployment in the `webhook.yaml` file.
+
+4. Create the deployment, service and admission configuration objects on the cluster.
+
+    ```bash
+    kubectl apply -f ./deploy/kubernetes/webhook-example
+    ```
+
+Once all the pods from the deployment are up and running, you should be ready to go.
+
+#### Verify the webhook works
+
+Try to create an invalid snapshot object, the snapshot creation should fail.
+
+```bash
+kubectl create -f ./examples/kubernetes/invalid-snapshot.yaml
+```
+
+### Other methods to deploy the webhook server
+
+Look into [cert-manager](https://cert-manager.io/) to handle the certificates, and this kube-builder [tutorial](https://book.kubebuilder.io/cronjob-tutorial/cert-manager.html) on how to deploy a webhook.
+
+#### Important
+
+Please see the deployment [yaml](./webhook.yaml) for the arguments expected by the webhook server. The snapshot validation webhook is served at the path `/snapshot`.

deploy/kubernetes/webhook-example/admission-configuration-template

@@ -0,0 +1,23 @@
+apiVersion: admissionregistration.k8s.io/v1
+kind: ValidatingWebhookConfiguration
+metadata:
+  name: "validation-webhook.snapshot.storage.k8s.io"
+  namespace: "default"
+webhooks:
+- name: "validation-webhook.snapshot.storage.k8s.io"
+  rules:
+  - apiGroups:   ["snapshot.storage.k8s.io"]
+    apiVersions: ["v1beta1"]
+    operations:  ["CREATE", "UPDATE"]
+    resources:   ["volumesnapshots", "volumesnapshotcontents"]
+    scope:       "*"
+  clientConfig:
+    service:
+      namespace: "default"
+      name: "snapshot-validation-service"
+      path: "/volumesnapshot"
+    caBundle: ${CA_BUNDLE}
+  admissionReviewVersions: ["v1", "v1beta1"]
+  sideEffects: None
+  failurePolicy: Ignore # We recommend switching to Fail only after successful installation of the webhook server and webhook.
+  timeoutSeconds: 2 # This will affect the latency and performance. Finetune this value based on your application's tolerance.

deploy/kubernetes/webhook-example/create-cert.sh

@@ -0,0 +1,126 @@
+#!/bin/bash
+# File originally from https://github.com/banzaicloud/admission-webhook-example/blob/blog/deployment/webhook-create-signed-cert.sh
+
+set -e
+
+usage() {
+    cat <<EOF
+Generate certificate suitable for use with a webhook service.
+This script uses k8s' CertificateSigningRequest API to a generate a
+certificate signed by k8s CA suitable for use with webhook
+services. This requires permissions to create and approve CSR. See
+https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster for
+detailed explantion and additional instructions.
+The server key/cert k8s CA cert are stored in a k8s secret.
+usage: ${0} [OPTIONS]
+The following flags are required.
+       --service          Service name of webhook.
+       --namespace        Namespace where webhook service and secret reside.
+       --secret           Secret name for CA certificate and server certificate/key pair.
+EOF
+    exit 1
+}
+
+while [[ $# -gt 0 ]]; do
+    case ${1} in
+        --service)
+            service="$2"
+            shift
+            ;;
+        --secret)
+            secret="$2"
+            shift
+            ;;
+        --namespace)
+            namespace="$2"
+            shift
+            ;;
+        *)
+            usage
+            ;;
+    esac
+    shift
+done
+
+[ -z ${service} ] && service=admission-webhook-example-svc
+[ -z ${secret} ] && secret=admission-webhook-example-certs
+[ -z ${namespace} ] && namespace=default
+
+if [ ! -x "$(command -v openssl)" ]; then
+    echo "openssl not found"
+    exit 1
+fi
+
+csrName=${service}.${namespace}
+tmpdir=$(mktemp -d)
+echo "creating certs in tmpdir ${tmpdir} "
+
+cat <<EOF >> ${tmpdir}/csr.conf
+[req]
+req_extensions = v3_req
+distinguished_name = req_distinguished_name
+[req_distinguished_name]
+[ v3_req ]
+basicConstraints = CA:FALSE
+keyUsage = nonRepudiation, digitalSignature, keyEncipherment
+extendedKeyUsage = serverAuth
+subjectAltName = @alt_names
+[alt_names]
+DNS.1 = ${service}
+DNS.2 = ${service}.${namespace}
+DNS.3 = ${service}.${namespace}.svc
+EOF
+
+openssl genrsa -out ${tmpdir}/server-key.pem 2048
+openssl req -new -key ${tmpdir}/server-key.pem -subj "/CN=${service}.${namespace}.svc" -out ${tmpdir}/server.csr -config ${tmpdir}/csr.conf
+
+# clean-up any previously created CSR for our service. Ignore errors if not present.
+kubectl delete csr ${csrName} 2>/dev/null || true
+
+# create  server cert/key CSR and  send to k8s API
+cat <<EOF | kubectl create -f -
+apiVersion: certificates.k8s.io/v1beta1
+kind: CertificateSigningRequest
+metadata:
+  name: ${csrName}
+spec:
+  groups:
+  - system:authenticated
+  request: $(cat ${tmpdir}/server.csr | base64 | tr -d '\n')
+  usages:
+  - digital signature
+  - key encipherment
+  - server auth
+EOF
+
+# verify CSR has been created
+while true; do
+    kubectl get csr ${csrName}
+    if [ "$?" -eq 0 ]; then
+        break
+    fi
+done
+
+# approve and fetch the signed certificate
+kubectl certificate approve ${csrName}
+# verify certificate has been signed
+for x in $(seq 10); do
+    serverCert=$(kubectl get csr ${csrName} -o jsonpath='{.status.certificate}')
+    if [[ ${serverCert} != '' ]]; then
+        break
+    fi
+    sleep 1
+done
+if [[ ${serverCert} == '' ]]; then
+    echo "ERROR: After approving csr ${csrName}, the signed certificate did not appear on the resource. Giving up after 10 attempts." >&2
+    exit 1
+fi
+echo ${serverCert} | openssl base64 -d -A -out ${tmpdir}/server-cert.pem
+
+
+# create the secret with CA cert and server cert/key
+kubectl create secret generic ${secret} \
+        --from-file=key.pem=${tmpdir}/server-key.pem \
+        --from-file=cert.pem=${tmpdir}/server-cert.pem \
+        --dry-run -o yaml |
+    kubectl -n ${namespace} apply -f -

deploy/kubernetes/webhook-example/patch-ca-bundle.sh

@@ -0,0 +1,16 @@
+#!/bin/bash
+# File originally from https://github.com/banzaicloud/admission-webhook-example/blob/blog/deployment/webhook-patch-ca-bundle.sh
+
+ROOT=$(cd $(dirname $0)/../../; pwd)
+
+set -o errexit
+set -o nounset
+set -o pipefail
+
+export CA_BUNDLE=$(kubectl config view --raw -o json | jq -r '.clusters[0].cluster."certificate-authority-data"' | tr -d '"')
+
+if command -v envsubst >/dev/null 2>&1; then
+    envsubst
+else
+    sed -e "s|\${CA_BUNDLE}|${CA_BUNDLE}|g"
+fi

deploy/kubernetes/webhook-example/webhook.yaml

@@ -0,0 +1,44 @@
+apiVersion: apps/v1
+kind: Deployment
+metadata:
+  name: snapshot-validation-deployment
+  namespace: default # NOTE: change the namespace
+  labels:
+    app: snapshot-validation
+spec:
+  replicas: 3
+  selector:
+    matchLabels:
+      app: snapshot-validation
+  template:
+    metadata:
+      labels:
+        app: snapshot-validation
+    spec:
+      containers:
+      - name: snapshot-validation
+        image: k8s.gcr.io/sig-storage/validation-webhook:v2.2.0 # change the image if you wish to use your own custom validation server image
+        args: ['--tls-cert-file=/etc/snapshot-validation-webhook/certs/cert.pem', '--tls-private-key-file=/etc/snapshot-validation-webhook/certs/key.pem']
+        ports:
+        - containerPort: 443 # change the port as needed
+        volumeMounts:
+          - name: snapshot-validation-webhook-certs
+            mountPath: /etc/snapshot-validation-webhook/certs
+            readOnly: true
+      volumes:
+        - name: snapshot-validation-webhook-certs
+          secret:
+            secretName: snapshot-validation-secret
+---
+apiVersion: v1
+kind: Service
+metadata:
+  name: snapshot-validation-service
+  namespace: default # NOTE: change the namespace
+spec:
+  selector:
+    app: snapshot-validation
+  ports:
+    - protocol: TCP
+      port: 443 # Change if needed
+      targetPort: 443 # Change if the webserver image expects a different port

examples/kubernetes/invalid-snapshot.yaml

@@ -0,0 +1,9 @@
+apiVersion: snapshot.storage.k8s.io/v1beta1
+kind: VolumeSnapshot
+metadata:
+  name: new-snapshot-demo
+spec:
+  volumeSnapshotClassName: csi-hostpath-snapclass
+  source: # Only one of the two fields should be set for a snapshot. Therefore this snapshot is invalid.
+    persistentVolumeClaimName: pvc 
+    volumeSnapshotContentName: vsc