| title | excerpt | updated |
|---|---|---|
Resizing Persistent Volumes |
Find out how to resize Persistent Volumes on OVHcloud Managed Kubernetes |
2021-10-19 |
In this tutorial we are going to guide you with the resize of Persistent Volumes (PVs) on your OVHcloud Managed Kubernetes Service.
The Kubernetes PersistentVolume subsystem provides an API for users and administrators that abstracts details of how storage is provided from how it is consumed. To do this Kubernetes provides two API resources: PersistentVolume (PVs) and PersistentVolumeClaim (PVCs).
Since Kubernetes 1.11, support for expanding PersistentVolumeClaims (PVCs) is enabled by default, and in this tutorial you will learn how to do it.
Warning
Kubernetes PVCs resizing only allows to expand volumes, not to decrease them.
This tutorial presupposes that you already have a working OVHcloud Managed Kubernetes cluster, and some basic knowledge of how to operate it. If you want to know more on those topics, please look at the deploying a Hello World application documentation.
You also need to know how PVs are handled on OVHcloud Managed Kubernetes service, please refer to the Persistent Volumes on OVHcloud Managed Kubernetes guide.
Warning
When a Persistent Volumes resource is created inside a Managed Kubernetes cluster, an associated Public Cloud Block Storage volume is automatically created with it. This volume is hourly charged and will appear in your Public Cloud project. For more information, please refer to the following documentation: Volume Block Storage price
To test the PVs resizing, we will need a PV associated to the cluster, i.e. we need to deploy a service making a PVC. To keep thing simple, we choose to deploy a single instance of MySQL.
Let's begin by creating a mysql-pvc.yaml to define an initial PVC with 2 GB of allocated space:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: mysql-pv-claim
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 2GiAnd apply it to the cluster:
kubectl apply -f mysql/mysql-pvc.yamlWe can then verify that the PVC is correctly created and bound to a PV:
kubectl describe pvc mysql-pv-claimNow we create a mysql-deployment.yaml file to deploy the MySQL using that PVC:
apiVersion: v1
kind: Service
metadata:
name: mysql
spec:
ports:
- port: 3306
selector:
app: mysql
clusterIP: None
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mysql
spec:
replicas: 1
selector:
matchLabels:
app: mysql
strategy:
type: Recreate
template:
metadata:
labels:
app: mysql
spec:
containers:
- image: mysql:5.6
name: mysql
env:
# Use secret in real usage
- name: MYSQL_ROOT_PASSWORD
value: password
ports:
- containerPort: 3306
name: mysql
volumeMounts:
- name: mysql-persistent-storage
mountPath: /var/lib/mysql
volumes:
- name: mysql-persistent-storage
persistentVolumeClaim:
claimName: mysql-pv-claimAnd we deploy and verify it:
kubectl apply -f mysql-deployment.yaml
kubectl describe deployment mysqlIn my example cluster, the precedent commands obtains:
$ kubectl apply -f mysql/mysql-pvc.yaml
persistentvolumeclaim/mysql-pv-claim created
$ kubectl describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass: csi-cinder-high-speed
Status: Bound
Volume: ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: mysql-c85f7f79c-wz4w7
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 72s (x2 over 72s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
Normal Provisioning 72s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b External provisioner is provisioning volume for claim "default/mysql-pv-claim"
Normal ProvisioningSucceeded 70s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
$ kubectl apply -f mysql/mysql-deployment.yaml
service/mysql created
deployment.apps/mysql created
$ kubectl describe deployment mysql
Name: mysql
Namespace: default
CreationTimestamp: Mon, 06 Jan 2020 11:45:03 +0100
Labels: <none>
Annotations: deployment.kubernetes.io/revision: 1
kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"apps/v1","kind":"Deployment","metadata":{"annotations":{},"name":"mysql","namespace":"default"},"spec":{"replicas":1,"selec...
Selector: app=mysql
Replicas: 1 desired | 1 updated | 1 total | 1 available | 0 unavailable
StrategyType: Recreate
MinReadySeconds: 0
Pod Template:
Labels: app=mysql
Containers:
mysql:
Image: mysql:5.6
Port: 3306/TCP
Host Port: 0/TCP
Environment:
MYSQL_ROOT_PASSWORD: password
Mounts:
/var/lib/mysql from mysql-persistent-storage (rw)
Volumes:
mysql-persistent-storage:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: mysql-pv-claim
ReadOnly: false
Conditions:
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: mysql-c85f7f79c (1/1 replicas created)
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 27s deployment-controller Scaled up replica set mysql-c85f7f79c to 1The preceding YAML file creates a service that allows other Pods in the cluster to access the database. The Service option clusterIP: None lets the Service DNS name resolve directly to the Pod’s IP address. This is optimal when you have only one Pod behind a Service and you don’t intend to increase the number of Pods.
Now we are going to use kubectl to create a mysql client on the fly to connect to the database:
kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppasswordAnd then simply create a database with a table:
CREATE DATABASE testingResize;
USE testingResize;
CREATE TABLE anEmptyTable (k VARCHAR(256), v TEXT);
SHOW TABLES;On my example cluster:
$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
If you don't see a command prompt, try pressing enter.
mysql> CREATE DATABASE testingResize;
Query OK, 1 row affected (0.01 sec)
mysql> USE testingResize;
Database changed
mysql> CREATE TABLE anEmptyTable (k VARCHAR(256), v TEXT);
Query OK, 0 rows affected (0.04 sec)
mysql> SHOW TABLES;
+-------------------------+
| Tables_in_testingResize |
+-------------------------+
| anEmptyTable |
+-------------------------+
1 row in set (0.00 sec)In order to expand the persistent volume, the first step is to unbound the PVC from the deployments using them.
To do that, we set the deployment's replicas to 0:
Warning
Do not forget to downscale your deployment before to resize your volume
kubectl patch deployment mysql -p '{ "spec": { "replicas": 0 }}'Then we path the PVC definition to expand the volume to 6 GB:
kubectl patch pvc mysql-pv-claim -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}'Warning
Kubernetes PVCs resizing only allows to expand volumes, not to decrease them. If you try to decrease the storage size, you will get a message like
The PersistentVolumeClaim "mysql-pv-claim" is invalid: spec.resources.requests.storage: Forbidden: field can not be less than previous value
We verify that the volume has been expanded:
kubectl describe pvc mysql-pv-claimIn the "conditions" field shown by the output of the previous command line, we can see that the PVC is waiting for user to start a pod to finish file system resize of the volume.
Let's put replicas back to 1 on mysql-deployment.yaml, and deploy it again to start a pod:
kubectl patch deployment mysql -p '{ "spec": { "replicas": 1 }}'After the pod starts, we can use again kubectl describe pvc mysql-pv-claim and we see that the PV size is 6 GB.
On my example cluster:
$ kubectl patch deployment mysql -p '{ "spec": { "replicas": 0 }}'
deployment.extensions/mysql patched
$ kubectl get deployment mysql
NAME READY UP-TO-DATE AVAILABLE AGE
mysql 0/0 0 0 4h0m
$ kubectl patch pvc mysql-pv-claim -p '{ "spec": { "resources": { "requests": { "storage": "6Gi" }}}}'
persistentvolumeclaim/mysql-pv-claim patched
$ kubectl describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass: csi-cinder-high-speed
Status: Bound
Volume: ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 2Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: <none>
Conditions:
Type Status LastProbeTime LastTransitionTime Reason Message
---- ------ ----------------- ------------------ ------ -------
FileSystemResizePending True Mon, 01 Jan 0001 00:00:00 +0000 Mon, 06 Jan 2020 11:47:22 +0100 Waiting for user to (re-)start a pod to finish file system resize of volume on node.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 2m27s (x2 over 2m27s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
Normal Provisioning 2m27s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b External provisioner is provisioning volume for claim "default/mysql-pv-claim"
Normal ProvisioningSucceeded 2m25s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Normal Resizing 9s (x9 over 13s) external-resizer cinder.csi.openstack.org External resizer is resizing volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Normal FileSystemResizeRequired 8s external-resizer cinder.csi.openstack.org Require file system resize of volume on node
$ kubectl patch deployment mysql -p '{ "spec": { "replicas": 1 }}'
deployment.extensions/mysql patched
$ kubectl get deployment mysql
NAME READY UP-TO-DATE AVAILABLE AGE
mysql 0/1 1 0 4h2m
$ kubectl get deployment mysql
NAME READY UP-TO-DATE AVAILABLE AGE
mysql 1/1 1 1 4h4m
$ kubectl describe pvc mysql-pv-claim
Name: mysql-pv-claim
Namespace: default
StorageClass: csi-cinder-high-speed
Status: Bound
Volume: ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"v1","kind":"PersistentVolumeClaim","metadata":{"annotations":{},"name":"mysql-pv-claim","namespace":"default"},"spec":{"acc...
pv.kubernetes.io/bind-completed: yes
pv.kubernetes.io/bound-by-controller: yes
volume.beta.kubernetes.io/storage-provisioner: cinder.csi.openstack.org
Finalizers: [kubernetes.io/pvc-protection]
Capacity: 6Gi
Access Modes: RWO
VolumeMode: Filesystem
Mounted By: mysql-c85f7f79c-9nn8q
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ExternalProvisioning 8m8s (x2 over 8m8s) persistentvolume-controller waiting for a volume to be created, either by external provisioner "cinder.csi.openstack.org" or manually created by system administrator
Normal Provisioning 8m8s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b External provisioner is provisioning volume for claim "default/mysql-pv-claim"
Normal ProvisioningSucceeded 8m6s cinder.csi.openstack.org_csi-cinder-controllerplugin-0_4da74c15-1973-486d-9dde-2ccf2f19811b Successfully provisioned volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Normal Resizing 5m50s (x9 over 5m54s) external-resizer cinder.csi.openstack.org External resizer is resizing volume ovh-managed-kubernetes-btw8lc-pvc-ab896768-b995-453b-85ab-4bcb378de01d
Normal FileSystemResizeRequired 5m49s external-resizer cinder.csi.openstack.org Require file system resize of volume on nodeSo we launch again a MySQL client to verify that we can still read our database:
kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppasswordAn SHOW DATABASES; should allow us to see our testingResize database, we can select it and find our anEmptyTable table.
On my example cluster:
$ kubectl run -it --rm --image=mysql:5.6 --restart=Never mysql-client -- mysql -h mysql -ppassword
If you don't see a command prompt, try pressing enter.
mysql> SHOW DATABASES;
+---------------------+
| Database |
+---------------------+
| information_schema |
| #mysql50#lost+found |
| mysql |
| performance_schema |
| testingResize |
+---------------------+
5 rows in set (0.01 sec)
mysql> USE testingResize;
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
mysql> SHOW TABLES;
+-------------------------+
| Tables_in_testingResize |
+-------------------------+
| anEmptyTable |
+-------------------------+
1 row in set (0.00 sec)Now you can expand the Persistent Volumes on your OVHcloud Managed Kubernetes cluster, and adapt them to the live of your data.
To learn more about using your Kubernetes cluster the practical way, we invite you to look at our OVHcloud Managed Kubernetes documentation site.
-
If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for assisting you on your specific use case of your project.
-
Join our community of users.