Merge pull request #9797 from tmorriso-rh/enterprise-3.9

[enterprise-3.9] Add edits to PR 8990

Author: Traci Morrison

dev_guide/expanding_persistent_volumes.adoc

@@ -20,7 +20,7 @@ cases.
 
 [NOTE]
 ====
-For more information on Red Hat Technology Preview features support scope,  see
+For more information on Red Hat Technology Preview features support scope, see
 https://access.redhat.com/support/offerings/techpreview/. 
 ====
 
@@ -66,10 +66,9 @@ kubernetesMasterConfig:
 [[expanding_glusterfs_pvc]]
 == Expanding GlusterFS-Based Persistent Volume Claims
 
-Expanding GlusterFS volumes is easiest. Once the  {product-title} administrator
-has created a StorageClass with `allowVolumeExpansion` set to `true`, you can
-create a PVC from that class, and afterwards, whenever needed, you can edit the
-PVC and request a new size. 
+Once the {product-title} administrator has created a StorageClass with
+`allowVolumeExpansion` set to `true`, you can create a PVC from that class, and
+afterwards, whenever needed, you can edit the PVC and request a new size. 
 
 For example:
 
@@ -116,3 +115,19 @@ object reflects the newly requested size in `PersistentVolume.Spec.Capacity`. At
 this point, you can create or re-create a new pod from the PVC to finish the
 file system resizing. Once the pod is running, the newly requested size is
 available and `FileSystemResizePending` condition is removed from the PVC.
+
+[[recover_from_resize_failure]]
+== Recovering from Failure when Expanding Volumes
+
+If expanding underlying storage fails either on master or node, the
+{product-title} administrator can manually recover the PVC state and cancel the resize
+requests that are continuously retried by the controller without administrator
+intervention.
+
+Currently, this can be done manually by completing the following steps:
+
+. Mark the PV that is bound to the claim (PVC) with the `Retain` reclaim policy. This can be done by editing the PV and changing `persistentVolumeReclaimPolicy` to `Retain`.
+. Delete the PVC (it will be recreated later).
+. To ensure that the newly created PVC can bind to the PV marked `Retain`, manually edit the PV and delete the `claimRef` entry from the PV specs. This marks the PV as `Available`. For more information about prebinding PVCs, see xref:./persistent_volumes.adoc#persistent-volumes-volumes-and-claim-prebinding[volume and claim prebinding].
+. Re-create the PVC in a smaller size or a size that can be allocated by the underlying storage provider. Also, set the `volumeName` field of the PVC to the name of the PV. This binds the PVC to the provisioned PV only.
+. Restore the reclaim policy on the PV.