Restore from snapshot is always empty

[bhavenjp]

Restore from snapshot creates a new volume however the data of the original volume is not restored.

Source PVC with data inside:
[root@k8smaster examples]# kubectl exec -it my-csi-app /bin/sh
/ #
/ #
/ # cd /data
/data # ls -lrt
total 4
-rw-r--r-- 1 root root 7 Feb 18 12:54 TESTFi
/data #
/data # cat TESTFi
FDFDSD
/data #

Created snapshot
[root@k8smaster csi-driver-host-path]# kubectl apply -f examples/csi-snapshot.yaml
volumesnapshot.snapshot.storage.k8s.io/new-snapshot-demo created
[root@k8smaster csi-driver-host-path]#

[root@k8smaster examples]# kubectl get volumesnapshot
NAME AGE
new-snapshot-demo 17m
[root@k8smaster examples]#
[root@k8smaster examples]# kubectl describe volumesnapshotcontent
Name: snapcontent-b42fa9dd-9405-4f1e-8044-4ef252c7703f
Namespace:
Labels:
Annotations:
API Version: snapshot.storage.k8s.io/v1alpha1
Kind: VolumeSnapshotContent
Metadata:
Creation Timestamp: 2020-02-18T12:55:33Z
Finalizers:
snapshot.storage.kubernetes.io/volumesnapshotcontent-protection
Generation: 1
Resource Version: 1179
Self Link: /apis/snapshot.storage.k8s.io/v1alpha1/volumesnapshotcontents/snapcontent-b42fa9dd-9405-4f1e-8044-4ef252c7703f
UID: 2ece1990-7397-467c-abcd-328aede2d570
Spec:
Csi Volume Snapshot Source:
Creation Time: 1582030533340217899
Driver: hostpath.csi.k8s.io
Restore Size: 1073741824
Snapshot Handle: f323fa2e-524d-11ea-a162-2e85f875df77
Deletion Policy: Delete
Persistent Volume Ref:
API Version: v1
Kind: PersistentVolume
Name: pvc-cdbe2464-e081-4614-b29c-f11fece38654
Resource Version: 809
UID: 61e0d65e-a4b6-44ef-8ecb-6993852b0c8c
Snapshot Class Name: csi-hostpath-snapclass
Volume Snapshot Ref:
API Version: snapshot.storage.k8s.io/v1alpha1
Kind: VolumeSnapshot
Name: new-snapshot-demo
Namespace: default
Resource Version: 1175
UID: b42fa9dd-9405-4f1e-8044-4ef252c7703f
Events:
[root@k8smaster examples]#

Restoring from snapshot
[root@k8smaster csi-driver-host-path]# kubectl apply -f examples/csi-restore.yaml
persistentvolumeclaim/hpvc-restore created
[root@k8smaster csi-driver-host-path]#

[root@k8smaster examples]# kubectl get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
csi-pvc Bound pvc-cdbe2464-e081-4614-b29c-f11fece38654 1Gi RWO csi-hostpath-sc 22m
hpvc-restore Bound pvc-6cff0d9c-785e-4f92-acb6-e4a013d120f0 1Gi RWO csi-hostpath-sc 17m
[root@k8smaster examples]#

However the restored volume is empty

• Tried multiple times with the same result*
  [root@k8smaster examples]# kubectl exec -it $(kubectl get pods --selector app=csi-hostpathplugin -o jsonpath='{.items[*].metadata.name}') -c hostpath /bin/sh
  / #
  / #
  / # cd /csi-data-dir/
  /csi-data-dir #
  /csi-data-dir # ls -lrt
  total 6
  drwxr-xr-x 2 root root 20 Feb 18 12:54 56c65c8a-524d-11ea-a162-2e85f875df77
  -rw-r--r-- 1 root root 135 Feb 18 12:55 f323fa2e-524d-11ea-a162-2e85f875df77.tgz
  drwxr-xr-x 2 root root 6 Feb 18 12:55 fcbe5c00-524d-11ea-a162-2e85f875df77
  /csi-data-dir # ls -l 56c65c8a-524d-11ea-a162-2e85f875df77
  total 4
  -rw-r--r-- 1 root root 7 Feb 18 12:54 TESTFi
  /csi-data-dir #
  /csi-data-dir # tar tvfz f323fa2e-524d-11ea-a162-2e85f875df77.tgz
  drwxr-xr-x root/root 0 2020-02-18 12:54:48 ./
  -rw-r--r-- root/root 7 2020-02-18 12:54:48 ./TESTFi
  /csi-data-dir #
  /csi-data-dir # ls -l fcbe5c00-524d-11ea-a162-2e85f875df77
  total 0
  /csi-data-dir #

[aayush-rangwala]

@bhavenjp Can you share the exported yaml of your restored pvc (-o yaml)?

[msau42]

cc @xing-yang

[xing-yang]

@bhavenjp Can you provide versions of external-snapshotter sidecar, external-provisioner sidecar, and hostpath CSI driver? Have you tried this with the snapshot beta CRD and controller?

[aayush-rangwala]

@bhavenjp @xing-yang I had faced a similar issue just like this before, the problem was, I was using csi-hospath-driver v1.2.0 for k8s-1.14 and I didn't enabled the feature gate VolumeSnapshotDataSourcce due to which, even when I give the VolumeSnapshot as the data source in restore pvc, the field was being removed

[bhavenjp]

Hi All,

Thank you for your reply. I understand that this feturegate is enabled by default in 1.17.
My test environment is crashed and I shall build a new one soon and test it again

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.