ws-mananger: refactor volume snapshot watcher

[jenting]

Description

Refactor volume snapshot watcher.

Using the Go channel to notify different event workers that the volume snapshot is ready seems risky.
We observe that at gen67 cluster, some workspaces with PVC can't be terminated. The symptom is the workspace pod starts the disposal process, and the volume snapshot is ready to use. But somehow, the event does not receive by the receiver or does not send to the sender.

We rewrite the watcher with the volume snapshot client watch + retry watcher to make sure the watcher happens in each event worker. To minimize the chance that the event does not receive by the receiver or does not send to the sender.
Moreover, we fall back to exponential retry to get the volume snapshot object if we can't create the retry watcher.

Related Issue(s)

Fixes #13280

How to test

Create an ephemeral cluster, and run loadgen test with 100 regular workspaces + 100 regular workspaces with PVC.
Ensure no workspace pod is terminating for a long time.

Release Notes

None

Documentation

None

Werft options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-integration-tests=workspace
  Valid options are all, workspace, webapp, ide

[werft-gitpod-dev-com]

started the job as gitpod-build-jenting-pvc-stuck-on-stopping.12 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-jenting-pvc-stuck-on-stopping.13 because the annotations in the pull request description changed
(with .werft/ from main)

[werft-gitpod-dev-com]

started the job as gitpod-build-jenting-pvc-stuck-on-stopping.17 because the annotations in the pull request description changed
(with .werft/ from main)

[jenting]

I use the us68 cluster to test loadgen with 100 regular workspaces + 100 regular workspaces with PVC, I can reproduce the workspace in the terminating state.

After I patch the ws-manager with this PR and rerun the same test (100 regular workspaces + 100 regular workspace with PVC), I can't reproduce the workspace in terminating state.

Therefore, I think it's good to review.

[sagor999]

/hold

LGTM, but hold for minor nit.

[jenting]

nit addressed.
/unhold