workspaces stuck in INITIALIZING

[geropl]

incident: https://app.incident.io/incidents/260
affected cluster: us72

We noticed that new workspace got stuck in INITIALIZING (their content readiness probe failed), piods had status RUNNING.
The only thing we identified was that ws-manager consumed a lot of RAM and CPU. Restarting it mitigated the symptoms.

These are the logs from around the time when CPU & allocation rates started rocketing: logs
I see only two errors:

• The container could not be located when the pod was deleted. The container used to be Running (instanceId: f5345fbd-2625-45ec-b58f-a719df59e477, log)
• doFinalize failed (instanceId: d309b3b3-61fe-428d-b46c-8e6c5558fc1d, log)

Other resources:

• Related slack thread
• ws-manager pprof

[geropl]

Potentially related changes:

• [ws-manager] Do not follow redirect when ready probe #13828
• [ws-manager] Fix potential leak of request body #13957

[mrsimonemms]

This appears to have triggered again just now - https://gitpod.pagerduty.com/incidents/Q1HZ3KVKZ31OZ8

[jenting]

ws-manager pprof

[atduarte]

Potentially resolved by #14214.

@aledbf were you able to reproduce the issue and verify that what is described in this issue is not happening anymore?

[utam0k]

The container could not be located when the pod was deleted. The container used to be Running (instanceId: f5345fbd-2625-45ec-b58f-a719df59e477, log)

I have created the issue about it
#12021

[kylos101]

@aledbf were you able to reproduce the issue and verify that what is described in this issue is not happening anymore?

For gen72, we did a pprof as @jenting shared above, also observed goroutine count were very high 10k and 20k, and fixed with gen73 via #14214.

We cannot necessarily recreate the original issue (being unable to start workspaces, accruing regular not active workspaces), though. However, the goroutine count is now stable in gen73 (below 1k for both clusters while active). We asserted the goroutine behavior prior to shipping gen73 when load testing, and updated our baseline accordingly.

If ws-manager is unable to start workspaces again, and they get stuck in Regular Not Active, we should carefully observe it's dashboard, and create a new issue.

I am going to move this issue to Done, we've validated the goroutine leak is fixed.