[bridge] Marks stuck stopping and pending instances as stopped

[laushinka]

Description

After validating through logging of the past two weeks (since 15.09.2022), this change marks stopping and pending states after the timeout duration to be stopped, as well as running states that the ws-manager does not know about.
Findings from the logging:

1. Most of the logs were for stopping state, and we could not find a clear explanation whether they were cases that we want to be stopped. Instances could be in a stopping state for longer than we expect (in the previous PR it was 10 seconds after stoppingTime) because instances with a big back-up size can take longer.
2. A lot of the pending states were stuck in that state for days, and we do want them stopped.
3. We found a few creating states that ended up running, and these we should not stop.

We are still unsure why these instances end up in these states. Therefore we:

1. Will focus on only stopping stuck instances in pending and stopping states after the timeout duration, and running states. These we know for sure we want stopped if ws-manager does not know about them.
2. To understand the why, we will create an issue.

Related Issue(s)

Fixes #11397

How to test

1. kubectl cordon the preview node
2. Run a workspace. It should be stuck in pending.
3. Change the creationTime in d_b_workspace_instance to 1 hour before (or wait 1 hour). It should be marked as stopped with a message "Stopped by ws-manager-bridge. Previously in phase pending.
4. Once done, kubectl uncordon the node, otherwise builds will fail because the node can't be scheduled.

Release Notes

NONE

Documentation

Werft options:

• /werft with-local-preview
  If enabled this will build install/preview
• /werft with-preview
• /werft with-integration-tests=all
  Valid options are all, workspace, webapp, ide

[werft-gitpod-dev-com]

started the job as gitpod-build-lau-pending-stopping-11397.4 because the annotations in the pull request description changed
(with .werft/ from main)

[geropl]

@laushinka Sorry for the long turnaround. I like the new layout of the loop! But we need it to include this case: If an instance in runningInstancesIdx is running, we still need to stop it unconditionally, as we do it at the moment.

[laushinka]

Makes sense that we should handle the running state as we do it now. Thanks, will add it!

[geropl]

Code LGTM, thx @laushinka ! 🙏
Will test now...

[laushinka]

Code LGTM, thx @laushinka ! 🙏 Will test now...

@geropl Cool! Let me know if the steps I described were helpful, or if you test in a different way 🙏🏽

[geropl]

Code LGTM, tested and works as expected! 🥇

Let me know if the steps I described were helpful, or if you test in a different way

Did as you described, just had two meetings in between 😉