[werft] sometimes Harvester VMs timeout, causing werft builds to fail

[kylos101]

Bug description

Given this job, and it's machine, it looks like it has been starting for 33 minutes.

From what I can tell, the VM pod was never scheduled to a node. See how in the attached first screenshot there is no centvester node associated with preview-wk-inte-te7319239f65? This is odd, because there are hosts which have available resources (second screenshot).

Our werft job failed with:
[Waiting for VM readiness] Timeout while waiting for VM to get ready. Timeout: 600

Steps to reproduce

n/a

Workspace affected

n/a

Expected behavior

The Harvester VM preview environment would start in < 10m

Example repository

No response

Anything else?

Related internal thread

[kylos101]

Related:
https://werft.gitpod-dev.com/job/gitpod-build-wk-inte-test-20220907015739.0/raw
https://werft.gitpod-dev.com/job/gitpod-build-wk-inte-test-20220907015734.0/raw
https://werft.gitpod-dev.com/job/gitpod-build-wk-inte-test-20220907013042.0/raw
https://werft.gitpod-dev.com/job/gitpod-build-wk-inte-test-20220907013036.0/raw

[mads-hartmann]

Investigation happening in the internal Slack channel inc-2022-09-07-internal-unable-to-create-preview-environment and this PR might cause VMs to start again #12719

[utam0k]

this PR might cause VMs to start again #12719

@mads-hartmann There is no problem. These all VMs are temporary VMs for running integration tests

[mads-hartmann]

Closing this as I believe it was due to the incident. If it happens again, please let us know 🧡