[usage] Add periodic job to detect invalid workspace instances in usage

[easyCZ]

Description

This PR does the following:

1. Adds a scheduled job (every 15m) which detects if there are any instances in the usage table which are for workspace instances which have stopped, but do not have a stoppingTime set. In this case, these instances would cause runaway credit usage for our customers.
2. When such an instance is detected, a metric is updated to allow us to alert on this.

To avoid code-duplication, the existing Ledger Job is refactored to introduce the following:

1. Wrapper which can compose a Job to enable concurrent execution prevention
2. Tests are updated to reflect this change

Related Issue(s)

Fixes #

How to test

Unit tests

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[svenefftinge]

Maintaining the consistency of workspace instance data is the responsibility of bridge. I really think we should fix any issues there and if needed add such metrics there as well.

[easyCZ]

While I agree, and the fix for this should go into bridge, the detection here joins against usage and gives us a signal in relation to the usage table. As such, I'd like this signal to be on usage as that's the closest domain for it.

To put it differently, I see this as defensive from the usage side.

[svenefftinge]

We talked about this in a call and decided to go with this for now and see in a couple of weeks if it provides value and otherwise remove it again.