Support for EC2 Spot Best practices - Diversification of instances using ABS

[ruecarlo]

Currently the lambda in charge of scaling up nodes (including Spot configurations), uses the RunInstance API to create a Spot instance. Spot instances as spare capacity might be limited. Best approach when using Spot instance is to diversify across a set of instances that qualify for the workload and use one of the API that allows for that diversification.

The suggestion here is to change the RunInstance call and instead use the Drop In replacement API for EC2 Fleet in instant and the Spot Capacity-Optimized allocation strategy. EC2 Fleet allows diversification and still provides a synchronous API that adhere to Spot best practices providing the spot instance types that are selected to minimise the frequency of interruptions for the workload. More examples here.

Another thing for consideration in the implementation is to use the newly released attribute based instance selection

[npalm]

I did a short experiment, see gist. It creates the instances but caues 2 issues

• it also tries to launch instances in the default vpc, which fails since the required security group is not available in the default VPC. Not figured out, how to avoid possible creation. The result of the createFleet contains 1 instance in a valid subnet. But also the error mentioned before.
• Creating the SSM properties does not work, did not investigate at all yet

[npalm]

With replacing the runInstance API call by createFleet also the dynamic launch templates can be removed. Finding a right spot instances (or on demand) should be moved to this API call. https://github.com/philips-labs/terraform-aws-github-runner/blob/dbba705038828c86f6f5adef18f7a7a35643c359/modules/runners/main.tf#L57

[npalm]

Currently we specify in the launch template via the option market_type whenever we create a spot or on-demand. Maybe we should move this logic also to the API call for creating a fleet.

[npalm]

Have a local working POC ready, will implement asap. This will replace the loop for creating instances. And add a fall back to on-demand instances