refactor: Group setup by feature flag

[dlipovetsky]

What type of PR is this?
/kind cleanup

What this PR does / why we need it:
While reviewing #3633, I noticed that we had set up reconcilers and webhooks based on feature flags in main itself, although we put some code in a separate enableGates function. I also noticed that we had two identical conditional blocks to setup EKS-only reconcilers and webhooks, respectively.

To make the setup code easier to follow, I grouped it by feature flag, because that seems like the most useful grouping. (This means we interleave the setup of reconcilers and webhooks; I think this is safe, but hope to get feedback).

I also moved the EKS-only setup into a separate function.

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #

Special notes for your reviewer:

Checklist:

• squashed commits
• includes documentation
• adds unit tests
• adds or updates e2e tests

[k8s-ci-robot]

@dlipovetsky: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[dlipovetsky]

Test failed:

=== RUN   TestAWSMachineReconciler_Reconcile/Should_not_Reconcile_if_AWSCluster_is_not_ready
I0804 00:20:43.000630   22680 http.go:97] controller-runtime/webhook/webhooks "msg"="received request" "UID"="d669b5fa-cd56-43e7-b8de-f0d23f9f7fdd" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","kind":"AWSMachine"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"awsmachines"} "webhook"="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-awsmachine"
I0804 00:20:43.001033   22680 http.go:136] controller-runtime/webhook/webhooks "msg"="wrote response" "UID"="d669b5fa-cd56-43e7-b8de-f0d23f9f7fdd" "allowed"=true "code"=200 "reason"="" "webhook"="/mutate-infrastructure-cluster-x-k8s-io-v1beta1-awsmachine"
I0804 00:20:43.003586   22680 http.go:97] controller-runtime/webhook/webhooks "msg"="received request" "UID"="15e768c4-d44b-452d-a280-7793a5de6352" "kind"={"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","kind":"AWSMachine"} "resource"={"group":"infrastructure.cluster.x-k8s.io","version":"v1beta1","resource":"awsmachines"} "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1beta1-awsmachine"
I0804 00:20:43.003815   22680 http.go:136] controller-runtime/webhook/webhooks "msg"="wrote response" "UID"="15e768c4-d44b-452d-a280-7793a5de6352" "allowed"=true "code"=200 "reason"="" "webhook"="/validate-infrastructure-cluster-x-k8s-io-v1beta1-awsmachine"
    awsmachine_controller_unit_test.go:2312: 
        Expected
            <*errors.StatusError | 0xc000b5ef00>: {
                ErrStatus: {
                    TypeMeta: {Kind: "", APIVersion: ""},
                    ListMeta: {
                        SelfLink: "",
                        ResourceVersion: "",
                        Continue: "",
                        RemainingItemCount: nil,
                    },
                    Status: "Failure",
                    Message: "Machine.cluster.x-k8s.io \"capi-test-machine\" not found",
                    Reason: "NotFound",
                    Details: {
                        Name: "capi-test-machine",
                        Group: "cluster.x-k8s.io",
                        Kind: "Machine",
                        UID: "",
                        Causes: nil,
                        RetryAfterSeconds: 0,
                    },
                    Code: 404,
                },
            }
        to be nil
...
    --- FAIL: TestAWSMachineReconciler_Reconcile/Should_not_Reconcile_if_AWSCluster_is_not_ready (0.06s)

I've repeated (not using cached results) this test 10 times locally but still haven't seen a failure.

[Ankitasw]

/retest

[Ankitasw]

retest passed, weird that it failed as this issue was fixed long back. lets see if we face this again, although little difficult to debug if not failing locally.

[richardcase]

/test ?

[k8s-ci-robot]

@richardcase: The following commands are available to trigger required jobs:

• /test pull-cluster-api-provider-aws-build
• /test pull-cluster-api-provider-aws-test
• /test pull-cluster-api-provider-aws-verify

The following commands are available to trigger optional jobs:

• /test pull-cluster-api-provider-aws-apidiff-main
• /test pull-cluster-api-provider-aws-e2e
• /test pull-cluster-api-provider-aws-e2e-blocking
• /test pull-cluster-api-provider-aws-e2e-clusterclass
• /test pull-cluster-api-provider-aws-e2e-conformance
• /test pull-cluster-api-provider-aws-e2e-conformance-with-ci-artifacts
• /test pull-cluster-api-provider-aws-e2e-eks

Use /test all to run the following jobs that were automatically triggered:

• pull-cluster-api-provider-aws-apidiff-main
• pull-cluster-api-provider-aws-build
• pull-cluster-api-provider-aws-test
• pull-cluster-api-provider-aws-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[richardcase]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[richardcase]

I like this change @dlipovetsky , its a lot cleaner 👍

From myside:

/lgtm

(assuming the e2e pass)

[Ankitasw]

/test pull-cluster-api-provider-aws-e2e
/test pull-cluster-api-provider-aws-e2e-eks

[sedefsavas]

/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sedefsavas

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [sedefsavas]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment