fix(grpc): Add startupProbe to check for grpc health readiness

[dinhxuanvu]

Currently, liveness and readiness probes may fail due to grpc is not
ready. Adding a startupProbe will ensure grpc is ready before
liveness and readiness probes are triggered.

Signed-off-by: Vu Dinh [email protected]

Description of the change:

Motivation for the change:

Architectural changes:

Testing remarks:

Reviewer Checklist

• Implementation matches the proposed design, or proposal is updated to match implementation
• Sufficient unit test coverage
• Sufficient end-to-end test coverage
• Bug fixes are accompanied by regression test(s)
• e2e tests and flake fixes are accompanied evidence of flake testing, e.g. executing the test 100(0) times
• tech debt/todo is accompanied by issue link(s) in comments in the surrounding code
• Tests are comprehensible, e.g. Ginkgo DSL is being used appropriately
• Docs updated or added to /doc
• Commit messages sensible and descriptive
• Tests marked as [FLAKE] are truly flaky and have an issue
• Code is properly formatted

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dinhxuanvu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [dinhxuanvu]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[grokspawn]

/lgtm

[grokspawn]

Several technical options were discussed to attempt to resolve this issue:

1. make GRPC serving and FBC reading disjoint, where the endpoint would return a "not ready" until the FBC load is complete (for e.g. '202 -- Request Accepted').
2. as type: image preparation, pre-process FBC into a binary blob to change FBC service as a linear load to constant-time.
3. add a startupProbe of sufficient length to allow the framework to understand that the CatalogSource pod takes more time to get initially ready.

3 was the fastest to implement while we continue to work through the problem domain to see if further effort is needed to either resolve some additional problem or simplify the catalog release issues which might arise from the selected approach.

[grokspawn]

Just in case anyone is landing here, looking for what happened next, the direction is here

[alfieyfc]

@grokspawn I see this fix was after the latest release v0.21.2. Is there a way to install olm with this fix now? Do we have to wait until operator-framework/operator-registry#977 to be merged for this to work? Or, is there a page for users to check when the next release is scheduled? 🙏

[dinhxuanvu]

@grokspawn I see this fix was after the latest release v0.21.2. Is there a way to install olm with this fix now? Do we have to wait until operator-framework/operator-registry#977 to be merged for this to work? Or, is there a page for users to check when the next release is scheduled? 🙏

We will release a new version of OLM soon. Sometime this week I believe. Stay tuned.