Add instrumentation to git provider API usage

[aThorp96]

Currently there are no metrics exposed around the API load PaC imposes on the git providers. Given that PaC ingests every event it receives and relies on these providers' APIs to determine if the event is actionable, a large number of repos, especially active repos, could result in a heavy API usage and even API throttling.

Instrumenting the providers with metrics on API usage would help with this in the following ways:

• SREs seeing API throttling may be able to inspect if there are particular repositories, events-types, etc which are causing undue load
• PaC contributors can use these metrics to identify API hot spots and redundancies in the code base, informing potential optimization efforts
• PaC e2e tests can track these metrics and ensure that API load does not change drastically or unexpectedly

I don't think it would suffice to instrument implementations of the provider.Interface at the interface level. Instead I would suggest we instrument each implementation's usage of their underlying api Client. The former is more consistent to implement in the code base, but the latter would be far more accurate. As an example (*gitlab.Provider) IsAllowed may make 3 API requests if the user is allowed via the OWNERS_ALIASES file (1: check project membership, 2: get OWNERS file, 3: get OWNERS_ALIASES file), but if the even's user is neither a project member nor an owner, then PaC makes an additional N requests to page over the comments in the MR, and for any comments which match /ok-to-test an additional 1-3 requests are made as we check if the comment author is an owner (breaking the loop on the first owner found, of course, but still).

[chmouel]

maybe we want to attach a OSP jira to it since it would be worthwhile to mention and document for downstream release

[aThorp96]

OSP jira ticket made: SRVKP-7116

[aThorp96]

/assign @aThorp96