feat: Track git provider API usage metrics #2005

aThorp96 · 2025-03-18T13:45:16Z

Changes

Every time we access a Git provider's API we may use some of that
provider's API limits. Medium-sized providers may have many repositories
using PaC but may not have the computer overhead to support permissive
API rate limits. In some cases we have seen pipelines delayed due to API
rate limiting.

By tracking the Git providers' API load by PaC we can gain insight into
potential redundancies, hot spots, and optimizations to the Git provider
API use.

Since the Git provider API clients do not expose these metrics and are
not designed in a way that is easy to decorate, the closest proxy to
tracking actual API calls is tracking accesses to the API client itself.
Since one client-method call equates to one API call, this metric should
track the API usage well. However if a developer saves a reference to the
API client then they will be able to make API calls without the metric
incrementing; we will have to watch out for this if we want the metric
to be accurate.

issue: #1925

Submitter Checklist

📝 Ensure your commit message is clear and informative. Refer to the How to write a git commit message guide. Include the commit message in the PR body rather than linking to an external site (e.g., Jira ticket).
♽ Run make test lint before submitting a PR to avoid unnecessary CI processing. Consider installing pre-commit and running pre-commit install in the repository root for an efficient workflow.
✨ We use linters to maintain clean and consistent code. Run make lint before submitting a PR. Some linters offer a --fix mode, executable with make fix-linters (ensure markdownlint and golangci-lint are installed).
📖 Document any user-facing features or changes in behavior.
🧪 While 100% coverage isn't required, we encourage unit tests for code changes where possible.
🎁 If feasible, add an end-to-end test. See README for details.
🔎 Address any CI test flakiness before merging, or provide a valid reason to bypass it (e.g., token rate limitations).
If adding a provider feature, fill in the following details:

Git Provider	Supported
GitHub App	✅️
GitHub Webhook	✅️
Gitea	✅️
GitLab	✅️
Bitbucket Cloud	✅️
Bitbucket Data Center	✅️

(update the documentation accordingly)

zakisk · 2025-03-19T09:32:46Z

@aThorp96 have you checked that controller is emitting metrics?, I checked and it doesn't. and also controller and watcher both call git providers APIs, so both are emitting metrics so it needs for users to configure metrics for both controller and watcher and that is big change. and if you configure metrics for both, sum of both is the number of API calls on a git event 🙁.
cc: @chmouel

zakisk · 2025-03-19T09:36:49Z

@aThorp96 also controller is a "knative adapter" and watcher is "knative reconciler" so I am not sure whether it is possible to emit metrics from adapter or not (it could be a topic for you)

zakisk · 2025-03-20T07:11:15Z

@aThorp96 yeah, I saw metrics emitted from controller, I was doing it before in wrong way.

zakisk · 2025-03-20T07:12:56Z

@aThorp96 have you checked that controller is emitting metrics?, I checked and it doesn't. and also controller and watcher both call git providers APIs, so both are emitting metrics so it needs for users to configure metrics for both controller and watcher and that is big change. and if you configure metrics for both, sum of both is the number of API calls on a git event 🙁. cc: @chmouel

@aThorp96 but still there is question about two metrics services, (watcher, controller)?

zakisk · 2025-03-20T07:15:06Z

pkg/metrics/metrics.go

+		return err
+	}
+
+	ctx, err := tag.New(


@aThorp96 at the moment metrics have event tag but user may want to filter out metrics based on event SHA, so it would be better to add it.

Event sha could be useful for debugging. I worry about the cardinality of tags though if we introduce the SHA as a tag. WDYT?

yeah, cardinality could be issue but there is no other way to differentiate that how many API calls were there on an event. for example it has eventType tag but there could be many push events so API request count in this metric will be cumulative in this case.

That's fair. I think for that use-case may be better suited for tracing, replaying an event, using a unit or e2e test.

The way I see this being used is to first identify particularly heavy event types and repositories. I think that is useful for an SRE alone, but if an SRE wanted to know how many API calls were made for a given event, I think there are more appropriate tools/techniques than metrics

That's fair. I think for that use-case may be better suited for tracing, replaying an event, using a unit or e2e test.

I don't think that user may want to run e2e tests in production 🙂

Requiring the client object be accessed via an accessor method allows for hooking into API usage for things like API metrics, since the github-client library API is designed in a way difficult to decorate.

Every time we access a Git provider's API we may use some of that provider's API limits. Medium-sized providers may have many repositories using PaC but may not have the computer overhead to support permissive API rate limits. In some cases we have seen pipelines delayed due to API rate limiting. By tracking the Git providers' API load by PaC we can gain insight into potential redundancies, hot spots, and optimizations to the Git provider API use. Since the Git provider API clients do not expose these metrics and are not designed in a way that is easy to decorate, the closest proxy to tracking actual API calls is tracking accesses to the API client itself. Since one client-method call equates to one API call, this metric should track the API usage well. However if a developer saves a reference to the API client then they will be able to make API calls without the metric incrementing; we will have to watch out for this if we want the metric to be accurate.

aThorp96 · 2025-03-21T20:43:46Z

@zakisk per our discussion, I added some documentation regarding the git-provider metrics being emitted from both services. Here is a screenshot of one of the example queries I included, taken after running some of our end to end tests

chmouel · 2025-03-26T11:58:11Z

/retest

chmouel · 2025-03-19T07:58:39Z

pkg/metrics/metrics.go

+				Description: gitProviderAPIRequestCount.Description(),
+				Measure:     gitProviderAPIRequestCount,
+				Aggregation: view.Count(),
+				TagKeys:     []tag.Key{R.provider, R.eventType, R.namespace, R.repository},


would that give the ability to track each provider separately ? i mean is the admin can get insight how many calls has been done for gitlab and for github app when both are used on the same cluster

@chmouel Yes, this metric can be filtered using provider tag.

Exactly. Additionally the provider tag for Gitlab and Gitea will use the API URL so you can see insight per provider-instance. Similarly Github will use either github or github-enterprise respectively

zakisk · 2025-04-04T07:02:42Z

pkg/provider/bitbucketcloud/bitbucket.go

+	return v.bbClient
+}
+
+func (v *Provider) recordAPIUsageMetrics() {


@aThorp96 this is implemented for every provider and seems repeated code, can you find a way for a common logic, wdyt?

zakisk · 2025-04-04T07:03:41Z

pkg/provider/bitbucketcloud/bitbucket.go

+	if v.metrics == nil {
+		m, err := metrics.NewRecorder()
+		if err != nil {
+			v.Logger.Errorf("Error initializing bitbucketcloud metrics recorder: %v", err)
+			return
+		}
+		v.metrics = m
+	}


@aThorp96 like doing this in a new method on provider interface e.g. GetMetrics and the using that method for emitting metrics from a single common function.

zakisk · 2025-04-04T07:06:56Z

pkg/provider/bitbucketcloud/bitbucket.go

+		namespace = v.repo.Namespace
+	}
+
+	if err := v.metrics.ReportGitProviderAPIUsage("bitbucketcloud", v.triggerEvent, namespace, name); err != nil {


@aThorp96 for provider name, you've GetConfig function implemented for every provider and it will persist uniformity of naming convention as well like see here bitbucket-cloud has hyphen. same for above error message as well (line no 47 in this file)

aThorp96 changed the title ~~Git api usage metrics~~ feat: Track git provider API usage metrics Mar 18, 2025

aThorp96 force-pushed the git_api_usage_metrics branch 2 times, most recently from 35c8495 to a45829a Compare March 18, 2025 15:55

aThorp96 marked this pull request as ready for review March 18, 2025 16:58

aThorp96 requested review from chmouel and zakisk March 18, 2025 16:58

zakisk reviewed Mar 20, 2025

View reviewed changes

aThorp96 force-pushed the git_api_usage_metrics branch from a45829a to 9f912e4 Compare March 21, 2025 20:30

aThorp96 temporarily deployed to PAC March 21, 2025 20:30 — with GitHub Actions Inactive

aThorp96 added 2 commits March 21, 2025 16:38

refactor: change providers' inner Client privacy to private

ba2c99d

Requiring the client object be accessed via an accessor method allows for hooking into API usage for things like API metrics, since the github-client library API is designed in a way difficult to decorate.

aThorp96 force-pushed the git_api_usage_metrics branch from 9f912e4 to 7704ffb Compare March 21, 2025 20:41

aThorp96 temporarily deployed to PAC March 21, 2025 20:41 — with GitHub Actions Inactive

Merge branch 'main' into git_api_usage_metrics

7975d30

osp-pac added the e2e label Mar 26, 2025

chmouel reviewed Mar 27, 2025

View reviewed changes

aThorp96 requested review from chmouel and zakisk March 28, 2025 12:02

zakisk reviewed Apr 4, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Track git provider API usage metrics #2005

feat: Track git provider API usage metrics #2005

aThorp96 commented Mar 18, 2025 •

edited by zakisk

Loading

zakisk commented Mar 19, 2025

zakisk commented Mar 19, 2025

zakisk commented Mar 20, 2025

zakisk commented Mar 20, 2025

zakisk Mar 20, 2025

aThorp96 Mar 20, 2025

zakisk Mar 26, 2025 •

edited

Loading

aThorp96 Mar 26, 2025

zakisk Apr 4, 2025

aThorp96 commented Mar 21, 2025

chmouel commented Mar 26, 2025

chmouel Mar 19, 2025

zakisk Mar 27, 2025

aThorp96 Mar 27, 2025 •

edited

Loading

zakisk Apr 4, 2025

zakisk Apr 4, 2025

zakisk Apr 4, 2025

feat: Track git provider API usage metrics #2005

Are you sure you want to change the base?

feat: Track git provider API usage metrics #2005

Conversation

aThorp96 commented Mar 18, 2025 • edited by zakisk Loading

Changes

Submitter Checklist

zakisk commented Mar 19, 2025

zakisk commented Mar 19, 2025

zakisk commented Mar 20, 2025

zakisk commented Mar 20, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zakisk Mar 26, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aThorp96 commented Mar 21, 2025

chmouel commented Mar 26, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aThorp96 Mar 27, 2025 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aThorp96 commented Mar 18, 2025 •

edited by zakisk

Loading

zakisk Mar 26, 2025 •

edited

Loading

aThorp96 Mar 27, 2025 •

edited

Loading