[usage] Alert on Usage and Invoice Reconciliations #12919
Conversation
| runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/GitpodUsageScheduledReconciliationFailures.md | ||
| summary: There are failed scheduled reconciliations in the usage component. | ||
| description: We have accumulated {{ printf "%.2f" $value }} failures. This affects how stale usage data is and/or updating invoices in Stripe. | ||
| runbook_url: https://github.com/gitpod-io/runbooks/blob/main/runbooks/GitpodUsageReconcileUsageFailures.md |
There was a problem hiding this comment.
Is this the new name for this runbook?
There was a problem hiding this comment.
Yeah I'll need to update the runbooks correspondingly. For now these would only be Slack warnings in webapp so we've some time to polish the runbooks before we get it to actually page
There was a problem hiding this comment.
| expr: sum(increase(grpc_server_handled_total{grpc_service="usage.v1.BillingService", grpc_method="ReconcileInvoices", grpc_code!="OK"})) > 1 | ||
| for: 30m |
There was a problem hiding this comment.
I'm relatively new to PromQL, but what does the for: 30m do here?
There was a problem hiding this comment.
It says that the expr above needs to "have values" - be firing for 30m to fire an alert
There was a problem hiding this comment.
So does that mean we alert when reconciliation fails for 30 minutes continuously, ie just one failure won't alert? (not sure how often we run reconciliation in prod currently.)
| expr: sum(increase(grpc_server_handled_total{grpc_service="usage.v1.BillingService", grpc_method="ReconcileInvoices", grpc_code!="OK"})) > 1 | ||
| for: 30m |
There was a problem hiding this comment.
So does that mean we alert when reconciliation fails for 30 minutes continuously, ie just one failure won't alert? (not sure how often we run reconciliation in prod currently.)
roboquat
added
deployed: webapp
Description
Alert on the two core RPCs used in usage reconciliation.
Related Issue(s)
How to test
Release Notes
Documentation
Werft options: