Skip to content

[ops] WebApp: Fix team-specific alerts #12588

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Sep 2, 2022
Merged

[ops] WebApp: Fix team-specific alerts #12588

Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
9852898
[ops] WebApp: Fix alert WebsocketConnectionRateHigh by using a rate(t…
geropl Sep 1, 2022
3b70e47
[ops] WebApp: Remove rate(memory): rate(gauge) does not work
geropl Sep 1, 2022
316223f
[ops] WebApp: Fix WebAppServicesCrashlooping
geropl Sep 1, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
25 changes: 5 additions & 20 deletions operations/observability/mixins/meta/rules/components/server/alerts.libsonnet
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,8 +69,8 @@
{
alert: 'WebsocketConnectionRateHigh',
// Reasoning: the values are taken from past data
expr: 'sum(rate(server_websocket_connection_count[2m])) > 30',
'for': '5m',
expr: 'sum(rate(gitpod_server_api_connections_total[2m])) by (pod) > 5',
'for': '10m',
labels: {
// sent to the team internal channel until we fine tuned it
severity: 'warning',
Expand Down Expand Up @@ -116,22 +116,6 @@
description: 'Messagebus pod not running',
},
},
{
alert: 'WebAppServicesHighMemoryUsage',
// Reasoning: high rates of RAM consumption should only be temporary. Values based on past data (around 5-10 is constant)
expr: 'sum(rate(container_memory_working_set_bytes{container!="POD", node=~".*", pod=~"(server|ws-manager-bridge|usage)-.*"}[30m])) by (pod, node) > 10000000',
'for': '15m',
labels: {
// sent to the team internal channel until we fine tuned it
severity: 'warning',
team: 'webapp'
},
annotations: {
runbook_url: 'https://github.com/gitpod-io/runbooks/blob/main/runbooks/WebAppServicesHighMemoryUsage.md',
summary: 'WebApp services consume excessive amounts of memory. Investigation required.',
description: 'WebApp Services execcisve memory usage',
},
},
{
alert: 'WebAppServicesHighCPUUsage',
// Reasoning: high rates of CPU consumption should only be temporary.
Expand All @@ -150,8 +134,9 @@
},
{
alert: 'WebAppServicesCrashlooping',
expr: 'sum(rate(container_cpu_usage_seconds_total{container!="POD", node=~".*", pod=~"(content-service|dashboard|db|db-sync|messagebus|payment-endpoint|proxy|server|ws-manager-bridge|usage)-.*"}[5m])) by (pod, node) > 0.80',
'for': '15m',
// Reasoning: alert if any pod is restarting more than 3 times / 5 minutes.
expr: 'increase(kube_pod_container_status_restarts_total{container!="POD", pod=~"(content-service|dashboard|db|db-sync|messagebus|payment-endpoint|proxy|server|ws-manager-bridge|usage)-.*"}[5m]) > 3',
'for': '5m',
labels: {
// sent to the team internal channel until we fine tuned it
severity: 'warning',
Expand Down