Skip to content

[kube-prometheus-stack] API-Server Dashboard fails to show panels if their query is using metrics with missing data #5407

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
mmueller-rs opened this issue Mar 6, 2025 · 2 comments · Fixed by kubernetes-monitoring/kubernetes-mixin#1038 · May be fixed by #5415
Labels
bug Something isn't working

Comments

@mmueller-rs
Copy link

mmueller-rs commented Mar 6, 2025

Describe the bug a clear and concise description of what the bug is.

We're currently in the process of upgrading the Prometheus-Stack to the latest version, but in our test environment, we noticed a few issues with the API-Server Monitoring.
In the Dashboard "Kubernetes / API server", one can see that several Panels show no data. While the "Write Availability (30d)" works fine, the following panels show "No Data":

  • Availability (30d) > 99.000%
  • ErrorBudget (30d) > 99.000%
  • Read Availability (30d)

In my tests, I noticed that for example the Read Availability works if I add the or vector(0) fallback to the sums:
Example Grafana Explore View (you need to have a Port-forward to Grafana on your local Port 3000, i.e. via kubectl port-forward svc/prometheus-grafana 3000:80)

What's your helm version?

version.BuildInfo{Version:"v3.17.1", GitCommit:"980d8ac1939e39138101364400756af2bdee1da5", GitTreeState:"clean", GoVersion:"go1.23.6"}

What's your kubectl version?

Client Version: v1.32.2 Kustomize Version: v5.5.0 Server Version: v1.32.2

Which chart?

kube-prometheus-stack

What's the chart version?

69.7.3

What happened?

After upgrading to the latest version of kube-prometheus-stack (69.7.3), the API-Server Availability Dashboard is broken. Several Panels show "No Data" when they should show an availability of up to 100%.

What you expected to happen?

The Panels should show an availability of 100% or less.

How to reproduce it?

  1. Install the kube-prometheus-stack via Helm
  2. Open Grafana:
    1. kubectl port-forward svc/prometheus-grafana 3000:80
    2. Open with Browser: http://localhost:3000
    3. Log in using default credentials (admin - prom-operator)
    4. Open the Dashboard "Kubernetes / API server"
    5. Wait a few minutes for Prometheus to gather data and calculate the rules

Enter the changed values of values.yaml?

NONE

Enter the command that you execute and failing/misfunctioning.

The broken Panels use the Prometheus-Recording-Rules for apiserver_request:availability30d with the verbs read and all.

Anything else we need to know?

Background: After upgrading from kube-prometheus-stack from 62.6.0 to 69.7.2, we've noticed alerts about alertname: KubeAPIErrorBudgetBurn. Checking out the API-Server Dashboard revealed that the availability Panels showed more than 100% (i.e. 5000% and growing).
As the Upgrade to 68 has updated the behaviour of different API-Server metrics, we've used the Admin-API to delete older Metrics. But then some panels could not be shown anymore.
The same issue can be reproduced in a freshly installed Installation of the Kube-Prometheus-Stack.

Edit: Improve comprehensibility of Link "Example Grafana Explore View"

@krankkkk
Copy link

Seems related to #5274

@mmueller-rs
Copy link
Author

@krankkkk thanks, you're right. Don't know how I could have overseen that.
I'll close this issue in favor of #5274

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
2 participants