Metric kube_pod_status_reason returns hundreds of results, but all with a zero value

[jgagnon44]

What happened:
Running a PromQL query on the kube_pod_status_reason metric returns hundreds of result records, but every one has a value of zero. This occurs even with pods in failed or pending status.

What you expected to happen:
I would expect for cases where any pods in failed or pending status to have a result from this metric with a non-zero value.

How to reproduce it (as minimally and precisely as possible):
Run a query in Prometheus on kube_pod_status_reason. No filtering is needed. I get hundreds of result entries, every one with a value of zero.

If you run the query slightly differently: kube_pod_status_reason > 0 you will get an empty response, even with pods in failed or pending status.

Anything else we need to know?:

Environment:

• kube-state-metrics version:

Not sure how to find this.

I don't know if this is equivalent. I inspected the pod and found the following:

  containers:
    - name: kube-state-metrics
      image: k8s.gcr.io/kube-state-metrics/kube-state-metrics:v2.0.0

• Kubernetes version (use kubectl version):

WARNING: This version information is deprecated and will be replaced with the output from kubectl version --short.  Use --output=yaml|json to get the full version.
Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.2", GitCommit:"f66044f4361b9f1f96f0053dd46cb7dce5e990a8", GitTreeState:"clean", BuildDate:"2022-06-15T14:22:29Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"windows/amd64"}
Kustomize Version: v4.5.4
Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.9", GitCommit:"9710807c82740b9799453677c977758becf0acbb", GitTreeState:"clean", BuildDate:"2022-12-08T10:08:06Z", GoVersion:"go1.18.9", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

Everything in on premises.

• Other info:

[CatherineF-dev]

Could you curl kube-state-metrics endpoint to get full metrics payload?

For example,

# change kube-system to correct namespace where kube-state-metrics exists
kubectl -n kube-system port-forward pod/kube-state-metrics-xxx 9443:9443

curl  http://localhost:9443/metrics | grep kube_pod_status_reason

[jgagnon44]

Just discovered that the kube-state-metrics pod is running in the lens-metrics namespace. Not sure what that means. Was expecting to see it in the namespace you mentioned.

In any case, tried to do as you suggested, but ran into a problem. I ran the port-forward command and then in a separate terminal entered the curl command. The curl command returns an empty reply and when I switched back to the other terminal, the port-forward had exited with an error.

$ k -n lens-metrics port-forward kube-state-metrics-775bd8fd9f-r5qgv 9443:9443
Forwarding from 127.0.0.1:9443 -> 9443
Forwarding from [::1]:9443 -> 9443
Handling connection for 9443
E0712 06:53:14.074707   38060 portforward.go:406] an error occurred forwarding 9443 -> 9443: error forwarding port 9443 to pod 45c511ca7caaa67f29593a81f9f974e8f5bcbc078d66216fd14ff9df01de5bf2, uid : port forward into network namespace "/var/run/netns/3ba1b4c7-1e76-4115-8d89-e98f8c3e3ec3": failed to connect to localhost:9443 inside namespace 45c511ca7caaa67f29593a81f9f974e8f5bcbc078d66216fd14ff9df01de5bf2: dial tcp 127.0.0.1:9443: connect: connection refused
E0712 06:53:14.076274   38060 portforward.go:234] lost connection to pod

I tried something else. In Lens, I accessed the pod and forwarded the port from there. It allowed me to open in a browser, whereupon I was able to access a metrics link. Clicking on this presented me with a page with tons of information. I did a search for the metric and found over 250 entries, each with a value zero.

Here's a sample:

kube_pod_status_reason{namespace="prometheus",pod="prometheus-prometheus-kube-prometheus-prometheus-0",reason="NodeLost"} 0
kube_pod_status_reason{namespace="prometheus",pod="prometheus-prometheus-kube-prometheus-prometheus-0",reason="Evicted"} 0
kube_pod_status_reason{namespace="prometheus",pod="prometheus-prometheus-kube-prometheus-prometheus-0",reason="UnexpectedAdmissionError"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28146870-w2nbx",reason="NodeLost"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28146870-w2nbx",reason="Evicted"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28146870-w2nbx",reason="UnexpectedAdmissionError"} 0
kube_pod_status_reason{namespace="kubeapps",pod="kubeapps-7db5f76cb5-qt2jq",reason="NodeLost"} 0
kube_pod_status_reason{namespace="kubeapps",pod="kubeapps-7db5f76cb5-qt2jq",reason="Evicted"} 0
kube_pod_status_reason{namespace="kubeapps",pod="kubeapps-7db5f76cb5-qt2jq",reason="UnexpectedAdmissionError"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28148310-sk282",reason="NodeLost"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28148310-sk282",reason="Evicted"} 0
kube_pod_status_reason{namespace="deploy",pod="clean-deploy-cronjob-28148310-sk282",reason="UnexpectedAdmissionError"} 0

[CatherineF-dev]

Could you list pods which are in failed or pending status? Also list the reasons.

It might be because current implementations only include these three reasons {NodeLost, Evicted, UnexpectedAdmissionError}.

[jgagnon44]

This is interesting. If I query this metric in Prometheus, I get 5 result entries per pod. If I port forward and access the kube-state-metrics /metrics, I'm only getting 3 result entries per pod. Here's an example:

From kube-state-metrics:

kube_pod_status_reason{namespace="ingress-nginx",pod="ingress-nginx-controller-f5rmk",reason="NodeLost"} 0
kube_pod_status_reason{namespace="ingress-nginx",pod="ingress-nginx-controller-f5rmk",reason="Evicted"} 0
kube_pod_status_reason{namespace="ingress-nginx",pod="ingress-nginx-controller-f5rmk",reason="UnexpectedAdmissionError"} 0

From Prometheus:

kube_pod_status_reason{container="kube-state-metrics", endpoint="http", instance="10.244.211.138:8080", job="kube-state-metrics", namespace="ingress-nginx", pod="ingress-nginx-controller-f5rmk", reason="Evicted", service="prometheus-kube-state-metrics", uid="29cd7cc2-86d4-4a9d-854e-1e122af3403b"} | 0
kube_pod_status_reason{container="kube-state-metrics", endpoint="http", instance="10.244.211.138:8080", job="kube-state-metrics", namespace="ingress-nginx", pod="ingress-nginx-controller-f5rmk", reason="NodeAffinity", service="prometheus-kube-state-metrics", uid="29cd7cc2-86d4-4a9d-854e-1e122af3403b"} | 0
kube_pod_status_reason{container="kube-state-metrics", endpoint="http", instance="10.244.211.138:8080", job="kube-state-metrics", namespace="ingress-nginx", pod="ingress-nginx-controller-f5rmk", reason="NodeLost", service="prometheus-kube-state-metrics", uid="29cd7cc2-86d4-4a9d-854e-1e122af3403b"} | 0
kube_pod_status_reason{container="kube-state-metrics", endpoint="http", instance="10.244.211.138:8080", job="kube-state-metrics", namespace="ingress-nginx", pod="ingress-nginx-controller-f5rmk", reason="Shutdown", service="prometheus-kube-state-metrics", uid="29cd7cc2-86d4-4a9d-854e-1e122af3403b"} | 0
kube_pod_status_reason{container="kube-state-metrics", endpoint="http", instance="10.244.211.138:8080", job="kube-state-metrics", namespace="ingress-nginx", pod="ingress-nginx-controller-f5rmk", reason="UnexpectedAdmissionError", service="prometheus-kube-state-metrics", uid="29cd7cc2-86d4-4a9d-854e-1e122af3403b"} | 0

Why would I get a different set of results?

[CatherineF-dev]

fyi: Pod metrics are implemented in this file https://github.com/kubernetes/kube-state-metrics/blob/main/internal/store/pod.go

You can have a look at this file to figure out what happened.

I am curious on whether kube_pod_status_reason didn't reports failed pods.

[jgagnon44]

Here are the current failed or pending pods on the cluster:

Pod	Namespace	Status	Reason
runner-goew1uzh-project-8-concurrent-0zk79h	runner-workspace-writer	Failed	ContainerStatusUnknown - exit code: 137
runner-goew1uzh-project-8-concurrent-1fhtb4	runner-workspace-writer	Failed	ContainerStatusUnknown - exit code: 137
clean-deploy-cronjob-28151190-fqqgs	deploy	Pending	Back-off pulling image "harbor.hulk.beast-code.com/phactory-images/cleanup-deploy:master"
clean-deploy-cronjob-28152630-jzrsj	deploy	Pending	Back-off pulling image "harbor.hulk.beast-code.com/phactory-images/cleanup-deploy:master"

[CatherineF-dev]

I found the reason. It's because using bounded reasons.

podStatusReasons           = []string{"Evicted", "NodeAffinity", "NodeLost", "Shutdown", "UnexpectedAdmissionError"}

https://github.com/kubernetes/kube-state-metrics/blob/main/internal/store/pod.go#L40

[jgagnon44]

Is it possible to reconfigure this to be non constrained?

[CatherineF-dev]

Could you try updating
https://github.com/kubernetes/kube-state-metrics/blob/main/internal/store/pod.go and rebuilding a new kube-state-metrics?

Wondering what's the reason for Back-off pulling image "harbor.hulk.beast-code.com/phactory-images/cleanup-deploy:master"

[CatherineF-dev]

Or could you run kubectl -n deploy get pod clean-deploy-cronjob-28152630-jzrsj -o yaml

[jgagnon44]

I'm guessing this is unrelated. In any case:

  containerStatuses:
  - image: harbor.hulk.beast-code.com/phactory-images/cleanup-deploy:master
    imageID: ""
    lastState: {}
    name: clean-deploy
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        message: Back-off pulling image "harbor.hulk.beast-code.com/phactory-images/cleanup-deploy:master"
        reason: ImagePullBackOff

[jgagnon44]

Not sure how to attempt what you suggest RE updating the .go file and rebuilding. Also, I don't know Go and am not sure what can be done to make it do a wildcard loop where the variable is used:

kube-state-metrics/internal/store/pod.go

Line 1475 in bb6e9f4

for _, reason := range podStatusReasons {

I guess a more important question is: why is it being constrained in the first place?

[CatherineF-dev]

I guess a more important question is: why is it being constrained in the first place?

Not sure all the reasons. One reason should be constraining metrics cardinality.

I think we can add ImagePullBackOff into the list in master branch. Could you list reason forrunner-goew1uzh-project-8-concurrent-0zk79h as well?

kubectl -n runner-workspace-writer get pod runner-goew1uzh-project-8-concurrent-0zk79h -o yaml