-
Notifications
You must be signed in to change notification settings - Fork 2.1k
fix: report correct reason in kube_pod_status_reason metric #2644
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -1541,15 +1541,12 @@ func createPodStatusReasonFamilyGenerator() generator.FamilyGenerator { | |
ms := []*metric.Metric{} | ||
|
||
for _, reason := range podStatusReasons { | ||
metric := &metric.Metric{} | ||
metric.LabelKeys = []string{"reason"} | ||
metric.LabelValues = []string{reason} | ||
if p.Status.Reason == reason { | ||
metric.Value = boolFloat64(true) | ||
} else { | ||
metric.Value = boolFloat64(false) | ||
m := &metric.Metric{ | ||
LabelKeys: []string{"reason"}, | ||
LabelValues: []string{reason}, | ||
Value: getPodStatusReasonValue(p, reason), | ||
} | ||
ms = append(ms, metric) | ||
ms = append(ms, m) | ||
} | ||
|
||
return &metric.Family{ | ||
|
@@ -1559,6 +1556,23 @@ func createPodStatusReasonFamilyGenerator() generator.FamilyGenerator { | |
) | ||
} | ||
|
||
func getPodStatusReasonValue(p *v1.Pod, reason string) float64 { | ||
if p.Status.Reason == reason { | ||
return 1 | ||
} | ||
for _, cond := range p.Status.Conditions { | ||
if cond.Reason == reason { | ||
return 1 | ||
} | ||
} | ||
Comment on lines
+1563
to
+1567
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should we only care about the last condition? If so, do we need to remove this part? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. No, it's necessary to iterate through all the conditions because the reason may be in any of them. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will it be a stale condition? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. it will not be a stale condition. Kubernetes regularly updates Pod conditions, so if a condition with the corresponding reason is found, it is assumed to be current. If a stale condition were detected, that would indicate an issue in Kubernetes, not in this logic. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Will a pod have multiple different reasons? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, a Pod can have different “Reasons” throughout its lifecycle. Each event or change in the Pod’s state (for example, container creation, image pulling, runtime errors, restarts, etc.) can trigger a different reason. In Kubernetes, these “Reasons” are recorded at different points in the Pod’s lifecycle, so it is entirely possible for a single Pod to go through multiple different “Reasons” as it transitions between states. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, I was thinking the case where the pod status is failed to image, then runtime errors, then restart. Will the above metric have all of these three status? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, if your Pod transitions through those states (e.g., failed to pull image, runtime errors, then restarts), the metric can capture each corresponding reason at the time it occurs. However, you won’t necessarily see all reasons simultaneously; rather, you’ll see them reflected as changes in the metric over the Pod’s lifecycle. |
||
for _, cs := range p.Status.ContainerStatuses { | ||
if cs.State.Terminated != nil && cs.State.Terminated.Reason == reason { | ||
return 1 | ||
} | ||
} | ||
return 0 | ||
} | ||
|
||
func createPodStatusScheduledFamilyGenerator() generator.FamilyGenerator { | ||
return *generator.NewFamilyGeneratorWithStability( | ||
"kube_pod_status_scheduled", | ||
|
Uh oh!
There was an error while loading. Please reload this page.