Support for Custom Labels in kube_node_* Metrics

[nareshku]

What would you like to be added:
The ability to include custom labels such as node.k8s.io/pool in kube_node_* metrics.

Why is this needed:
This enhancement would enable tracking nodes as a group, making it easier to analyze and filter them based on custom attributes. Other similar use cases include labeling nodes with nvidia.com/gpu_type, etc.

With this change a node metric will look something like below:

kube_node_status_condition{node=<nodename>,label_node_k8s_io_pool=<name>,condition=<ReadyForPods>,status="true"} 1

Describe the solution you'd like
I attempted to achieve this using CustomResourceStateMetrics, but it didn't work as expected due to multiple node conditions being present. It might have worked if there were only a single node condition per node.

I also tried modifying the metric name by incorporating the condition type, e.g., kube__v1_Node_kube_node_status_condition_NetworkUnavailable, but this approach resulted in duplicate metrics.

Here's is an example

    kind: CustomResourceStateMetrics
    spec:
      resources:
        - groupVersionKind:
            group: ""
            kind: "Node"
            version: "v1"
          labelsFromPath:
            name: [metadata, name]
            label_node_k8s_io_pool: [metadata, labels, "node.k8s.io/pool"]
          metrics:
            - name: "kube_node_status_condition_NetworkUnavailable"
              help: "Node condition status with pool label"
              each:
                type: StateSet
                stateSet:
                  path: [status, conditions, "[type=NetworkUnavailable]"]
                  labelName: status
                  valueFrom: [status]
                  list: [True, False, Unknown]

The configuration resulted in duplicate metrics, as shown below:

kube__v1_Node_kube_node_status_condition_NetworkUnavailable{name="node1.example.com",label_node_k8s_io_pool="test",status="True"} 0
kube__v1_Node_kube_node_status_condition_NetworkUnavailable{name="node1.example.com",label_node_k8s_io_pool="test"status="Unknown"} 0
kube__v1_Node_kube_node_status_condition_NetworkUnavailable{name="node1.example.com",label_node_k8s_io_pool="test"status="False"} 1
kube__v1_Node_kube_node_status_condition_NetworkUnavailable{name="node1.example.com",label_node_k8s_io_pool="test"status="True"} 0
kube__v1_Node_kube_node_status_condition_NetworkUnavailable{name="node1.example.com",label_node_k8s_io_pool="test"status="Unknown"} 0

Additional context
Would appreciate guidance on the best approach to incorporate custom labels into kube_node_* metrics.

[richabanker]

/triage accepted
/assign

[nareshku]

@richabanker Please share your thoughts on this request. Also, what's the process for contributions? I can probably work on this.

[richabanker]

Hi @nareshku apologies for the delay on this, but I dont think Including the node.k8s.io/pool label is feasible (and was intentionally avoided) due to the potential for unbounded cardinality. Clusters can have an arbitrary number of nodepools, and exposing this as a metric label would lead to excessive resource consumption.

[nareshku]

Totally agree to this. However, can this be part of a configuration option instead?

[jceb]

@nareshku how did you manage to set group: "" despite issue #2434 not being fixed?