From 31f6aaddbf3d67ed0298042079c5fae4ec835b20 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Wed, 26 Jul 2017 11:34:01 -0700 Subject: [PATCH 1/5] [WIP] Proposal for Exposing Storage Metrics --- .../design-proposals/storage-metrics.md | 64 +++++++++++++++++++ 1 file changed, 64 insertions(+) create mode 100644 contributors/design-proposals/storage-metrics.md diff --git a/contributors/design-proposals/storage-metrics.md b/contributors/design-proposals/storage-metrics.md new file mode 100644 index 00000000000..8573c3a23a4 --- /dev/null +++ b/contributors/design-proposals/storage-metrics.md @@ -0,0 +1,64 @@ +# [WIP] Expose Storage Metrics to Users + +**Author**: Jing Xu (@jingxu97) + +**Last Updated**: 7/26/2017 + +**Status**: Proposal + +This document explores different ways of exposing storage metrics to users + +## Background and Motivation + +Monitoring resource usage is critical for users/admins to manage their systems. For storage, there are three levels of metrics: + - Node-level: the capacity, used, and allocatable storage on local host + - Pod-level: Pod volume could be network attached (GCE PD, AWS EBS etc.) or local (emptyDir, hostPath, or local storage volume). Each volume has a total capacity and used bytes + - Container-level: How much storage is used by each container’s writable layer and logs (standard input/output). + +All the above metrics are used for operation of Kubernetes internal components and core utilities as explains below. + + - Eviction manager checks the allocatable local storage (scratch space) and take actions. Scheduler also needs the node-level local storage capacity and availability to determine how to place pods on nodes + - In 1.7, an alpha feature is added to isolation emptyDir volume usage and container’s overlay in local storage. User can set a sizeLimit for Pod EmptyDir Volume and eviction manager will take action if the emptyDir volume usage exceeds the configured size. Similarly, user can set request/limit on container’s overlay and eviction manager can take actions based on the monitoring data and the limits. + - There is also a proposal targeting for 1.8 [volume resize proposal](https://github.com/kubernetes/community/pull/657) to dynamically resize the volume (network attached) if the disk resource is insufficient. It requires a monitoring pipeline to dynamically check the the disk usage for each volume (represented by PVC) + +### Current Monitoring Options + +There are a number of monitoring options supported by Kubernetes + - Resource Metrics API (Alpha feature in progress [Resource Metrics API](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/resource-metrics-api.md) + - Collect metrics in Heapster. (metrics has to exposed by Kubelet in Summary API) + - Export metrics through StackDriver + - Kubectl Command (kubectl top, kubectl describe) + +All the storage metrics mentioned in Background are collected by Kubelet Summary API so that it could be integrated with the above monitoring system. However, pod volume metrics is provided in PodStat object’s VolumeStat and VolumeStat only has volume name specified in pod specification. There is no direct way of checking volume usage information by PVC name, which is more preferable by users. + + +## Goals +The gola of this proposal is to expose storage usage metrics to users for monitoring their storage systems. + +## Proposal + +The first part of this proposal is to add volume usage information indexed by PVC name in Kubelet Summary API + +### volume usage information indexed by PVC + +The basic idea is to cache PVC and the volume information in kubelet volume manager which is similar to caching the pod and volume information. In Summary API, we could add new API type PVCStats which is similar to NodeStats and PodStats + +``` +// PVCStats holds pod-level unprocessed sample volume stats refered by PVC. +type PVCStats struct { + // Reference to the measured PVC. + PVCRef PVCReference `json:"podRef"` + // The time at which data collection for the PVC-scoped volume stats was (re)started. + StartTime metav1.Time `json:"startTime"` + // Stats pertaining to volume usage of filesystem resources. + // VolumeStats.UsedBytes is the number of bytes used by the Volume + // +optional + // +patchMergeKey=name + // +patchStrategy=merge + VolumeStats VolumeStats `json:"volume,omitempty" patchStrategy:"merge" patchMergeKey:"name"` +} +``` + +## Implementation Timeline: +The feature is targeted for kubernetes v1.8 + From ec8476c73b837ad5e381eb1e13032ae40a1a6802 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Wed, 2 Aug 2017 10:30:53 -0700 Subject: [PATCH 2/5] Update storage-metrics.md --- .../design-proposals/storage-metrics.md | 51 ++++++++++++++----- 1 file changed, 39 insertions(+), 12 deletions(-) diff --git a/contributors/design-proposals/storage-metrics.md b/contributors/design-proposals/storage-metrics.md index 8573c3a23a4..a3c51d27377 100644 --- a/contributors/design-proposals/storage-metrics.md +++ b/contributors/design-proposals/storage-metrics.md @@ -37,27 +37,54 @@ The gola of this proposal is to expose storage usage metrics to users for monito ## Proposal -The first part of this proposal is to add volume usage information indexed by PVC name in Kubelet Summary API +This proposal has three parts, the first part is trying to address the issue of adding volume usage information indexed by PVC and PV name in Kubelet Summary API. The second part is to register the volume metrics to . The third part is to add more storage metrics to Heapster. -### volume usage information indexed by PVC -The basic idea is to cache PVC and the volume information in kubelet volume manager which is similar to caching the pod and volume information. In Summary API, we could add new API type PVCStats which is similar to NodeStats and PodStats +### volume usage information indexed by PVC/PV in Kubelet Summary API + +The basic idea is to cache PVC and the volume information in kubelet volume manager which is similar to caching the pod and volume information. In Summary API, In Summary API, we could add PVC and PV references into VolumeStats which is included in PodStats. ``` -// PVCStats holds pod-level unprocessed sample volume stats refered by PVC. -type PVCStats struct { +// VolumeStats contains data about Volume filesystem usage. +type VolumeStats struct { // Reference to the measured PVC. PVCRef PVCReference `json:"podRef"` - // The time at which data collection for the PVC-scoped volume stats was (re)started. - StartTime metav1.Time `json:"startTime"` - // Stats pertaining to volume usage of filesystem resources. - // VolumeStats.UsedBytes is the number of bytes used by the Volume +// Reference to the measured PV. + PVRef PVReference `json:"podRef"` +// Embedded FsStats + FsStats +// Name is the name given to the Volume // +optional - // +patchMergeKey=name - // +patchStrategy=merge - VolumeStats VolumeStats `json:"volume,omitempty" patchStrategy:"merge" patchMergeKey:"name"` + Name string `json:"name,omitempty"` +} + +// PVCReference contains enough information to locate the referenced PVC. +type PodReference struct { + Name string `json:"name"` + Namespace string `json:"namespace"` + UID string `json:"uid"` +} + +// PVReference contains enough information to locate the referenced PV. +type PodReference struct { + Name string `json:"name"` + UID string `json:"uid"` } + ``` +### Register the VolumeStats metrics to Prometheus + +The following metrics could be registered to Prometheus + + +| Metric name | Metric type | Labels/tags | +|-------------|-------------|-------------| +| volume_stats_capacityBytes | Gauge | namespace=\
persistentvolumeclaim=\
persistentvolume=\ | +| volume_stats_usedBytes | Gauge | namespace=\
persistentvolumeclaim=\
persistentvolume=\ | +| volume_stats_availableBytes | Gauge | namespace=\
persistentvolumeclaim=\
persistentvolume= | +| volume_stats_InodesFree | Gauge | nnamespace=\
persistentvolumeclaim=\
persistentvolume=\ | +| volume_stats_Inodes | Gauge | namespace=\
persistentvolumeclaim=\
persistentvolume=\ | +| volume_stats_InodesUsed | Gauge | namespace=\
persistentvolumeclaim=\
persistentvolume=\ | ## Implementation Timeline: The feature is targeted for kubernetes v1.8 From bc90c9d56035a428ae0d15935eb1a7ed4417c0ca Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Wed, 2 Aug 2017 10:31:39 -0700 Subject: [PATCH 3/5] Update storage-metrics.md --- contributors/design-proposals/storage-metrics.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/contributors/design-proposals/storage-metrics.md b/contributors/design-proposals/storage-metrics.md index a3c51d27377..735c2a88b91 100644 --- a/contributors/design-proposals/storage-metrics.md +++ b/contributors/design-proposals/storage-metrics.md @@ -49,11 +49,11 @@ The basic idea is to cache PVC and the volume information in kubelet volume mana type VolumeStats struct { // Reference to the measured PVC. PVCRef PVCReference `json:"podRef"` -// Reference to the measured PV. + // Reference to the measured PV. PVRef PVReference `json:"podRef"` -// Embedded FsStats + // Embedded FsStats FsStats -// Name is the name given to the Volume + // Name is the name given to the Volume // +optional Name string `json:"name,omitempty"` } From 09d3d245c70d3056da6630e7ea732eb4b0c12295 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Wed, 16 Aug 2017 11:07:16 -0700 Subject: [PATCH 4/5] Update storage-metrics.md --- contributors/design-proposals/storage-metrics.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/contributors/design-proposals/storage-metrics.md b/contributors/design-proposals/storage-metrics.md index 735c2a88b91..9df74b517db 100644 --- a/contributors/design-proposals/storage-metrics.md +++ b/contributors/design-proposals/storage-metrics.md @@ -59,14 +59,14 @@ type VolumeStats struct { } // PVCReference contains enough information to locate the referenced PVC. -type PodReference struct { +type PVCReference struct { Name string `json:"name"` Namespace string `json:"namespace"` UID string `json:"uid"` } // PVReference contains enough information to locate the referenced PV. -type PodReference struct { +type PVReference struct { Name string `json:"name"` UID string `json:"uid"` } From 895e25f6a3c82790734fc3aa39f72df529d60af4 Mon Sep 17 00:00:00 2001 From: Jing Xu Date: Fri, 8 Sep 2017 10:02:18 -0700 Subject: [PATCH 5/5] Update storage-metrics.md --- contributors/design-proposals/storage-metrics.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/contributors/design-proposals/storage-metrics.md b/contributors/design-proposals/storage-metrics.md index 9df74b517db..fd4a83cb106 100644 --- a/contributors/design-proposals/storage-metrics.md +++ b/contributors/design-proposals/storage-metrics.md @@ -1,4 +1,4 @@ -# [WIP] Expose Storage Metrics to Users +# Expose Storage Metrics to Users **Author**: Jing Xu (@jingxu97)