Network latency SLI

Author: wojtek-t

sig-scalability/slos/network_latency.md

@@ -0,0 +1,49 @@
+## In-cluster network latency SLIs/SLOs details
+
+### Definition
+
+| Status | SLI | SLO |
+| --- | --- | --- |
+| __WIP__ | In-cluster network latency from a single node, measured as latency of per second ping from node to "null service", measured as 99th percentile over last 5 minutes. | In default Kubernetes installataion with RTT between nodes <= Y, 99th percentile of (99th percentile over all nodes) per cluster-day <= X |
+
+### User stories
+- As a user of vanilla Kubernetes, I want some guarantee how fast my http
+request to some Kubernetes service reaches its endpoint
+
+### Other notes
+- We obviously can't give any guarantee in a general case, because cluster
+administrators may configure cluster as they want.
+- As a result, we define the SLI to be very generic (no matter how your cluster
+is set up), but we provide SLO only for default installations with an additional
+requirement that low-level RTT between nodes is lower than Y.
+- Network latency is one of the most crucial aspects from the point of view
+of application performance, especially in microservices world. As a result, to
+meet user expectations, we need to provide some guarantees arount that.
+- We decided for the SLI definition as formulated above, because:
+  - it represents a user oriented end-to-end flow - it involves among others
+    latency of in-cluster network programming mechanism (e.g. iptables).
+    __TODO:__ We considered making DNS resolution part of it, but decided not
+    to mix them. However, longer term we should consider joining them.
+  - it is easily measurable in all running clusters without any significant
+    overhead (e.g. measuring request latencies coming from all pods on a given
+    node would require some additional instrumentation, such as a side car for
+    each of them, and that overhead may be not acceptable in many cases)
+  - it is not application-specific
+
+### Caveats
+- The SLI is formulated for a single node, even though users are mostly
+interested in the aggregation across all nodes (that is done only at the SLO
+level). However, that provides very similar guarantees and makes it fairly
+easy to measure.
+- The prober reporting that is fairly trivial and itself needs only negligible
+amount of resources. However, to avoid any visible overhead in the cluster
+(in terms of additionally needed components) **we will make it part of kube-proxy
+(which is running on every node anyway) and make it possible to switch it off
+by via config option (potentially also a flag)**.
+- We don't have any "null service" running in cluster, so an administrator has
+to set up one to make the SLI measurable in real cluster. We will provide
+an appropriate "null service" in our tests.
+
+### Test scenario
+
+__TODO: Describe test scenario.__

sig-scalability/slos/slos.md

@@ -107,6 +107,7 @@ Prerequisite: Kubernetes cluster is available and serving.
 | __Official__ | Latency of non-streaming read-only API calls for every (resource, scope pair, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, for every (resource, scope) pair, excluding virtual and aggregated resources and Custom Resource Definitions, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> (a) <= 1s if `scope=resource` (b) <= 5s if `scope=namespace` (c) <= 30s if `scope=cluster` | [Details](./api_call_latency.md) |
 | __Official__ | Startup latency of stateless and schedulable pods, excluding time to pull images and run init containers, measured from pod creation timestamp to when all its containers are reported as started and observed via watch, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile per cluster-day<sup>[1](#footnote1)</sup> <= 5s | [Details](./pod_startup_latency.md) |
 | __WIP__ | Latency of programming a single (e.g. iptables on a given node) in-cluster load balancing mechanism, measured from when service spec or list of its `Ready` pods change to when it is reflected in load balancing mechanism, measured as 99th percentile over last 5 minutes | In default Kubernetes installation, 99th percentile of (99th percentiles across all programmers (e.g. iptables)) per cluster-day<sup>[1](#footnote1)</sup> <= X | [Details](./networking_programming_latency.md) |
+| __WIP__ | In-cluster network latency from a single node, measured as latency of per second ping from node to "null service", measured as 99th percentile over last 5 minutes. | In default Kubernetes installataion with RTT between nodes <= Y, 99th percentile of (99th percentile over all nodes) per cluster-day <= X | [Details](./network_latency.md)
 
 <a name="footnote1">\[1\]</a> For the purpose of visualization it will be a
 sliding window. However, for the purpose of reporting the SLO, it means one