Adds telemeter alert TelemeterClientFailures

Issue: https://issues.redhat.com/browse/MON-2727

Problem: in-cluster admins and folks monitoring submitted Insights should have a way to figure out that the cluster is trying and failing to submit Telemetry.

Solution: alert that will trigger when the rate of failed requests
reaches a total of 20% of the total rate of requests in a 15 min window

Author: JoaoBraveCoding

assets/telemeter-client/prometheus-rule.yaml

@@ -9,3 +9,21 @@ spec:
     rules:
     - expr: max(federate_samples - federate_filtered_samples)
       record: cluster:telemetry_selected_series:count
+    - alert: TelemeterClientFailures
+      annotations:
+        description: |-
+          The telemeter client in namespace {{ $labels.namespace }} fails {{ $value | humanize }} of the requests to the telemeter service.
+          Check the logs of the telemeter-client pod with the following command:
+          oc logs -n openshift-monitoring deployment.apps/telemeter-client -c telemeter-client
+          If the telemeter client fails to authenticate with the telemeter service, make sure that the global pull secret is up to date, see https://docs.openshift.com/container-platform/latest/openshift_images/managing_images/using-image-pull-secrets.html#images-update-global-pull-secret_using-image-pull-secrets for more details.
+        summary: Telemeter client fails to send metrics
+      expr: |
+        sum by (namespace) (
+          rate(federate_requests_failed_total{job="telemeter-client"}[15m])
+        ) /
+        sum by (namespace) (
+          rate(federate_requests_total{job="telemeter-client"}[15m])
+        ) > 0.2
+      for: 1h
+      labels:
+        severity: warning

jsonnet/jsonnetfile.lock.json

@@ -120,8 +120,8 @@
           "subdir": "jsonnet/telemeter"
         }
       },
-      "version": "320b9a967574c0a57690dea1987e1f294dbc22e5",
-      "sum": "jPX3JQZndZSVPDmkW2HZEib7/oeuVpxGOB/rXSgyOcI=",
+      "version": "4d304019274307c21afefa108493c8af89a2429d",
+      "sum": "079UoqPnQJWKoVi2qMsVUANGD0cBkx25D+S7guvrcGc=",
       "name": "telemeter-client"
     },
     {