Skip to content

Latest commit

 

History

History
280 lines (236 loc) · 7.97 KB

consuming-metrics.md

File metadata and controls

280 lines (236 loc) · 7.97 KB

Consuming Metrics

!!! warning Metrics endpoints and ports are available as an alpha release and are subject to change in future versions. The following procedure is provided as an example for testing purposes. Do not depend on alpha features in production clusters.

In OLM v1, you can use the provided metrics with tools such as the Prometheus Operator. By default, Operator Controller and catalogd export metrics to the /metrics endpoint of each service.

You must grant the necessary permissions to access the metrics by using role-based access control (RBAC) polices. Because the metrics are exposed over HTTPS by default, you need valid certificates to use the metrics with services such as Prometheus. The following sections cover enabling metrics, validating access, and provide a reference of a ServiceMonitor to illustrate how you might integrate the metrics with the Prometheus Operator or other third-part solutions.

Enabling metrics for the Operator Controller

  1. To enable access to the Operator controller metrics, create a ClusterRoleBinding resource by running the following command:
kubectl create clusterrolebinding operator-controller-metrics-binding \
   --clusterrole=operator-controller-metrics-reader \
   --serviceaccount=olmv1-system:operator-controller-controller-manager

Validating Access Manually

  1. Generate a token for the service account and extract the required certificates:
TOKEN=$(kubectl create token operator-controller-controller-manager -n olmv1-system)
echo $TOKEN
  1. Apply the following YAML to deploy a pod in a namespace to consume the metrics:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: curl-metrics
  namespace: olmv1-system
spec:
  serviceAccountName: operator-controller-controller-manager
  containers:
  - name: curl
    image: curlimages/curl:latest
    command:
    - sh
    - -c
    - sleep 3600
    securityContext:
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      runAsUser: 1000
      runAsGroup: 1000
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
    volumeMounts:
    - mountPath: /tmp/cert
      name: olm-cert
      readOnly: true
  volumes:
  - name: olm-cert
    secret:
      secretName: olmv1-cert
  securityContext:
    runAsNonRoot: true
  restartPolicy: Never
EOF
  1. Access the pod:
kubectl exec -it curl-metrics -n olmv1-system -- sh
  1. Run the following command using the TOKEN value obtained above to check the metrics:
curl -v -k -H "Authorization: Bearer <TOKEN>" \
https://operator-controller-service.olmv1-system.svc.cluster.local:8443/metrics
  1. Run the following command to validate the certificates and token:
curl -v --cacert /tmp/cert/ca.crt --cert /tmp/cert/tls.crt --key /tmp/cert/tls.key \
-H "Authorization: Bearer <TOKEN>" \
https://operator-controller-service.olmv1-system.svc.cluster.local:8443/metrics

Enabling metrics for the Operator CatalogD

  1. To enable access to the CatalogD metrics, create a ClusterRoleBinding for the CatalogD service account:
kubectl create clusterrolebinding catalogd-metrics-binding \
   --clusterrole=catalogd-metrics-reader \
   --serviceaccount=olmv1-system:catalogd-controller-manager

Validating Access Manually

  1. Generate a token and get the required certificates:
TOKEN=$(kubectl create token catalogd-controller-manager -n olmv1-system)
echo $TOKEN
  1. Run the following command to obtain the name of the secret which store the certificates:
OLM_SECRET=$(kubectl get secret -n olmv1-system -o jsonpath="{.items[*].metadata.name}" | tr ' ' '\n' | grep '^catalogd-service-cert')
echo $OLM_SECRET
  1. Apply the following YAML to deploy a pod in a namespace to consume the metrics:
kubectl apply -f - <<EOF
apiVersion: v1
kind: Pod
metadata:
  name: curl-metrics-catalogd
  namespace: olmv1-system
spec:
  serviceAccountName: catalogd-controller-manager
  containers:
  - name: curl
    image: curlimages/curl:latest
    command:
    - sh
    - -c
    - sleep 3600
    securityContext:
      runAsNonRoot: true
      readOnlyRootFilesystem: true
      runAsUser: 1000
      runAsGroup: 1000
      allowPrivilegeEscalation: false
      capabilities:
        drop:
        - ALL
    volumeMounts:
    - mountPath: /tmp/cert
      name: catalogd-cert
      readOnly: true
  volumes:
  - name: catalogd-cert
    secret:
      secretName: $OLM_SECRET
  securityContext:
    runAsNonRoot: true
  restartPolicy: Never
EOF
  1. Access the pod:
kubectl exec -it curl-metrics-catalogd -n olmv1-system -- sh
  1. Run the following command using the TOKEN value obtained above to check the metrics:
curl -v -k -H "Authorization: Bearer <TOKEN>" \
https://catalogd-service.olmv1-system.svc.cluster.local:7443/metrics
  1. Run the following command to validate the certificates and token:
curl -v --cacert /tmp/cert/ca.crt --cert /tmp/cert/tls.crt --key /tmp/cert/tls.key \
-H "Authorization: Bearer <TOKEN>" \
https://catalogd-service.olmv1-system.svc.cluster.local:7443/metrics

Integrating the metrics endpoints with third-party solutions

In many cases, you must provide the certificates and the ServiceName resources to integrate metrics endpoints with third-party solutions. The following example illustrates how to create a ServiceMonitor resource to scrape metrics for the Prometheus Operator in OLM v1.

!!! note The following manifests are provided as a reference mainly to let you know how to configure the certificates. The following procedure is not a complete guide to configuring the Prometheus Operator or how to integrate within. To integrate with Prometheus Operator you might need to adjust your configuration settings, such as the serviceMonitorSelector resource, and the namespace where you apply the ServiceMonitor resource to ensure that metrics are properly scraped.

Example for Operator-Controller

kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: operator-controller-controller-manager
  name: controller-manager-metrics-monitor
  namespace: olmv1-system
spec:
  endpoints:
    - path: /metrics
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        insecureSkipVerify: false 
        serverName: operator-controller-service.olmv1-system.svc
        ca:
          secret:
            name: olmv1-cert
            key: ca.crt
        cert:
          secret:
            name: olmv1-cert
            key: tls.crt
        keySecret:
          name: olmv1-cert
          key: tls.key
  selector:
    matchLabels:
      control-plane: operator-controller-controller-manager
EOF

Example for CatalogD

OLM_SECRET=$(kubectl get secret -n olmv1-system -o jsonpath="{.items[*].metadata.name}" | tr ' ' '\n' | grep '^catalogd-service-cert')
echo $OLM_SECRET
kubectl apply -f - <<EOF
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    control-plane: catalogd-controller-manager
  name: catalogd-metrics-monitor
  namespace: olmv1-system
spec:
  endpoints:
    - path: /metrics
      port: https
      scheme: https
      bearerTokenFile: /var/run/secrets/kubernetes.io/serviceaccount/token
      tlsConfig:
        serverName: catalogd-service.olmv1-system.svc
        insecureSkipVerify: false
        ca:
          secret:
            name: $OLM_SECRET
            key: ca.crt
        cert:
          secret:
            name: $OLM_SECRET
            key: tls.crt
        keySecret:
          name: $OLM_SECRET
          key: tls.key
  selector:
    matchLabels:
      control-plane: catalogd-controller-manager
EOF