MON-3134: allow to query alerts from thanos-querier tenancy port

[slashpai]

Allows to query alerts of application
namespaces as an application user.

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

[openshift-ci-robot]

@slashpai: This pull request references MON-3134 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.15.0" version, but no target version was set.

In response to this:

Allows to query alerts and rules of application
namespaces as an application user.

Tested locally by port forwarding the tenancy port

From terminal 1

oc port-forward svc/thanos-querier 9092

From terminal 2
Following https://docs.openshift.com/container-platform/4.14/monitoring/accessing-third-party-monitoring-apis.html

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/alerts?namespace=openshift-monitoring"
{"status":"success","data":{"alerts":[{"labels":{"alertname":"Watchdog","namespace":"openshift-monitoring","severity":"none"},"annotations":{"description":"This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n","summary":"An alert that should always be firing to certify that Alertmanager is working properly."},"state":"firing","activeAt":"2023-12-06T04:43:53.913490184Z","value":"1e+00"},{"labels":{"alertname":"AlertmanagerReceiversNotConfigured","namespace":"openshift-monitoring","severity":"warning"},"annotations":{"description":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.","summary":"Receivers (notification integrations) are not configured on Alertmanager"},"state":"firing","activeAt":"2023-12-06T04:44:43.013474132Z","value":"0e+00"}]}}

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/alerts"
Bad Request. The request or configuration is malformed.

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/rules"
Bad Request. The request or configuration is malformed.

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/rules?namespace=openshift-monitoring"
{"status":"success","data":{"groups":[{"name":"general.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-cd59ec39-eacd-46ba-97a8-7c47dd8249fb.yaml","rules":[{"name":"Watchdog","query":"vector(1)","duration":0,"labels":{"namespace":"openshift-monitoring","prometheus":"openshift-monitoring/k8s","severity":"none"},"annotations":{"description":"This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n","summary":"An alert that should always be firing to certify that Alertmanager is working properly."},"alerts":[{"labels":{"alertname":"Watchdog","namespace":"openshift-monitoring","severity":"none"},"annotations":{"description":"This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n","summary":"An alert that should always be firing to certify that Alertmanager is working properly."},"state":"firing","activeAt":"2023-12-06T04:43:53.913490184Z","value":"1e+00"}],"health":"ok","type":"alerting"}],"interval":30},{"name":"openshift-kubernetes.rules","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-cluster-monitoring-operator-prometheus-rules-cd59ec39-eacd-46ba-97a8-7c47dd8249fb.yaml","rules":[{"name":"AlertmanagerReceiversNotConfigured","query":"cluster:alertmanager_integrations:max == 0","duration":600,"labels":{"namespace":"openshift-monitoring","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.","summary":"Receivers (notification integrations) are not configured on Alertmanager"},"alerts":[{"labels":{"alertname":"AlertmanagerReceiversNotConfigured","namespace":"openshift-monitoring","severity":"warning"},"annotations":{"description":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.","summary":"Receivers (notification integrations) are not configured on Alertmanager"},"state":"firing","activeAt":"2023-12-06T04:44:13.013474132Z","value":"0e+00"}],"health":"ok","type":"alerting"}],"interval":30},{"name":"kube-state-metrics","file":"/etc/prometheus/rules/prometheus-k8s-rulefiles-0/openshift-monitoring-kube-state-metrics-rules-f602ee94-ef44-4af1-b101-2b226d14f7f7.yaml","rules":[{"name":"KubeStateMetricsListErrors","query":"(sum by (cluster) (rate(kube_state_metrics_list_total{job=\"kube-state-metrics\",result=\"error\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_list_total{job=\"kube-state-metrics\"}[5m]))) \u003e 0.01","duration":900,"labels":{"namespace":"openshift-monitoring","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"kube-state-metrics is experiencing errors at an elevated rate in list operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.","summary":"kube-state-metrics is experiencing errors in list operations."},"alerts":[],"health":"ok","type":"alerting"},{"name":"KubeStateMetricsWatchErrors","query":"(sum by (cluster) (rate(kube_state_metrics_watch_total{job=\"kube-state-metrics\",result=\"error\"}[5m])) / sum by (cluster) (rate(kube_state_metrics_watch_total{job=\"kube-state-metrics\"}[5m]))) \u003e 0.01","duration":900,"labels":{"namespace":"openshift-monitoring","prometheus":"openshift-monitoring/k8s","severity":"warning"},"annotations":{"description":"kube-state-metrics is experiencing errors at an elevated rate in watch operations. This is likely causing it to not be able to expose metrics about Kubernetes objects correctly or at all.","summary":"kube-state-metrics is experiencing errors in watch operations."},"alerts":[],"health":"ok","type":"alerting"}],"interval":30}]}}

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci-robot]

@slashpai: This pull request references MON-3134 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.15.0" version, but no target version was set.

In response to this:

Allows to query alerts of application
namespaces as an application user.

Tested locally by port forwarding the tenancy port

From terminal 1

oc port-forward svc/thanos-querier 9092

From terminal 2
Following https://docs.openshift.com/container-platform/4.14/monitoring/accessing-third-party-monitoring-apis.html

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/alerts?namespace=openshift-monitoring"
{"status":"success","data":{"alerts":[{"labels":{"alertname":"Watchdog","namespace":"openshift-monitoring","severity":"none"},"annotations":{"description":"This is an alert meant to ensure that the entire alerting pipeline is functional.\nThis alert is always firing, therefore it should always be firing in Alertmanager\nand always fire against a receiver. There are integrations with various notification\nmechanisms that send a notification when this alert is not firing. For example the\n\"DeadMansSnitch\" integration in PagerDuty.\n","summary":"An alert that should always be firing to certify that Alertmanager is working properly."},"state":"firing","activeAt":"2023-12-06T04:43:53.913490184Z","value":"1e+00"},{"labels":{"alertname":"AlertmanagerReceiversNotConfigured","namespace":"openshift-monitoring","severity":"warning"},"annotations":{"description":"Alerts are not configured to be sent to a notification system, meaning that you may not be notified in a timely fashion when important failures occur. Check the OpenShift documentation to learn how to configure notifications with Alertmanager.","summary":"Receivers (notification integrations) are not configured on Alertmanager"},"state":"firing","activeAt":"2023-12-06T04:44:43.013474132Z","value":"0e+00"}]}}

░▒▓    ~  curl -G -s -k -H "Authorization: Bearer $token"  "https://localhost:9092/api/v1/alerts"
Bad Request. The request or configuration is malformed.

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[slashpai]

cc: @juzhao

[simonpasquier]

/hold

/api/v1/alerts should be added to the kube-rbac-proxy-rules container.

[slashpai]

@simonpasquier addressed comments

[openshift-ci-robot]

@slashpai: This pull request references MON-3134 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the epic to target the "4.15.0" version, but no target version was set.

In response to this:

Allows to query alerts of application
namespaces as an application user.

• I added CHANGELOG entry for this change.
• No user facing changes, so no entry in CHANGELOG was needed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[simonpasquier]

/lgtm
/hold cancel

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: simonpasquier, slashpai

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [simonpasquier,slashpai]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[slashpai]

cc: @bburt-rh for docs approval

[slashpai]

/jira refresh

[openshift-ci-robot]

@slashpai: This pull request references MON-3134 which is a valid jira issue.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[openshift-ci]

@slashpai: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build cluster-monitoring-operator-container-v4.15.0-202312071954.p0.g6010a5c.assembly.stream for distgit cluster-monitoring-operator.
All builds following this will include this PR.