Bug 1921335: Fix and adjust ThanosSidecarUnhealthy alert #1090
Conversation
Signed-off-by: Damien Grisonnet <[email protected]>
Decrease severity of all the Thanos sidecar alerts to warning and increase their duration to 1h. Signed-off-by: Damien Grisonnet <[email protected]>
openshift-ci-robot
added
the
bugzilla/severity-high
|
@dgrisonnet: This pull request references Bugzilla bug 1921335, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
bugzilla/invalid-bug
|
@dgrisonnet: This pull request references Bugzilla bug 1921335, which is invalid:
Comment In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
openshift-ci-robot
added
the
approved
|
/bugzilla refresh |
openshift-ci-robot
added
bugzilla/valid-bug
|
@dgrisonnet: This pull request references Bugzilla bug 1921335, which is valid. The bug has been moved to the POST state. The bug has been updated to refer to the pull request using the external bug tracker. 3 validation(s) were run on this bug
Requesting review from QA contact: In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Note that this PR should fulfill all the requirements of the Bugzilla, except making the Thanos sidecar alerts more resilient to WAL replays as this is still in progress. Should I open a separate BZ just for this effort? |
|
@dgrisonnet: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here. |
|
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: dgrisonnet, s-urbaniak The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/retest Please review the full test history for this PR and help us cut down flakes. |
1 similar comment
|
/retest Please review the full test history for this PR and help us cut down flakes. |
|
@dgrisonnet: All pull requests linked via external trackers have merged: Bugzilla bug 1921335 has been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
Bring the following upstream fix thanos-io/thanos@95b2fe9 that should prevent the ThanosSidecarUnhealthy alert from firing in CI.
Also readjust the Thanos sidecar alerts to
warningseverity and their duration to1has per the recent discussions around alerting in OCP.