Skip to content

OCPBUGS-39574: (fix) registry pods do not come up again after node failure (#3366) #855

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Sep 6, 2024
Merged

OCPBUGS-39574: (fix) registry pods do not come up again after node failure (#3366) #855

merged 1 commit into from
Sep 6, 2024

Conversation

anik120
Copy link
Contributor

@anik120 anik120 commented Sep 5, 2024

PR 3201 attempted to solve for the issue by deleting the pods stuck in Terminating due to unreachable node. However, the logic to do that was included in EnsureRegistryServer, which only gets executed if polling in requested by the user.

This PR moves the logic of checking for dead pods out of EnsureRegistryServer, and puts it in CheckRegistryServer instead. This way, if there are any dead pods detected during CheckRegistryServer, the value of healthy is returned false, which inturn triggers EnsureRegistryServer.

Upstream-repository: operator-lifecycle-manager
Upstream-commit: f2431893193e7112f78298ad7682ff3e1b179d8c

@anik120
(fix) registry pods do not come up again after node failure (#3366)
2648ff3 
[PR 3201](operator-framework/operator-lifecycle-manager#3201) attempted to solve for the issue by deleting the pods stuck in
`Terminating` due to unreachable node. However, the logic to do that was
included in `EnsureRegistryServer`, which only gets executed if polling in
requested by the user.

This PR moves the logic of checking for dead pods out of `EnsureRegistryServer`,
and puts it in `CheckRegistryServer` instead. This way, if there are any dead pods
detected during `CheckRegistryServer`, the value of `healthy` is returned `false`,
which inturn triggers `EnsureRegistryServer`.

Upstream-repository: operator-lifecycle-manager
Upstream-commit: f2431893193e7112f78298ad7682ff3e1b179d8c
@openshift-ci-robot openshift-ci-robot added jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 5, 2024
@openshift-ci-robot
Copy link

@anik120: This pull request references Jira Issue OCPBUGS-39574, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected dependent Jira Issue OCPBUGS-36661 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

PR 3201 attempted to solve for the issue by deleting the pods stuck in Terminating due to unreachable node. However, the logic to do that was included in EnsureRegistryServer, which only gets executed if polling in requested by the user.

This PR moves the logic of checking for dead pods out of EnsureRegistryServer, and puts it in CheckRegistryServer instead. This way, if there are any dead pods detected during CheckRegistryServer, the value of healthy is returned false, which inturn triggers EnsureRegistryServer.

Upstream-repository: operator-lifecycle-manager
Upstream-commit: f2431893193e7112f78298ad7682ff3e1b179d8c

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci OpenShift CI
Copy link
Contributor

openshift-ci bot commented Sep 5, 2024

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: anik120

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 5, 2024
@anik120
Copy link
Contributor Author

anik120 commented Sep 5, 2024

/jira refresh

@openshift-ci-robot
Copy link

@anik120: This pull request references Jira Issue OCPBUGS-39574, which is invalid:

  • expected dependent Jira Issue OCPBUGS-36661 to be in one of the following states: MODIFIED, ON_QA, VERIFIED, but it is POST instead

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@anik120
Copy link
Contributor Author

anik120 commented Sep 5, 2024

/retest

@jianzhangbjz
Copy link
Contributor

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Sep 6, 2024
@openshift-ci-robot
Copy link

@jianzhangbjz: This pull request references Jira Issue OCPBUGS-39574, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.17.0) matches configured target version for branch (4.17.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-36661 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-36661 targets the "4.18.0" version, which is one of the valid target versions: 4.18.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@jianzhangbjz
Copy link
Contributor

Pre-merge test pass, details: https://issues.redhat.com//browse/OCPBUGS-39574
/lgtm
/label qe-approved
/label cherry-pick-approved

@openshift-ci openshift-ci bot added qe-approved Signifies that QE has signed off on this PR cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. labels Sep 6, 2024
@openshift-ci-robot
Copy link

@anik120: This pull request references Jira Issue OCPBUGS-39574, which is valid.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.17.0) matches configured target version for branch (4.17.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-36661 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-36661 targets the "4.18.0" version, which is one of the valid target versions: 4.18.0
  • bug has dependents

No GitHub users were found matching the public email listed for the QA contact in Jira ([email protected]), skipping review request.

In response to this:

PR 3201 attempted to solve for the issue by deleting the pods stuck in Terminating due to unreachable node. However, the logic to do that was included in EnsureRegistryServer, which only gets executed if polling in requested by the user.

This PR moves the logic of checking for dead pods out of EnsureRegistryServer, and puts it in CheckRegistryServer instead. This way, if there are any dead pods detected during CheckRegistryServer, the value of healthy is returned false, which inturn triggers EnsureRegistryServer.

Upstream-repository: operator-lifecycle-manager
Upstream-commit: f2431893193e7112f78298ad7682ff3e1b179d8c

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Sep 6, 2024
@perdasilva
Copy link
Contributor

/retest

@jianzhangbjz
Copy link
Contributor

Hi @perdasilva , I guess it needs the backport-risk-assessed label.

@perdasilva
Copy link
Contributor

/label backport-risk-assessed

@openshift-ci openshift-ci bot added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Sep 6, 2024
@perdasilva
Copy link
Contributor

/retest

@openshift-merge-bot openshift-merge-bot bot merged commit ff240fe into openshift:release-4.17 Sep 6, 2024
11 of 12 checks passed
@openshift-ci-robot
Copy link

@anik120: Jira Issue OCPBUGS-39574: Some pull requests linked via external trackers have merged:

The following pull requests linked via external trackers have not merged:

These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with /jira refresh.

Jira Issue OCPBUGS-39574 has not been moved to the MODIFIED state.

In response to this:

PR 3201 attempted to solve for the issue by deleting the pods stuck in Terminating due to unreachable node. However, the logic to do that was included in EnsureRegistryServer, which only gets executed if polling in requested by the user.

This PR moves the logic of checking for dead pods out of EnsureRegistryServer, and puts it in CheckRegistryServer instead. This way, if there are any dead pods detected during CheckRegistryServer, the value of healthy is returned false, which inturn triggers EnsureRegistryServer.

Upstream-repository: operator-lifecycle-manager
Upstream-commit: f2431893193e7112f78298ad7682ff3e1b179d8c

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@anik120
Copy link
Contributor Author

anik120 commented Sep 6, 2024

/jira refresh

@openshift-ci-robot
Copy link

@anik120: Jira Issue OCPBUGS-39574: All pull requests linked via external trackers have merged:

Jira Issue OCPBUGS-39574 has been moved to the MODIFIED state.

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: operator-lifecycle-manager
This PR has been included in build operator-lifecycle-manager-container-v4.17.0-202409061435.p0.gff240fe.assembly.stream.el9.
All builds following this will include this PR.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: operator-registry
This PR has been included in build operator-registry-container-v4.17.0-202409061435.p0.gff240fe.assembly.stream.el9.
All builds following this will include this PR.

@openshift-bot
Copy link
Contributor

[ART PR BUILD NOTIFIER]

Distgit: ose-operator-framework-tools
This PR has been included in build ose-operator-framework-tools-container-v4.17.0-202409061435.p0.gff240fe.assembly.stream.el9.
All builds following this will include this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ankitathomas ankitathomas Awaiting requested review from ankitathomas

@joelanford joelanford Awaiting requested review from joelanford

Assignees

@bandrade bandrade

@jianzhangbjz jianzhangbjz

@emmajiafan emmajiafan

@kuiwang02 kuiwang02

@Xia-Zhao-rh Xia-Zhao-rh

@KeenonLee KeenonLee

Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/severity-important Referenced Jira bug's severity is important for the branch this PR is targeting. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. qe-approved Signifies that QE has signed off on this PR
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
10 participants
@anik120 @openshift-ci-robot @jianzhangbjz @perdasilva @openshift-bot @bandrade @emmajiafan @kuiwang02 @Xia-Zhao-rh @KeenonLee