OCPBUGS-44693: Revert "disable ResilientWatchCacheInitialization feature" #2192
Conversation
openshift-ci-robot
added
the
backports/unvalidated-commits
|
@benluddy: the contents of this pull request could not be automatically validated. The following commits could not be validated and must be approved by a top-level approver:
Comment |
|
/hold |
openshift-ci
bot
added
the
do-not-merge/hold
benluddy
force-pushed
the
drop-temporary-disablement-resilientwatchcacheinitialization
branch
from
3a3459a to
c07d5a2
Compare
|
/hold cancel |
openshift-ci
bot
removed
the
do-not-merge/hold
|
@benluddy: the contents of this pull request could not be automatically validated. The following commits could not be validated and must be approved by a top-level approver:
Comment |
openshift-ci
bot
added
the
approved
|
/assign @bertinatto |
|
/cc @p0lyn0mial |
benluddy
changed the title
openshift-ci-robot
added
jira/valid-reference
|
@benluddy: This pull request references Jira Issue OCPBUGS-44693, which is valid. 3 validation(s) were run on this bug
Requesting review from QA contact: The bug has been updated to refer to the pull request using the external bug tracker. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
@benluddy: The following test failed, say
Full PR test history. Your PR dashboard. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here. |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/7cab8890-fe06-11ef-9ce0-fd46b737721c-0 |
|
/payload 4.19 nightly blocking |
|
@bertinatto: trigger 12 job(s) of type blocking for the nightly release of OCP 4.19
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/8d03f380-fe06-11ef-96de-41de0e739063-0 |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 The previous run failed with: |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/9ae043d0-fe67-11ef-8c58-f8ae53f72ac7-0 |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/6a170ce0-fea1-11ef-8cb8-136ff5981dc6-0 |
|
/payload 4.19 nightly informing |
|
/payload 4.19 nightly informing |
|
@bertinatto: trigger 65 job(s) of type informing for the nightly release of OCP 4.19
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b1236530-feb4-11ef-8fa4-da9e74b5539a-0 |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/b7405660-fed9-11ef-9e61-89eec6697b98-0 |
…heInitialization feature" This reverts commit 4772890.
benluddy
force-pushed
the
drop-temporary-disablement-resilientwatchcacheinitialization
branch
from
c07d5a2 to
12b2c09
Compare
|
@benluddy: the contents of this pull request could not be automatically validated. The following commits could not be validated and must be approved by a top-level approver:
Comment |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/c882c300-ff64-11ef-8455-d7605d0b6295-0 |
|
/payload-aggregate periodic-ci-openshift-release-master-nightly-4.19-e2e-agent-ha-dualstack-conformance 10 |
|
@bertinatto: trigger 1 job(s) for the /payload-(with-prs|job|aggregate|job-with-prs|aggregate-with-prs) command
See details on https://pr-payload-tests.ci.openshift.org/runs/ci/ccb6b400-fff9-11ef-9b2a-c23d7650b39f-0 |
|
/retest-required |
Based on this, I agreed with TRT we should give it a shot. We'll revert if things get bad. /remove-label backports/unvalidated-commits |
openshift-ci
bot
added
backports/validated-commits
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: benluddy, bertinatto The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
openshift-merge-bot
bot
merged commit d6f2dd2
into
openshift:master
|
@benluddy: Jira Issue OCPBUGS-44693: Some pull requests linked via external trackers have merged:
The following pull requests linked via external trackers have not merged:
These pull request must merge or be unlinked from the Jira bug in order for it to move to the next state. Once unlinked, request a bug refresh with Jira Issue OCPBUGS-44693 has not been moved to the MODIFIED state. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository. |
|
[ART PR BUILD NOTIFIER] Distgit: openshift-enterprise-pod |
|
[ART PR BUILD NOTIFIER] Distgit: kube-proxy |
|
[ART PR BUILD NOTIFIER] Distgit: openshift-enterprise-hyperkube |
|
[ART PR BUILD NOTIFIER] Distgit: ose-installer-kube-apiserver-artifacts |
When this feature is enabled, watch requests that are to be served from the watch cache immediately return 429 if the cache is not initialized and the client retries. When disabled, the same watch requests "hang" until they either time out or complete successfully.
There is an OCP test that counts the number of watch requests during a job on a per-user basis by scraping audit logs. The test fails if a user exceeds an arbitrary threshold that has been selected based on historical observations. With this feature enabled, any issue that delays watch cache initialization or forces a watch cache to reinitialize now results in an increase in the number of watch requests appearing in the audit logs (due to the retries), which in turn causes the test thresholds to breach.
This was temporarily disabled for kube-apiserver to improve the CI signal-to-noise ratio during the 1.31 rebase. It was not disabled for openshift-apiserver.
Sample job from the 1.31 rebase process before the feature was disabled: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/openshift-cluster-kube-apiserver-operator-1734-openshift-kubernetes-2055-openshift-cluster-kube-apiserver-operator-1734-nightly-4.18-e2e-aws-ovn-single-node-serial/1835775665903767552