Skip to content

[CI] AutoFollowIT testAutoFollowPatterns failing #84403

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
ywangd opened this issue Feb 28, 2022 · 4 comments · Fixed by #85278
Closed

[CI] AutoFollowIT testAutoFollowPatterns failing #84403

ywangd opened this issue Feb 28, 2022 · 4 comments · Fixed by #85278
Assignees
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI

Comments

@ywangd
Copy link
Member

ywangd commented Feb 28, 2022

Build scan:
https://gradle-enterprise.elastic.co/s/kd44ouaevzdw2/tests/:x-pack:plugin:ccr:qa:multi-cluster:follow-cluster/org.elasticsearch.xpack.ccr.AutoFollowIT/testAutoFollowPatterns

Reproduction line:
./gradlew ':x-pack:plugin:ccr:qa:multi-cluster:follow-cluster' --tests "org.elasticsearch.xpack.ccr.AutoFollowIT.testAutoFollowPatterns" -Dtests.seed=E70A4552B6E0DF9E -Dtests.locale=en-AU -Dtests.timezone=Cuba -Druntime.java=17

Applicable branches:
master

Reproduces locally?:
No

Failure history:
https://gradle-enterprise.elastic.co/scans/tests?tests.container=org.elasticsearch.xpack.ccr.AutoFollowIT&tests.test=testAutoFollowPatterns

Failure excerpt:

java.lang.AssertionError: 
Expected: a value equal to or greater than <1>
     but: <0> was less than <1>

  at __randomizedtesting.SeedInfo.seed([E70A4552B6E0DF9E:4EADFB797E0E4E1A]:0)
  at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
  at org.junit.Assert.assertThat(Assert.java:956)
  at org.junit.Assert.assertThat(Assert.java:923)
  at org.elasticsearch.xpack.ccr.ESCCRRestTestCase.verifyAutoFollowMonitoring(ESCCRRestTestCase.java:247)
  at org.elasticsearch.xpack.ccr.AutoFollowIT.lambda$testAutoFollowPatterns$2(AutoFollowIT.java:180)
  at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:1068)
  at org.elasticsearch.xpack.ccr.AutoFollowIT.testAutoFollowPatterns(AutoFollowIT.java:178)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(NativeMethodAccessorImpl.java:-2)
  at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
  at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
  at java.lang.reflect.Method.invoke(Method.java:568)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.invoke(RandomizedRunner.java:1758)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$8.evaluate(RandomizedRunner.java:946)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$9.evaluate(RandomizedRunner.java:982)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$10.evaluate(RandomizedRunner.java:996)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleSetupTeardownChained$1.evaluate(TestRuleSetupTeardownChained.java:44)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleThreadAndTestName$1.evaluate(TestRuleThreadAndTestName.java:45)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.forkTimeoutingTask(ThreadLeakControl.java:824)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$3.evaluate(ThreadLeakControl.java:475)
  at com.carrotsearch.randomizedtesting.RandomizedRunner.runSingleTest(RandomizedRunner.java:955)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$5.evaluate(RandomizedRunner.java:840)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$6.evaluate(RandomizedRunner.java:891)
  at com.carrotsearch.randomizedtesting.RandomizedRunner$7.evaluate(RandomizedRunner.java:902)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
  at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
  at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
  at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
  at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
  at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:375)
  at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:831)
  at java.lang.Thread.run(Thread.java:833)

@ywangd ywangd added :Distributed Indexing/CCR Issues around the Cross Cluster State Replication features >test-failure Triaged test failures from CI labels Feb 28, 2022
@elasticmachine elasticmachine added the Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. label Feb 28, 2022
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed (Team:Distributed)

@dakrone
Copy link
Member

dakrone commented Mar 4, 2022

This happened again today on master: https://gradle-enterprise.elastic.co/s/beukhk4arbluk and once on 8.0: https://gradle-enterprise.elastic.co/s/j6vlqr3nwlmg2

@tlrx tlrx self-assigned this Mar 8, 2022
@jtibshirani
Copy link
Contributor

Another example just popped up today: https://gradle-enterprise.elastic.co/s/qqha33leq36ai. Perhaps we could mute as we work on a fix?

@DaveCTurner
Copy link
Contributor

DaveCTurner commented Mar 18, 2022

I muted it in e27f64e on master and cda166a on 8.1 (suspecting that the failure Lee saw on 8.0 also affects 8.1)

DaveCTurner added a commit that referenced this issue Mar 18, 2022
DaveCTurner added a commit that referenced this issue Mar 18, 2022
DaveCTurner added a commit that referenced this issue Mar 18, 2022
This reverts commit 7e38134.
elasticsearchmachine pushed a commit that referenced this issue Mar 23, 2022
…85278)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose. 

Closes #84403
tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 23, 2022
…lastic#85278)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose. 

Closes elastic#84403
tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 23, 2022
…lastic#85278)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose. 

Closes elastic#84403
tlrx added a commit to tlrx/elasticsearch that referenced this issue Mar 23, 2022
…lastic#85278)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose.

Closes elastic#84403
elasticsearchmachine pushed a commit that referenced this issue Mar 23, 2022
…85278) (#85285)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose. 

Closes #84403
elasticsearchmachine pushed a commit that referenced this issue Mar 23, 2022
…85278) (#85288)

The test `AutoFollowIT.testAutoFollowPatterns()` failed multiple times
when waiting for CCR's auto-follow stats to be collected and indexed in
the `.monitoring-es-*` index. This is possibly because monitoring data
are collected every 10 seconds and some of them can take more time to be
collected, exceeding the 30s timeout in the test.

This pull request re-uses the `assertLongBusy()` so that the test will
wait more time for the data to show up and in case the data are not here
it will prints out the current CCR's auto-follow stats for debugging
purpose.

Closes #84403
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Distributed Indexing/CCR Issues around the Cross Cluster State Replication features Team:Distributed (Obsolete) Meta label for distributed team (obsolete). Replaced by Distributed Indexing/Coordination. >test-failure Triaged test failures from CI
Projects
None yet
Development

Successfully merging a pull request may close this issue.

6 participants