Skip to content

Zen1IT testMultipleNodeMigrationFromZen1ToZen2WithThreeNodes failes on DeprecationLogging #45443

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
pgomulka opened this issue Aug 12, 2019 · 2 comments
Closed

Zen1IT testMultipleNodeMigrationFromZen1ToZen2WithThreeNodes failes on DeprecationLogging #45443

pgomulka opened this issue Aug 12, 2019 · 2 comments
Labels
:Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI

Comments

@pgomulka
Copy link
Contributor

pgomulka commented Aug 12, 2019 •
edited
Loading

very likely this is a race condition. Can only happen in test env as in prod we only have one thread context
https://gradle-enterprise.elastic.co/s/rf4gigie3ky22
It fails on:

aused by: java.lang.IllegalStateException: threadcontext is already closed
	at __randomizedtesting.SeedInfo.seed([D61C383C0E0E7A14]:0)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextThreadLocal.ensureOpen(ThreadContext.java:673)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextThreadLocal.get(ThreadContext.java:666)
	at org.elasticsearch.common.util.concurrent.ThreadContext.getHeader(ThreadContext.java:293)
	at org.elasticsearch.common.logging.DeprecationLogger.lambda$getXOpaqueId$1(DeprecationLogger.java:266)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:174)
	at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:175)
	at java.util.Spliterators$ArraySpliterator.tryAdvance(Spliterators.java:958)
	at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
	at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464)
	at org.elasticsearch.common.logging.DeprecationLogger.getXOpaqueId(DeprecationLogger.java:267)
	at org.elasticsearch.common.logging.DeprecationLogger.deprecatedAndMaybeLog(DeprecationLogger.java:137)
	at org.elasticsearch.common.settings.Setting.checkDeprecation(Setting.java:504)
	at org.elasticsearch.common.settings.Setting.getRaw(Setting.java:478)
	at org.elasticsearch.common.settings.Setting.get(Setting.java:430)
	at org.elasticsearch.common.settings.Setting.get(Setting.java:426)
	at org.elasticsearch.cluster.coordination.DiscoveryUpgradeService.activate(DiscoveryUpgradeService.java:131)
	at org.elasticsearch.cluster.coordination.Coordinator.becomeCandidate(Coordinator.java:549)
	at org.elasticsearch.cluster.coordination.Coordinator.onLeaderFailure(Coordinator.java:220)
	at org.elasticsearch.cluster.coordination.LeaderChecker$CheckScheduler$2.run(LeaderChecker.java:299)
	at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:703)

when this should be protected by
DeprecationLogger.getXOpaqueId

public String getXOpaqueId(Set<ThreadContext> threadContexts) {
        return threadContexts.stream()
                             .filter(t -> t.isClosed() == false)
                             .filter(t -> t.getHeader(Task.X_OPAQUE_ID) != null)
                             .findFirst()
                             .map(t -> t.getHeader(Task.X_OPAQUE_ID))
                             .orElse("");
    }

https://build-stats.elastic.co/app/kibana#/doc/b646ed00-7efc-11e8-bf69-63c8ef516157/build-*/t?id=20190812090310-5EDADE9C&_g=()
@pgomulka pgomulka added :Core/Infra/Core Core issues without another label >test-failure Triaged test failures from CI labels Aug 12, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-infra

@pgomulka pgomulka self-assigned this Aug 12, 2019
pgomulka added a commit to pgomulka/elasticsearch that referenced this issue Aug 12, 2019
@pgomulka
Fix race condition in getXOpaqueId in tests
6aa716c 
Due to a race condition between isClosed and getHeader it is possible
that getXOpaqueId can throw an IllegalStateException.
This only  occurs when run with multiple tests on the same JVM. This
value is not to be asserted in these tests. Only when running in
isolation in its own JVM.
This won't happen in production as we only expect one ThreadContext per
Node (jvm)

closes elastic#45443
@pgomulka pgomulka mentioned this issue Aug 12, 2019
Closed
@rjernst rjernst added the Team:Core/Infra Meta label for core/infra team label May 4, 2020
@rjernst
Copy link
Member

rjernst commented Aug 6, 2020

I believe this was fixed by #43249

@rjernst rjernst closed this as completed Aug 6, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
:Core/Infra/Core Core issues without another label Team:Core/Infra Meta label for core/infra team >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix race condition in getXOpaqueId in tests pgomulka/elasticsearch
3 participants
@rjernst @pgomulka @elasticmachine