Skip to content

[CI] ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability fails with out of memory #44429

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
matriv opened this issue Jul 16, 2019 · 1 comment · Fixed by #44444
Closed

[CI] ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability fails with out of memory #44429

matriv opened this issue Jul 16, 2019 · 1 comment · Fixed by #44444
Assignees
@ywelsch
Labels
:Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >test-failure Triaged test failures from CI

Comments

@matriv
Copy link
Contributor

matriv commented Jul 16, 2019

CI link: https://scans.gradle.com/s/4sjpqlk4dr5pk/console-log?task=:server:integTest (fails in 7.x)

Cannot reproduce locally:

./gradlew :server:integTest --tests "org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability" \
  -Dtests.seed=FDE6B8EF8F6C8595 \
  -Dtests.security.manager=true \
  -Dtests.locale=bg \
  -Dtests.timezone=Europe/Bucharest \
  -Dcompiler.java=12 \
  -Druntime.java=8

Stacktrace:


java.lang.OutOfMemoryError: GC overhead limit exceeded
--
Dumping heap to /var/lib/jenkins/workspace/elastic+elasticsearch+7.x+multijob+fast+part1/server/build/heapdump/java_pid42272.hprof ...
Heap dump file created [678836066 bytes in 5.889 secs]
 
 
org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT > testSeqNoCASLinearizability FAILED
java.lang.OutOfMemoryError: GC overhead limit exceeded
at __randomizedtesting.SeedInfo.seed([FDE6B8EF8F6C8595:26289685957EA721]:0)
at org.apache.lucene.util.FixedBitSet.clone(FixedBitSet.java:480)
at org.elasticsearch.cluster.coordination.LinearizabilityChecker.isLinearizable(LinearizabilityChecker.java:253)
at org.elasticsearch.cluster.coordination.LinearizabilityChecker.lambda$isLinearizable$0(LinearizabilityChecker.java:233)
at org.elasticsearch.cluster.coordination.LinearizabilityChecker$$Lambda$2691/1989389279.test(Unknown Source)
at java.util.stream.MatchOps$1MatchSink.accept(MatchOps.java:90)
at java.util.Collections$2.tryAdvance(Collections.java:4717)
at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126)
at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:498)
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:485)
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:230)
at java.util.stream.MatchOps$MatchOp.evaluateSequential(MatchOps.java:196)
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
at java.util.stream.ReferencePipeline.allMatch(ReferencePipeline.java:454)
at org.elasticsearch.cluster.coordination.LinearizabilityChecker.isLinearizable(LinearizabilityChecker.java:233)
at org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT$Partition.isLinearizable(ConcurrentSeqNoVersioningIT.java:447)
at org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT$Partition.assertLinearizable(ConcurrentSeqNoVersioningIT.java:461)
at org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT$$Lambda$2687/910861838.accept(Unknown Source)
at java.util.ArrayList.forEach(ArrayList.java:1257)
at org.elasticsearch.versioning.ConcurrentSeqNoVersioningIT.testSeqNoCASLinearizability(ConcurrentSeqNoVersioningIT.java:194)

Not muting as there are only a few occurrences.

Full logs can be found here
@matriv matriv added :Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >test-failure Triaged test failures from CI labels Jul 16, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@ywelsch ywelsch mentioned this issue Jul 16, 2019
Merged
@ywelsch ywelsch self-assigned this Jul 16, 2019
ywelsch added a commit that referenced this issue Jul 17, 2019
@ywelsch
Terminate linearizability check early on large histories (#44444)
998bb47 
Large histories can be problematic and have the linearizability checker occasionally run OOM. As it's
very difficult to bound the size of the histories just right, this PR will let it instead run for 10 seconds
on large histories and then abort.

Closes #44429
ywelsch added a commit that referenced this issue Jul 17, 2019
@ywelsch
Terminate linearizability check early on large histories (#44444)
f78e64e 
Large histories can be problematic and have the linearizability checker occasionally run OOM. As it's
very difficult to bound the size of the histories just right, this PR will let it instead run for 10 seconds
on large histories and then abort.

Closes #44429
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ywelsch ywelsch

Labels
:Distributed Indexing/CRUD A catch all label for issues around indexing, updating and getting a doc by id. Not search. >test-failure Triaged test failures from CI
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Terminate linearizability check early on large histories ywelsch/elasticsearch
3 participants
@ywelsch @matriv @elasticmachine