Skip to content

Fix Deadlock from Thread.suspend in Test #39261

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 1 commit into from
Feb 22, 2019
Merged

Fix Deadlock from Thread.suspend in Test #39261

merged 1 commit into from
Feb 22, 2019

Conversation

original-brownbear
Copy link
Contributor

See stacktraces from failed run, that deadlocked:

  1>  [test_node][1]
  1> ----
  1> java.lang.Thread.yield(Native Method)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$2(LongGCDisruptionTests.java:136)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1750/1219702866.run(Unknown Source)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$LockedExecutor.executeLocked(LongGCDisruptionTests.java:47)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$3(LongGCDisruptionTests.java:135)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1749/1018511683.run(Unknown Source)
  1> java.lang.Thread.run(Thread.java:748)
  1> [test_node][3]
  1> ----
  1> java.lang.Thread.yield(Native Method)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$3(LongGCDisruptionTests.java:140)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1749/1018511683.run(Unknown Source)
  1> java.lang.Thread.run(Thread.java:748)
  1> [test_node][0]
  1> ----
  1> java.lang.Thread.yield(Native Method)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$3(LongGCDisruptionTests.java:140)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1749/1018511683.run(Unknown Source)
  1> java.lang.Thread.run(Thread.java:748)
  1> [test_node][2]
  1> ----
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1750/1219702866.<init>(Unknown Source)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1750/1219702866.get$Lambda(Unknown Source)
  1> java.lang.invoke.LambdaForm$DMH/1525416409.invokeStatic_L_L(LambdaForm$DMH)
  1> java.lang.invoke.LambdaForm$MH/1692675110.linkToTargetMethod(LambdaForm$MH)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$3(LongGCDisruptionTests.java:135)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1749/1018511683.run(Unknown Source)
  1> java.lang.Thread.run(Thread.java:748)
  1> [test_node][4]
  1> ----
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1750/1219702866.get$Lambda(Unknown Source)
  1> java.lang.invoke.LambdaForm$DMH/1525416409.invokeStatic_L_L(LambdaForm$DMH)
  1> java.lang.invoke.LambdaForm$MH/1692675110.linkToTargetMethod(LambdaForm$MH)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests.lambda$testNotBlockingUnsafeStackTraces$3(LongGCDisruptionTests.java:135)
  1> org.elasticsearch.test.disruption.LongGCDisruptionTests$$Lambda$1749/1018511683.run(Unknown Source)
  1> java.lang.Thread.run(Thread.java:748)
  1> [2019-02-21T05:37:30,979][INFO ][o.e.t.d.LongGCDisruptionTests] [testNotBlockingUnsafeStackTraces] after test
ERROR   31.9s J4 | LongGCDisruptionTests.testNotBlockingUnsafeStackTraces <<< FAILURES!
   > Throwable #1: java.lang.RuntimeException: suspending node threads took too long
   > 	at __randomizedtesting.SeedInfo.seed([6AB1F3531F128FA:4B8578A15DF9093]:0)
   > 	at org.elasticsearch.test.disruption.LongGCDisruption.startDisrupting(LongGCDisruption.java:124)
   > 	at org.elasticsearch.test.disruption.LongGCDisruptionTests.testNotBlockingUnsafeStackTraces(LongGCDisruptionTests.java:149)
   > 	at java.lang.Thread.run(Thread.java:748)
Completed [37/37] on J4 in 32.12s, 3 tests, 1 error <<< FAILURES!

@original-brownbear
Fix Deadlock from Thread.suspend in Test
521f895 
* The lambda invoked by the `lockedExecutor` eventually gets JITed (which runs a static initializer that we will suspend in with a very tiny chance).
   * Fixed by creating the `Runnable` in the main test thread and using the same instance in all threads
* Closes elastic#35686
@original-brownbear original-brownbear added >test Issues or PRs that are addressing/adding tests :Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. v8.0.0 v7.2.0 labels Feb 21, 2019
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-distributed

@original-brownbear original-brownbear mentioned this pull request Feb 21, 2019
Closed
jasontedor
jasontedor approved these changes Feb 22, 2019
View reviewed changes
Copy link
Member

@jasontedor jasontedor left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, good analysis.

@original-brownbear
Copy link
Contributor Author

@jasontedor thanks!

@original-brownbear original-brownbear merged commit 525bf40 into elastic:master Feb 22, 2019
@original-brownbear original-brownbear deleted the 35686 branch February 22, 2019 05:06
weizijun pushed a commit to weizijun/elasticsearch that referenced this pull request Feb 22, 2019
@original-brownbear @weizijun
Fix Deadlock from Thread.suspend in Test (elastic#39261)
9018090 
* The lambda invoked by the `lockedExecutor` eventually gets JITed (which runs a static initializer that we will suspend in with a very tiny chance).
   * Fixed by creating the `Runnable` in the main test thread and using the same instance in all threads
* Closes elastic#35686
weizijun pushed a commit to weizijun/elasticsearch that referenced this pull request Feb 22, 2019
@original-brownbear @weizijun
Fix Deadlock from Thread.suspend in Test (elastic#39261)
7991981 
* The lambda invoked by the `lockedExecutor` eventually gets JITed (which runs a static initializer that we will suspend in with a very tiny chance).
   * Fixed by creating the `Runnable` in the main test thread and using the same instance in all threads
* Closes elastic#35686
original-brownbear added a commit to original-brownbear/elasticsearch that referenced this pull request Feb 25, 2019
@original-brownbear
Fix Deadlock from Thread.suspend in Test (elastic#39261)
d5bda9d 
* The lambda invoked by the `lockedExecutor` eventually gets JITed (which runs a static initializer that we will suspend in with a very tiny chance).
   * Fixed by creating the `Runnable` in the main test thread and using the same instance in all threads
* Closes elastic#35686
@original-brownbear original-brownbear mentioned this pull request Feb 25, 2019
Merged
original-brownbear added a commit that referenced this pull request Feb 25, 2019
@original-brownbear
Fix Deadlock from Thread.suspend in Test (#39261) (#39341)
50d2736 
* The lambda invoked by the `lockedExecutor` eventually gets JITed (which runs a static initializer that we will suspend in with a very tiny chance).
   * Fixed by creating the `Runnable` in the main test thread and using the same instance in all threads
* Closes #35686
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jasontedor jasontedor jasontedor approved these changes

Assignees
No one assigned
Labels
:Distributed Indexing/Distributed A catch all label for anything in the Distributed Indexing Area. Please avoid if you can. >test Issues or PRs that are addressing/adding tests v7.2.0 v8.0.0-alpha1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[CI] LongGCDisruptionTests testNotBlockingUnsafeStackTraces failed
4 participants
@original-brownbear @elasticmachine @jasontedor @jakelandis