[SPARK-5788] [PySpark] capture the exception in python write thread #4577

davies · 2015-02-12T22:30:55Z

The exception in Python writer thread will shutdown executor.

davies · 2015-02-12T22:32:30Z

cc @aarondav

SparkQA · 2015-02-12T22:32:55Z

Test build #27389 has started for PR 4577 at commit 139b0db.

This patch merges cleanly.

rxin · 2015-02-12T23:09:05Z

core/src/main/scala/org/apache/spark/api/python/PythonRDD.scala

@@ -202,7 +202,7 @@ private[spark] class PythonRDD(
      this.interrupt()
    }

-    override def run(): Unit = Utils.logUncaughtExceptions {
+    override def run(): Unit = Utils.tryLog {


Instead of doing a big tryLog (which is sort of dangerous because of the potential of users adding a return statement that would break this code), can we add an explicit try around the two worker.shutdownOutput() calls?

The run can not have return value. The InterruptedException could happen in any place (when the task is canceled), so I think it's better to do it here.

you can still call return even if the value is Unit

btw this happened a few times in the past in Spark. somebody added a return in a giant block that is wrapped in a closure. Code reviewers couldn't catch it because it didn't show up in the diff.

@rxin, is the concern that a user might write something like

def foo(): Unit = Utils.tryLog { return someCallThatThrowsAnException() }

and thereby skip the exception-handling logic?

SparkQA · 2015-02-12T23:50:59Z

Test build #27389 has finished for PR 4577 at commit 139b0db.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-12T23:51:03Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27389/
Test PASSed.

SparkQA · 2015-02-17T00:12:35Z

Test build #27599 has started for PR 4577 at commit eb0ceff.

This patch merges cleanly.

davies · 2015-02-17T00:31:48Z

@rxin @JoshRosen fixed.

SparkQA · 2015-02-17T01:39:49Z

Test build #27599 has finished for PR 4577 at commit eb0ceff.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

AmplabJenkins · 2015-02-17T01:39:53Z

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/27599/
Test PASSed.

JoshRosen · 2015-02-17T01:49:24Z

LGTM.

rxin · 2015-02-17T01:55:35Z

lgtm too. josh can u merge this?

The exception in Python writer thread will shutdown executor. Author: Davies Liu <[email protected]> Closes #4577 from davies/exception and squashes the following commits: eb0ceff [Davies Liu] Update PythonRDD.scala 139b0db [Davies Liu] capture the exception in python write thread (cherry picked from commit b1bd1dd) Signed-off-by: Josh Rosen <[email protected]>

JoshRosen · 2015-02-17T01:58:45Z

I've merged this into master (1.4.0), branch-1.3 (1.3.0), and branch-1.2 (1.2.2). Thanks!

The exception in Python writer thread will shutdown executor. Author: Davies Liu <[email protected]> Closes apache#4577 from davies/exception and squashes the following commits: eb0ceff [Davies Liu] Update PythonRDD.scala 139b0db [Davies Liu] capture the exception in python write thread (cherry picked from commit b1bd1dd) Signed-off-by: Josh Rosen <[email protected]>

### What changes were proposed in this pull request? This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2. ### Why are the changes needed? There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind): [#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock) [#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input [#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation) [#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor [#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id' [#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling The full release note of 2.17.2: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47241 from wayneguow/upgrade_jackson. Authored-by: Wei Guo <[email protected]> Signed-off-by: yangjie01 <[email protected]>

### What changes were proposed in this pull request? This PR amis to upgrade `fasterxml.jackson` from 2.17.1 to 2.17.2. ### Why are the changes needed? There are some bug fixes about [Databind](https://github.com/FasterXML/jackson-databind): [apache#4561](FasterXML/jackson-databind#4561): Issues using jackson-databind 2.17.1 with Reactor (wrt DeserializerCache and ReentrantLock) [apache#4575](FasterXML/jackson-databind#4575): StdDelegatingSerializer does not consider a Converter that may return null for a non-null input [apache#4577](FasterXML/jackson-databind#4577): Cannot deserialize value of type java.math.BigDecimal from String "3." (not a valid representation) [apache#4595](FasterXML/jackson-databind#4595): No way to explicitly disable wrapping in custom annotation processor [apache#4607](FasterXML/jackson-databind#4607): MismatchedInput: No Object Id found for an instance of X to assign to property 'id' [apache#4610](FasterXML/jackson-databind#4610): DeserializationFeature.FAIL_ON_UNRESOLVED_OBJECT_IDS does not work when used with Polymorphic type handling The full release note of 2.17.2: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.17.2 ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? Pass GA. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47241 from wayneguow/upgrade_jackson. Authored-by: Wei Guo <[email protected]> Signed-off-by: yangjie01 <[email protected]>

capture the exception in python write thread

139b0db

rxin reviewed Feb 12, 2015
View reviewed changes

Update PythonRDD.scala

eb0ceff

asfgit closed this in b1bd1dd Feb 17, 2015

[SPARK-5788] [PySpark] capture the exception in python write thread #4577

[SPARK-5788] [PySpark] capture the exception in python write thread #4577

Uh oh!

Conversation

davies commented Feb 12, 2015

Uh oh!

davies commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

rxin Feb 12, 2015

Choose a reason for hiding this comment

Uh oh!

davies Feb 12, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Feb 12, 2015

Choose a reason for hiding this comment

Uh oh!

rxin Feb 12, 2015

Choose a reason for hiding this comment

Uh oh!

JoshRosen Feb 16, 2015

Choose a reason for hiding this comment

Uh oh!

SparkQA commented Feb 12, 2015

Uh oh!

AmplabJenkins commented Feb 12, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

davies commented Feb 17, 2015

Uh oh!

SparkQA commented Feb 17, 2015

Uh oh!

AmplabJenkins commented Feb 17, 2015

Uh oh!

JoshRosen commented Feb 17, 2015

Uh oh!

rxin commented Feb 17, 2015

Uh oh!

JoshRosen commented Feb 17, 2015

Uh oh!

Uh oh!