Spark job cancellation no longer marks application as failed #8701
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
paul-laffon-dd
added
inst: apache spark
|
Hi! 👋 Thanks for your pull request! 🎉 To help us review it, please make sure to:
If you need help, please check our contributing guidelines. |
mobuchowski
approved these changes
Apr 16, 2025
paul-laffon-dd
changed the title
svc-squareup-copybara
pushed a commit
to cashapp/misk
that referenced
this pull request
May 7, 2025
| Package | Type | Package file | Manager | Update | Change | |---|---|---|---|---|---| | [com.google.cloud:google-cloud-logging](https://github.com/googleapis/java-logging) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `3.22.2` -> `3.22.3` | | [com.google.cloud:google-cloud-datastore](https://github.com/googleapis/java-datastore) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.28.0` -> `2.28.1` | | [com.datadoghq:dd-trace-api](https://github.com/datadog/dd-trace-java) | dependencies | misk/gradle/libs.versions.toml | gradle | minor | `1.48.2` -> `1.49.0` | | [software.amazon.awssdk:sdk-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:sqs](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb-enhanced](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:dynamodb](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:aws-core](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:bom](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | | [software.amazon.awssdk:auth](https://aws.amazon.com/sdkforjava) | dependencies | misk/gradle/libs.versions.toml | gradle | patch | `2.31.36` -> `2.31.37` | --- ### Release Notes <details> <summary>googleapis/java-logging (com.google.cloud:google-cloud-logging)</summary> ### [`v3.22.3`](https://github.com/googleapis/java-logging/blob/HEAD/CHANGELOG.md#3223-2025-05-06) ##### Bug Fixes - **deps:** Update the Java code generator (gapic-generator-java) to 2.56.3 ([844f4fa](googleapis/java-logging@844f4fa)) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.46.3 ([#​1801](googleapis/java-logging#1801)) ([d7aa7bc](googleapis/java-logging@d7aa7bc)) - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#​1803](googleapis/java-logging#1803)) ([5967ffe](googleapis/java-logging@5967ffe)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#​1804](googleapis/java-logging#1804)) ([e9a27ec](googleapis/java-logging@e9a27ec)) </details> <details> <summary>googleapis/java-datastore (com.google.cloud:google-cloud-datastore)</summary> ### [`v2.28.1`](https://github.com/googleapis/java-datastore/blob/HEAD/CHANGELOG.md#2281-2025-05-06) ##### Dependencies - Update dependency com.google.cloud:sdk-platform-java-config to v3.47.0 ([#​1841](googleapis/java-datastore#1841)) ([ac393e6](googleapis/java-datastore@ac393e6)) - Update googleapis/sdk-platform-java action to v2.57.0 ([#​1842](googleapis/java-datastore#1842)) ([0745906](googleapis/java-datastore@0745906)) </details> <details> <summary>datadog/dd-trace-java (com.datadoghq:dd-trace-api)</summary> ### [`v1.49.0`](https://github.com/DataDog/dd-trace-java/releases/tag/v1.49.0): 1.49.0 ### Components #### Configuration at Runtime - ✨ Add process tags as list to remote config payload ([#​8705](DataDog/dd-trace-java#8705) - [@​amarziali](https://github.com/amarziali)) #### Continuous Integration Visibility - 🐛 Add span propagation for Pekko scheduled tasks ([#​8765](DataDog/dd-trace-java#8765) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) - ✨ Update test.retry_reason to use full name of the feature ([#​8689](DataDog/dd-trace-java#8689) - [@​daniel-mohedano](https://github.com/daniel-mohedano)) - 🧹 Remove unused TestEventsHandler methods ([#​8674](DataDog/dd-trace-java#8674) - [@​nikita-tkachenko-datadog](https://github.com/nikita-tkachenko-datadog)) #### Dynamic Instrumentation - 🐛 Fix exclude identifiers normalization ([#​8742](DataDog/dd-trace-java#8742) - [@​jpbempel](https://github.com/jpbempel)) - ✨ Make source file tracking asynchronous ([#​8684](DataDog/dd-trace-java#8684) - [@​jpbempel](https://github.com/jpbempel)) - ✨ Add scope filtering for symbol extraction ([#​8676](DataDog/dd-trace-java#8676) - [@​jpbempel](https://github.com/jpbempel)) - ✨ Add support for [@​key](https://github.com/key) and [@​value](https://github.com/value) for Map filtering ([#​8669](DataDog/dd-trace-java#8669) - [@​jpbempel](https://github.com/jpbempel)) #### Library Injection - ✨ Add system property to force injection of the tracing library even though multiple javaagents have been detected ([#​8697](DataDog/dd-trace-java#8697) - [@​cecile75](https://github.com/cecile75)) #### Metrics - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#​8693](DataDog/dd-trace-java#8693) - [@​randomanderson](https://github.com/randomanderson)) #### Profiling - ✨ Bump ddprof-java to 1.25.1 ([#​8750](DataDog/dd-trace-java#8750) - [@​jbachorik](https://github.com/jbachorik)) - 🐛 Remove cleanup-on-shutdown for temporary files ([#​8746](DataDog/dd-trace-java#8746) - [@​jbachorik](https://github.com/jbachorik)) - ✨⚡ Replace a regex-based SMAP parser with a hand-crafted one ([#​8730](DataDog/dd-trace-java#8730) - [@​jbachorik](https://github.com/jbachorik)) - ✨ Improve error reporting on profiler startup ([#​8714](DataDog/dd-trace-java#8714) - [@​jbachorik](https://github.com/jbachorik)) - ✨ Exclude ProxyLeakTask exception from exception profiling ([#​8666](DataDog/dd-trace-java#8666) - [@​jbachorik](https://github.com/jbachorik)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#​8641](DataDog/dd-trace-java#8641) - [@​MattAlp](https://github.com/MattAlp)) #### Telemetry - ✨ Allow dogstatsd port to be configurable with DD_DOGSTATSD_PORT ([#​8693](DataDog/dd-trace-java#8693) - [@​randomanderson](https://github.com/randomanderson)) - 🐛 Fix appsec.waf.requests telemetry metric ([#​8644](DataDog/dd-trace-java#8644) - [@​jandro996](https://github.com/jandro996)) #### Tracer core - ✨ Exclude jackson afterburner dynamic classes from instrumentation ([#​8747](DataDog/dd-trace-java#8747) - [@​amarziali](https://github.com/amarziali)) - ✨ Introduce Java 8 bytecode bridge for instrumentation API ([#​8736](DataDog/dd-trace-java#8736) - [@​PerfectSlayer](https://github.com/PerfectSlayer)) - ⚡🧹 Use byte-buddy classes optimized for Java8+ ([#​8735](DataDog/dd-trace-java#8735) - [@​mcculls](https://github.com/mcculls)) - 🐛 Do not set the hibernate or datanucleus span service name when disabled ([#​8727](DataDog/dd-trace-java#8727) - [@​ygree](https://github.com/ygree)) - ✨ Update bytebuddy and ASM to support JDK 24 ([#​8720](DataDog/dd-trace-java#8720) - [@​sarahchen6](https://github.com/sarahchen6)) - 🐛 Turn off JDK socket support by default ([#​8715](DataDog/dd-trace-java#8715) - [@​mcculls](https://github.com/mcculls)) - 🐛 Log warning when trace buffer overflow occurs ([#​8712](DataDog/dd-trace-java#8712) - [@​ygree](https://github.com/ygree)) - ✨🧪 Introducing an internal integration name ([#​8708](DataDog/dd-trace-java#8708) - [@​amarziali](https://github.com/amarziali)) - ✨ Add process tags to client stats payload ([#​8704](DataDog/dd-trace-java#8704) - [@​amarziali](https://github.com/amarziali)) - ✨ Collect process tags for tracing ([#​8698](DataDog/dd-trace-java#8698) - [@​amarziali](https://github.com/amarziali)) - ✨ Stable Config file: target system properties in process_arguments and support template variables in YamlParser ([#​8690](DataDog/dd-trace-java#8690) - [@​mtoffl01](https://github.com/mtoffl01)) - ✨⚡ Use prefix trie for proxy ignores ([#​8678](DataDog/dd-trace-java#8678) - [@​amarziali](https://github.com/amarziali)) - ✨ Allow agent to be automatically injected when running aside Log4J patch agent ([#​8648](DataDog/dd-trace-java#8648) - [@​paullegranddc](https://github.com/paullegranddc)) - ✨ Use jvmstat for JDKs 9+ programmatically ([#​8641](DataDog/dd-trace-java#8641) - [@​MattAlp](https://github.com/MattAlp)) #### Tracer internal logging - 🐛 Delete print line ([#​8686](DataDog/dd-trace-java#8686) - [@​sarahchen6](https://github.com/sarahchen6)) ### Instrumentations #### Akka instrumentation - 🐛 Handle reentrant scope cleanup in Akka/Pekko actor instrumentations ([#​8722](DataDog/dd-trace-java#8722) - [@​mcculls](https://github.com/mcculls)) #### Apache Spark instrumentation - ✨ Use OpenLineage root parent information to generate trace id ([#​8726](DataDog/dd-trace-java#8726) - [@​mobuchowski](https://github.com/mobuchowski)) - ✨ Spark job cancellation no longer marks application as failed ([#​8701](DataDog/dd-trace-java#8701) - [@​paul-laffon-dd](https://github.com/paul-laffon-dd)) #### JDBC instrumentation - 💡 Add support for sybase tds jdbc driver ([#​8764](DataDog/dd-trace-java#8764) - [@​amarziali](https://github.com/amarziali)) #### Kotlin instrumentation - 🐛 Take defensive copy of parent scope stack when closing nested coroutines ([#​8749](DataDog/dd-trace-java#8749) - [@​mcculls](https://github.com/mcculls)) #### Reactor instrumentation - ✨⚡ Do not inspect reactor context when not needed ([#​8745](DataDog/dd-trace-java#8745) - [@​amarziali](https://github.com/amarziali)) </details> --- ### Configuration 📅 **Schedule**: Branch creation - "after 6pm every weekday,before 2am every weekday" in timezone Australia/Melbourne, Automerge - At any time (no schedule defined). 🚦 **Automerge**: Enabled. ♻ **Rebasing**: Never, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://github.com/renovatebot/renovate/discussions) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate). GitOrigin-RevId: 795f347ae34d056efc1194c2f606cee7bca1beea
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
Stop marking spark application as failed when the last spark job was cancelled
Motivation
Jira Ticket: DJMS-53
Additional Notes
From sparkListener interface, there is no standard way to know if a job was cancelled (cancellation uses the same class
JobFailed), so checking if the exception contains the "cancelled" keywordContributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issue