YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs #7129

TaoYang526 · 2024-10-25T04:57:53Z

Description of PR

Handle uncaught exception for async-scheduling threads to prevent scheduler hangs.

How was this patch tested?

Includes 2 unit tests:
(1) HA-disabled: RM should be shutdown after async-scheduling thread exit.
(2) HA-enabled: RM1 should be transitioned from active to standby after async-scheduling thread exit, and this RM should work once it is activated again.

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

TaoYang526 · 2024-10-25T05:00:14Z

@shameersss1 @slfan1989 Could you help to review this PR please? Thanks.

hadoop-yetus · 2024-10-25T08:55:55Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 29s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 17s		trunk passed
-1 ❌	compile	0m 30s	/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 54s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 52s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 0s		trunk passed
+1 💚	shadedclient	36m 10s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
-1 ❌	compile	0m 23s	/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javac	0m 23s	/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 47s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 47s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 41s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 6 new + 108 unchanged - 0 fixed = 114 total (was 108)
+1 💚	mvnsite	0m 46s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 43s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	36m 41s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	105m 12s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		236m 41s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/1/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 5b9c79faa7db 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `e613bd7`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/1/testReport/
Max. process+thread count	931 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/1/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-10-25T09:06:04Z

💔 -1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 30s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 1s		codespell was not available.
+0 🆗	detsecrets	0m 1s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 51s		trunk passed
-1 ❌	compile	0m 31s	/branch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in trunk failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 55s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 54s		trunk passed
+1 💚	mvnsite	0m 59s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 51s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 57s		trunk passed
+1 💚	shadedclient	35m 53s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
-1 ❌	compile	0m 22s	/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
-1 ❌	javac	0m 22s	/patch-compile-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager-jdkUbuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.txt	hadoop-yarn-server-resourcemanager in the patch failed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04.
+1 💚	compile	0m 49s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 49s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	mvnsite	0m 48s		the patch passed
+1 💚	javadoc	0m 45s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	35m 43s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	105m 42s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		236m 48s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/2/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux de7685ea8385 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `808208b`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/2/testReport/
Max. process+thread count	959 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

slfan1989 · 2024-10-25T09:30:40Z

@TaoYang526 I'm sorry I've been busy this past week. I'll take a look at the PR you submitted over the weekend.

shameersss1

Thanks for taking this up quickly.
The changes look good to me +1

…o prevent scheduler hangs.

TaoYang526 · 2024-10-30T03:02:51Z

Thanks for taking this up quickly. The changes look good to me +1

Thanks @shameersss1 for the review.

I'll merge and then force-push to trigger testing for the patch.

hadoop-yetus · 2024-10-30T07:16:21Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 3s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 45s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 57s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 55s		trunk passed
+1 💚	mvnsite	1m 0s		trunk passed
+1 💚	javadoc	1m 0s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 0s		trunk passed
+1 💚	shadedclient	35m 25s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 49s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 48s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 42s		the patch passed
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 43s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	1m 57s		the patch passed
+1 💚	shadedclient	35m 44s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	105m 33s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		248m 45s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/3/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 735471b24c12 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `edf8db8`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/3/testReport/
Max. process+thread count	949 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-10-31T08:58:59Z

@slfan1989 - Could you please review this as well ? (This is a minor change)

hadoop-yetus · 2024-11-05T06:21:05Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	12m 4s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	44m 0s		trunk passed
+1 💚	compile	1m 1s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	compile	0m 57s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	checkstyle	0m 56s		trunk passed
+1 💚	mvnsite	1m 0s		trunk passed
+1 💚	javadoc	1m 0s		trunk passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 50s		trunk passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 1s		trunk passed
+1 💚	shadedclient	35m 24s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 48s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 48s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	javac	0m 48s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0 ⚠️	checkstyle	0m 44s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 108 unchanged - 0 fixed = 109 total (was 108)
+1 💚	mvnsite	0m 51s		the patch passed
+1 💚	javadoc	0m 45s		the patch passed with JDK Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04
+1 💚	javadoc	0m 42s		the patch passed with JDK Private Build-1.8.0_422-8u422-b05-1~20.04-b05
+1 💚	spotbugs	2m 0s		the patch passed
+1 💚	shadedclient	36m 16s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	105m 33s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 38s		The patch does not generate ASF License warnings.
		248m 50s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/4/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux 87fa2278e1fa 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `8bcfd73`
Default Java	Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.24+8-post-Ubuntu-1ubuntu320.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_422-8u422-b05-1~20.04-b05
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/4/testReport/
Max. process+thread count	940 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/4/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

shameersss1 · 2024-11-28T12:00:46Z

@slfan1989 Gentle reminder for review.
This is blocking YARN-7327

shameersss1 · 2024-12-04T11:20:06Z

@TaoYang526 - What could be the real life cases when the Async thread could crash ? Just curious to know

TaoYang526 · 2024-12-05T09:53:58Z

@TaoYang526 - What could be the real life cases when the Async thread could crash ? Just curious to know

@shameersss1 Async-thread may crash when runtime exceptions like NPE are thrown inside the scheduling process, this PR can guarantee that RM will have a chance to be recovered in those scenarios, otherwise RM will hang without obvious error (so that user have to dig ERROR info from large volume of RM logs). There's no known cases for me, but there may still be unresolved issues or new problems introduced in the future.

TaoYang526 · 2024-12-05T10:04:53Z

@Hexiaoqiao Could you please help to review this PR at your convenience? Thanks.

Hexiaoqiao

LGTM. +1 from my side. Will commit if no more comment while two workdays later. Thanks.

slfan1989 · 2024-12-09T02:07:36Z

LGTM +1.

slfan1989 · 2024-12-09T02:16:32Z

@TaoYang526 Can we fix the checkstyle issues?

./hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestRMHAForAsyncScheduler.java:33:import org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils;:8: Unused import - org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils. [UnusedImports]

TaoYang526 · 2024-12-09T02:59:11Z

@slfan1989 Sure, thank for reminding that!

TaoYang526 · 2024-12-09T03:29:16Z

Removed the unused import for UT.
Thanks @Hexiaoqiao @slfan1989 @shameersss1 for the review!

slfan1989 · 2024-12-09T03:37:10Z

+1

slfan1989 · 2024-12-09T03:43:13Z

@slfan1989 Gentle reminder for review.
This is blocking YARN-7327

@TaoYang526 Could you please help review this PR(#7138) ? While I have confidence in @shameersss1 code quality, I'm not very familiar with the capacity scheduler, so I'm hesitant to draw any conclusions. I know @brumi1024 is familiar with this, and I’ve already invited him to help with the review.

hadoop-yetus · 2024-12-09T06:14:33Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	8m 24s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	32m 45s		trunk passed
+1 💚	compile	0m 29s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 27s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 30s		trunk passed
+1 💚	mvnsite	0m 32s		trunk passed
+1 💚	javadoc	0m 36s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 33s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 11s		trunk passed
+1 💚	shadedclient	19m 27s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 26s		the patch passed
+1 💚	compile	0m 28s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 28s		the patch passed
+1 💚	compile	0m 25s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 25s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 22s		the patch passed
+1 💚	mvnsite	0m 26s		the patch passed
+1 💚	javadoc	0m 24s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 27s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 0s		the patch passed
+1 💚	shadedclient	19m 30s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	88m 47s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 24s		The patch does not generate ASF License warnings.
		177m 52s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/6/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux b47560ce6e80 5.15.0-124-generic #134-Ubuntu SMP Fri Sep 27 20:20:17 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `806cdde`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/6/testReport/
Max. process+thread count	947 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/6/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

hadoop-yetus · 2024-12-09T07:18:50Z

🎊 +1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	14m 5s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any @author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	45m 24s		trunk passed
+1 💚	compile	1m 0s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 53s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	checkstyle	0m 54s		trunk passed
+1 💚	mvnsite	0m 58s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 56s		trunk passed
+1 💚	shadedclient	33m 30s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 46s		the patch passed
+1 💚	compile	0m 51s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 51s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	javac	0m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 42s		the patch passed
+1 💚	mvnsite	0m 47s		the patch passed
+1 💚	javadoc	0m 43s		the patch passed with JDK Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 40s		the patch passed with JDK Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
+1 💚	spotbugs	1m 51s		the patch passed
+1 💚	shadedclient	33m 21s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	103m 54s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 37s		The patch does not generate ASF License warnings.
		245m 7s

Subsystem	Report/Notes
Docker	ClientAPI=1.47 ServerAPI=1.47 base: https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/5/artifact/out/Dockerfile
GITHUB PR	#7129
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux ce210d7ccb4e 5.15.0-122-generic #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk / `806cdde`
Default Java	Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.25+9-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_432-8u432-ga~~us1-0ubuntu2~~20.04-ga
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/5/testReport/
Max. process+thread count	966 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7129/5/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0 https://yetus.apache.org

This message was automatically generated.

Hexiaoqiao · 2024-12-09T13:47:29Z

Committed to trunk. Thanks @TaoYang526 for your contribution and @shameersss1 @slfan1989 for your reviews.

github-actions bot added YARN trunk labels Oct 25, 2024

shameersss1 approved these changes Oct 25, 2024

View reviewed changes

[YARN-10058] Handle uncaught exception for async-scheduling threads t…

edf8db8

…o prevent scheduler hangs.

TaoYang526 force-pushed the YARN-10058 branch from 808208b to edf8db8 Compare October 30, 2024 03:06

shameersss1 mentioned this pull request Nov 4, 2024

YARN-7327: Enable asynchronous scheduling by default for capacity scheduler #7138

Merged

4 tasks

Imporve UT: use GenericTestUtils#waitFor and avoid intermittent failure.

8bcfd73

TaoYang526 changed the title ~~[YARN-10058] Handle uncaught exception for async-scheduling threads to prevent scheduler hangs~~ YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs Nov 5, 2024

Hexiaoqiao approved these changes Dec 8, 2024

View reviewed changes

slfan1989 approved these changes Dec 9, 2024

View reviewed changes

Fix checkstyle issue.

806cdde

TaoYang526 force-pushed the YARN-10058 branch from b548f32 to 806cdde Compare December 9, 2024 03:15

Hexiaoqiao merged commit aa5fe6f into apache:trunk Dec 9, 2024
4 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs #7129

YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs #7129

TaoYang526 commented Oct 25, 2024 •

edited

Loading

TaoYang526 commented Oct 25, 2024

hadoop-yetus commented Oct 25, 2024

hadoop-yetus commented Oct 25, 2024

slfan1989 commented Oct 25, 2024

shameersss1 left a comment

TaoYang526 commented Oct 30, 2024

hadoop-yetus commented Oct 30, 2024

shameersss1 commented Oct 31, 2024

hadoop-yetus commented Nov 5, 2024

shameersss1 commented Nov 28, 2024

shameersss1 commented Dec 4, 2024

TaoYang526 commented Dec 5, 2024 •

edited

Loading

TaoYang526 commented Dec 5, 2024

Hexiaoqiao left a comment

slfan1989 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

TaoYang526 commented Dec 9, 2024

TaoYang526 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

hadoop-yetus commented Dec 9, 2024

hadoop-yetus commented Dec 9, 2024

Hexiaoqiao commented Dec 9, 2024

YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs #7129

YARN-10058. Handle uncaught exception for async-scheduling threads to prevent scheduler hangs #7129

Conversation

TaoYang526 commented Oct 25, 2024 • edited Loading

Description of PR

How was this patch tested?

For code changes:

TaoYang526 commented Oct 25, 2024

hadoop-yetus commented Oct 25, 2024

hadoop-yetus commented Oct 25, 2024

slfan1989 commented Oct 25, 2024

shameersss1 left a comment

Choose a reason for hiding this comment

TaoYang526 commented Oct 30, 2024

hadoop-yetus commented Oct 30, 2024

shameersss1 commented Oct 31, 2024

hadoop-yetus commented Nov 5, 2024

shameersss1 commented Nov 28, 2024

shameersss1 commented Dec 4, 2024

TaoYang526 commented Dec 5, 2024 • edited Loading

TaoYang526 commented Dec 5, 2024

Hexiaoqiao left a comment

Choose a reason for hiding this comment

slfan1989 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

TaoYang526 commented Dec 9, 2024

TaoYang526 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

slfan1989 commented Dec 9, 2024

hadoop-yetus commented Dec 9, 2024

hadoop-yetus commented Dec 9, 2024

Hexiaoqiao commented Dec 9, 2024

TaoYang526 commented Oct 25, 2024 •

edited

Loading

TaoYang526 commented Dec 5, 2024 •

edited

Loading