Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Undertow 2.3.10 #32445

Closed
chrisdennis opened this issue Mar 14, 2024 · 0 comments
Closed

Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Undertow 2.3.10 #32445

chrisdennis opened this issue Mar 14, 2024 · 0 comments
Assignees
@jhoeller
Labels
in: web Issues in web modules (web, webmvc, webflux, websocket) status: backported An issue that has been backported to maintenance branches type: bug A general bug
Milestone
6.1.7

Comments

@chrisdennis
Copy link

chrisdennis commented Mar 14, 2024 •
edited
Loading

Affects: Spring Framework 6.1.3

I have managed to produce (but not yet reproduce) what I think is a deadlock (or maybe more accurately a stall) in Spring 6.1.3 with Undertow 2.3.10 between the SpringApplicationShutdownHook and an XNIO thread attempting to close out a websocket channel. The error occurred in a CI job and I cannot reproduce locally, but given the similar historical occurences in #21031 I didn't want to leave this one hanging without at least reporting it. The two relevant stacks are below:

"XNIO-1 I/O-24" #136 prio=5 os_prio=0 cpu=60.57ms elapsed=4496.02s tid=0x00007f5783c05610 nid=0x1d592 waiting on condition  [0x00007f574481b000]
   java.lang.Thread.State: WAITING (parking)
	at jdk.internal.misc.Unsafe.park([email protected]/Native Method)
	- parking to wait for  <0x0000000614151cc8> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
	at java.util.concurrent.locks.LockSupport.park([email protected]/LockSupport.java:211)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:715)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire([email protected]/AbstractQueuedSynchronizer.java:938)
	at java.util.concurrent.locks.ReentrantLock$Sync.lock([email protected]/ReentrantLock.java:153)
	at java.util.concurrent.locks.ReentrantLock.lock([email protected]/ReentrantLock.java:322)
	at org.springframework.web.socket.handler.ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:251)
	at org.springframework.web.socket.messaging.StompSubProtocolHandler.sendErrorMessage(StompSubProtocolHandler.java:421)
	at org.springframework.web.socket.messaging.StompSubProtocolHandler.handleError(StompSubProtocolHandler.java:388)
	at org.springframework.web.socket.messaging.StompSubProtocolHandler.handleMessageFromClient(StompSubProtocolHandler.java:375)
	at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.handleMessage(SubProtocolWebSocketHandler.java:336)
	at org.springframework.web.socket.handler.WebSocketHandlerDecorator.handleMessage(WebSocketHandlerDecorator.java:75)
	at org.springframework.web.socket.handler.LoggingWebSocketHandlerDecorator.handleMessage(LoggingWebSocketHandlerDecorator.java:56)
	at org.springframework.web.socket.handler.ExceptionWebSocketHandlerDecorator.handleMessage(ExceptionWebSocketHandlerDecorator.java:58)
	at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.delegateMessages(AbstractSockJsSession.java:346)
	at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.handleMessage(WebSocketServerSockJsSession.java:195)
	at org.springframework.web.socket.sockjs.transport.handler.SockJsWebSocketHandler.handleTextMessage(SockJsWebSocketHandler.java:93)
	at org.springframework.web.socket.handler.AbstractWebSocketHandler.handleMessage(AbstractWebSocketHandler.java:43)
	at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter.handleTextMessage(StandardWebSocketHandlerAdapter.java:113)
	at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:84)
	at org.springframework.web.socket.adapter.standard.StandardWebSocketHandlerAdapter$3.onMessage(StandardWebSocketHandlerAdapter.java:81)
	at io.undertow.websockets.jsr.FrameHandler$7.run(FrameHandler.java:288)
	at io.undertow.websockets.jsr.ServerWebSocketContainer$1.call(ServerWebSocketContainer.java:170)
	at io.undertow.websockets.jsr.ServerWebSocketContainer$1.call(ServerWebSocketContainer.java:167)
	at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
	at io.undertow.websockets.jsr.ServerWebSocketContainer.invokeEndpointMethod(ServerWebSocketContainer.java:618)
	at io.undertow.websockets.jsr.ServerWebSocketContainer.invokeEndpointMethod(ServerWebSocketContainer.java:608)
	at io.undertow.websockets.jsr.FrameHandler.invokeTextHandler(FrameHandler.java:268)
	at io.undertow.websockets.jsr.FrameHandler.onFullTextMessage(FrameHandler.java:319)
	at io.undertow.websockets.core.AbstractReceiveListener$2.complete(AbstractReceiveListener.java:156)
	at io.undertow.websockets.core.AbstractReceiveListener$2.complete(AbstractReceiveListener.java:152)
	at io.undertow.websockets.core.BufferedTextMessage.read(BufferedTextMessage.java:105)
	at io.undertow.websockets.core.AbstractReceiveListener.readBufferedText(AbstractReceiveListener.java:152)
	at io.undertow.websockets.core.AbstractReceiveListener.bufferFullMessage(AbstractReceiveListener.java:90)
	at io.undertow.websockets.jsr.FrameHandler.onText(FrameHandler.java:184)
	at io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:44)
	at io.undertow.websockets.core.AbstractReceiveListener.handleEvent(AbstractReceiveListener.java:33)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:974)
	at io.undertow.server.protocol.framed.AbstractFramedChannel$FrameReadListener.handleEvent(AbstractFramedChannel.java:954)
	at org.xnio.ChannelListeners.invokeChannelListener(ChannelListeners.java:92)
	at org.xnio.conduits.ReadReadyHandler$ChannelListenerHandler.readReady(ReadReadyHandler.java:66)
	at org.xnio.nio.NioSocketConduit.handleReady(NioSocketConduit.java:89)
	at org.xnio.nio.WorkerThread.run(WorkerThread.java:591)

"SpringApplicationShutdownHook" #26 prio=5 os_prio=0 cpu=12.76ms elapsed=4469.15s tid=0x00007f53ec0023a0 nid=0x1ea32 in Object.wait()  [0x00007f53d7ffd000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
	at java.lang.Object.wait([email protected]/Native Method)
	- waiting on <no object reference available>
	at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:329)
	- locked <0x00000006140184a0> (a java.lang.Object)
	at io.undertow.server.protocol.framed.AbstractFramedStreamSinkChannel.awaitWritable(AbstractFramedStreamSinkChannel.java:313)
	at org.xnio.channels.Channels.flushBlocking(Channels.java:64)
	at io.undertow.websockets.jsr.WebSocketSessionRemoteEndpoint$BasicWebSocketSessionRemoteEndpoint.sendText(WebSocketSessionRemoteEndpoint.java:287)
	at org.springframework.web.socket.adapter.standard.StandardWebSocketSession.sendTextMessage(StandardWebSocketSession.java:215)
	at org.springframework.web.socket.adapter.AbstractWebSocketSession.sendMessage(AbstractWebSocketSession.java:108)
	at org.springframework.web.socket.sockjs.transport.session.WebSocketServerSockJsSession.writeFrameInternal(WebSocketServerSockJsSession.java:224)
	at org.springframework.web.socket.sockjs.transport.session.AbstractSockJsSession.close(AbstractSockJsSession.java:188)
	at org.springframework.web.socket.handler.WebSocketSessionDecorator.close(WebSocketSessionDecorator.java:160)
	at org.springframework.web.socket.handler.ConcurrentWebSocketSessionDecorator.close(ConcurrentWebSocketSessionDecorator.java:271)
	at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.stop(SubProtocolWebSocketHandler.java:289)
	at org.springframework.web.socket.messaging.SubProtocolWebSocketHandler.stop(SubProtocolWebSocketHandler.java:302)
	- locked <0x000000060e1b0518> (a java.lang.Object)
	at org.springframework.context.support.DefaultLifecycleProcessor.doStop(DefaultLifecycleProcessor.java:344)
	at org.springframework.context.support.DefaultLifecycleProcessor$LifecycleGroup.stop(DefaultLifecycleProcessor.java:483)
	at org.springframework.context.support.DefaultLifecycleProcessor$$Lambda$2269/0x00007f56f881b850.accept(Unknown Source)
	at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
	at org.springframework.context.support.DefaultLifecycleProcessor.stopBeans(DefaultLifecycleProcessor.java:313)
	at org.springframework.context.support.DefaultLifecycleProcessor.onClose(DefaultLifecycleProcessor.java:214)
	at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1136)
	at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.doClose(ServletWebServerApplicationContext.java:174)
	at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1090)
	at org.springframework.boot.SpringApplicationShutdownHook.closeAndWait(SpringApplicationShutdownHook.java:145)
	at org.springframework.boot.SpringApplicationShutdownHook$$Lambda$2262/0x00007f56f881a610.accept(Unknown Source)
	at java.lang.Iterable.forEach([email protected]/Iterable.java:75)
	at org.springframework.boot.SpringApplicationShutdownHook.run(SpringApplicationShutdownHook.java:114)
	at java.lang.Thread.run([email protected]/Thread.java:840)
@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Mar 14, 2024
@jhoeller jhoeller added the in: web Issues in web modules (web, webmvc, webflux, websocket) label Mar 18, 2024
@jhoeller jhoeller changed the title Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Spring 6.2.1 and Undertow 2.3.10 Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Spring 6.1.3 and Undertow 2.3.10 Apr 16, 2024
@jhoeller jhoeller self-assigned this Apr 23, 2024
@jhoeller jhoeller added type: bug A general bug and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Apr 23, 2024
@jhoeller jhoeller added this to the 6.1.7 milestone Apr 23, 2024
@github-actions github-actions bot added status: backported An issue that has been backported to maintenance branches and removed for: backport-to-6.0.x labels Apr 23, 2024
@github-actions github-actions bot mentioned this issue Apr 23, 2024
Closed
@jhoeller jhoeller changed the title Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Spring 6.1.3 and Undertow 2.3.10 Deadlock/Stall in ConcurrentWebSocketSessionDecorator with Undertow 2.3.10 Apr 23, 2024
@github-actions github-actions bot mentioned this issue Apr 23, 2024
Closed
jhoeller added a commit that referenced this issue Apr 23, 2024
@jhoeller
Skip close lock if acquired by other thread already
d9330bc 
Closes gh-32445

(cherry picked from commit d151931)
jhoeller added a commit that referenced this issue Apr 23, 2024
@jhoeller
Skip close lock if acquired by other thread already
6b6beec 
Closes gh-32445

(cherry picked from commit d151931)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jhoeller jhoeller

Labels
in: web Issues in web modules (web, webmvc, webflux, websocket) status: backported An issue that has been backported to maintenance branches type: bug A general bug
Projects
None yet
Milestone
6.1.7
Development

No branches or pull requests
3 participants
@chrisdennis @jhoeller @spring-projects-issues