Better protect against concurrent error handling for async requests #32340
Assignees
Labels
in: web
Issues in web modules (web, webmvc, webflux, websocket)
status: backported
An issue that has been backported to maintenance branches
type: bug
A general bug
Milestone
Comments
rstoyanchev
added
in: web
github-actions
bot
added
status: backported
This was referenced Feb 28, 2024
rstoyanchev
added a commit
that referenced
this issue
Mar 3, 2024
The wrapped response prevents use after AsyncListener onError or completion to ensure compliance with Servlet Spec 2.3.3.4. The wrapped response is applied in RequestMappingHandlerAdapter. The wrapped response raises AsyncRequestNotUsableException that is now handled in DefaultHandlerExceptionResolver. See gh-32340
rstoyanchev
added a commit
that referenced
this issue
Mar 3, 2024
1. Use state transitions 2. Increase synchronized scope in setConcurrentResultAndDispatch See gh-32340
rstoyanchev
added a commit
that referenced
this issue
Mar 4, 2024
In addition to using the ServletOutputStream, it's also possible to call ServletResponse#flushBuffer, so the ServletOutputStream wrapper logic needs to apply there as well. See gh-32340
rstoyanchev
added a commit
that referenced
this issue
Mar 5, 2024
rstoyanchev
added a commit
that referenced
this issue
Mar 7, 2024
webmvc.fn now also uses the StandardServletAsyncWebRequest wrapped response to enforce lifecycle rules from Servlet spec (section 2.3.3.4). See gh-32340
bclozel
added a commit
that referenced
this issue
May 13, 2024
Prior to this commit, the `ReactiveTypeHandler` would handle `Flux`-like return types from controller methods and adapt them to SSE streams using the `SseEmitter`/`ResponseBodyEmitter` APIs. In case an `IOException` is thrown while writing to the HTTP response stream, the `ReactiveTypeHandler` would rely on the Servlet container to call `AsyncListener#onError` - this would be the signal for Spring MVC to complete the async exchange. To prevent racing issues between this signal and the actual handling of the exception, changes like gh-20173 were applied. Since then, robust checks were added with gh-32340 in `StandardServletAsyncWebRequest.LifecycleHttpServletResponse`. With Jetty 12, `AsyncListener#onError` would not be called as the error would happen while writing in blocking mode to the response (so, not using the Servlet WriteListener contract). But still, such `IOException` would still result in the closing of the HTTP connection. As of Jetty 12.0.4, this is no longer the case and the party managing the async lifecycle is in charge of completing the exchange, as it should. This means that the current behavior leaks HTTP connections for these cases and causes memory issues. This commit ensures that such exceptions happening during response writes are caught and result in the completion of the `SSEEmitter` and the closing of the exchange. Even if other Servlet containers still propagate the error `AsyncListener#onError`, competing signals are still managed with gh-32340. Closes gh-32629
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Section 3.2.2.4 of the Servlet spec indicates the request and response objects are not guaranteed to be thread safe, and the application must ensure that access to the request and response objects are thread safe. This is reasonably straight forward for applications to handle, if they even write concurrently, which may not be the case. However, the Servlet container may also trigger an
AsyncListener#onErrornotification at any time as a result of network issues, which Spring MVC delegates toCallableProcessingInterceptors andDeferredProcessingInterceptors before performing an ASYNC dispatch for final handling of the error.The
onErrorcomes on a Servlet container thread and may compete with the application write that would also fail. Currently,WebAsyncManageris not sufficiently well protected. For example, the failed write may be first to set the concurrentResult, and soon after perform an ASYNC dispatch. MeanwhileonErrormay return quickly, noticing the result is set and not taking any action in which case the Servlet container performs an ERROR dispatch that would race with the ASYNC dispatch. Another possibility is thatonErroris first to set the concurrentResult, which is later cleared during handling in the resulting dispatch, and that could trigger an additional ASYNC dispatch if the write fails after that.The recently reported #32042 is related to such a race. The fix prevented an ASYNC dispatch if the error implies a disconnected client. However, that depends on the exception being recognized as such (also not wrapped by the application), and also precludes any ASYNC dispatch for final error handling where at least one should go through. We need a more conclusive fix for this.
In addition to that, the spec requires the request and response objects to only be accessed within the object’s life cycle. So as part of this work, we should also ensure compliance with this rule by wrapping the
ServletOutputStreamto reject further use, rather than expect all applications to synchronize withonErrornotifications in order to be compliant with the rule.The text was updated successfully, but these errors were encountered: