Quarantine RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted

[JamesNK]

Failing Test(s)

• Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2ConnectionTests.RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted

Error Message

Assert.Equal() Failure
Expected: GOAWAY
Actual:   HEADERS

Stacktrace

   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.VerifyGoAway(Http2Frame frame, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1264
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.WaitForConnectionErrorAsyncDoNotCloseTransport[TException](Boolean ignoreNonGoAwayFrames, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode, String[] expectedErrorMessage) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1292
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.WaitForConnectionErrorAsync[TException](Boolean ignoreNonGoAwayFrames, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode, String[] expectedErrorMessage) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1275
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2ConnectionTests.RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2ConnectionTests.cs:line 3423
--- End of stack trace from previous location ---

Logs

[0.001s] [TestLifetime] [Information] Starting test RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted at 2024-01-31T14:48:02
[0.010s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending SETTINGS frame for stream ID 0 with length 24 and flags NONE.
[0.010s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending WINDOW_UPDATE frame for stream ID 0 with length 4 and flags 0x0.
[0.011s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" received SETTINGS frame for stream ID 0 with length 6 and flags NONE.
[0.013s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending SETTINGS frame for stream ID 0 with length 0 and flags ACK.
[0.015s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" received HEADERS frame for stream ID 1 with length 29 and flags END_HEADERS.
[0.015s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending HEADERS frame for stream ID 1 with length 36 and flags END_STREAM, END_HEADERS.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "TestConnectionId", Request id "TestConnectionId:00000001": started reading request body.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel] [Debug] Connection id "TestConnectionId", Request id "TestConnectionId:00000001": done reading request body.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel] [Information] Connection id "TestConnectionId", Request id "TestConnectionId:00000001": the application completed without reading the entire request body.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending RST_STREAM frame for stream ID 1 with length 4 and flags 0x0.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" received DATA frame for stream ID 1 with length 1 and flags NONE.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" received RST_STREAM frame for stream ID 1 with length 4 and flags 0x0.
[0.016s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" received WINDOW_UPDATE frame for stream ID 1 with length 4 and flags 0x0.
[0.017s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Debug] Connection id "TestConnectionId": HTTP/2 connection error.
Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2ConnectionErrorException: HTTP/2 connection error (STREAM_CLOSED): A frame of type WINDOW_UPDATE was received after stream 1 was reset or aborted.
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessWindowUpdateFrameAsync() in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 1077
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessFrameAsync[TContext](IHttpApplication`1 application, ReadOnlySequence`1& payload) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 645
   at Microsoft.AspNetCore.Server.Kestrel.Core.Internal.Http2.Http2Connection.ProcessRequestsAsync[TContext](IHttpApplication`1 application) in /_/src/Servers/Kestrel/Core/src/Internal/Http2/Http2Connection.cs:line 303
[0.017s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Debug] Connection id "TestConnectionId" is closed. The last processed stream ID was 1.
[0.017s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Verbose] Connection id "TestConnectionId" sending GOAWAY frame for stream ID 0 with length 8 and flags 0x0.
[0.017s] [Microsoft.AspNetCore.Server.Kestrel.Http2] [Debug] The connection queue processing loop for TestConnectionId completed.
[0.038s] [Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2ConnectionTests] [Error] Test threw an exception.
Xunit.Sdk.EqualException: Assert.Equal() Failure
Expected: GOAWAY
Actual:   HEADERS
   at Xunit.Assert.Equal[T](T expected, T actual, IEqualityComparer`1 comparer) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 96
   at Xunit.Assert.Equal[T](T expected, T actual) in /_/src/xunit.assert/Asserts/EqualityAsserts.cs:line 63
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.VerifyGoAway(Http2Frame frame, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1264
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.WaitForConnectionErrorAsyncDoNotCloseTransport[TException](Boolean ignoreNonGoAwayFrames, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode, String[] expectedErrorMessage) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1292
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2TestBase.WaitForConnectionErrorAsync[TException](Boolean ignoreNonGoAwayFrames, Int32 expectedLastStreamId, Http2ErrorCode expectedErrorCode, String[] expectedErrorMessage) in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2TestBase.cs:line 1275
   at Microsoft.AspNetCore.Server.Kestrel.Core.Tests.Http2ConnectionTests.RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted() in /_/src/Servers/Kestrel/test/InMemory.FunctionalTests/Http2/Http2ConnectionTests.cs:line 3423
   at Xunit.Sdk.TestInvoker`1.<>c__DisplayClass48_0.<<InvokeTestMethodAsync>b__1>d.MoveNext() in /_/src/xunit.execution/Sdk/Frameworks/Runners/TestInvoker.cs:line 264
--- End of stack trace from previous location ---
   at Xunit.Sdk.ExecutionTimer.AggregateAsync(Func`1 asyncAction) in /_/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at Xunit.Sdk.ExceptionAggregator.RunAsync(Func`1 code) in /_/src/xunit.core/Sdk/ExceptionAggregator.cs:line 90
[0.039s] [TestLifetime] [Information] Finished test RST_STREAM_IncompleteRequest_AdditionalWindowUpdateFrame_ConnectionAborted in 0.0386109s

Build

https://dev.azure.com/dnceng-public/public/_build/results?buildId=547710&view=results

[amcasey]

This hasn't failed in 30 days.

[amcasey]

The failure mode appears to indicate that the server started to send a valid response even though it should have aborted the stream in response to seeing an incoming reset followed by an incoming window update.

It's not immediately obvious to me that all of the incoming frames have to be processed strictly before the response starts sending.

[amcasey]

The test fails when I add a delay at the start of Http2Connection.ProcessRstStreamFrameAsync, so there's a race between starting the response and aborting the connection.

[amcasey]

The easy way out would be to set ignoreNonGoAwayFrames: true and accept that there may be a HEADERS frame.