System.InvalidOperationException at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported

[Tadimsky]

I'm using Microsoft.AspNetCore.Mvc.Versioning version 2.2.0 and recently started seeing this in my service logs:

Field	Value
Message	Operations that change non-concurrent collections must have exclusive access. A concurrent update was performed on this collection and corrupted its state. The collection's state is no longer correct.
Exception type	System.InvalidOperationException
Failed method	System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported

Stack trace:

System.InvalidOperationException:
   at System.ThrowHelper.ThrowInvalidOperationException_ConcurrentOperationsNotSupported (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.Dictionary`2.FindEntry (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.Dictionary`2.TryGetValue (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Abstractions.ActionDescriptorExtensions.GetProperty (Microsoft.AspNetCore.Mvc.Abstractions, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Abstractions.ActionDescriptorExtensions.IsImplicitlyMappedTo (Microsoft.AspNetCore.Mvc.Versioning, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Versioning.ApiVersionActionSelector.ActionIsSatisfiedBy (Microsoft.AspNetCore.Mvc.Versioning, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Linq.Enumerable+WhereSelectEnumerableIterator`2.MoveNext (System.Linq, Version=4.2.1.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a)
   at System.Collections.Generic.List`1.AddEnumerable (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Collections.Generic.List`1.InsertRange (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Mvc.Versioning.ApiVersionActionSelector.SelectBestActions (Microsoft.AspNetCore.Mvc.Versioning, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Versioning.ApiVersionActionSelector.SelectBestCandidate (Microsoft.AspNetCore.Mvc.Versioning, Version=2.2.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Mvc.Internal.MvcAttributeRouteHandler.RouteAsync (Microsoft.AspNetCore.Mvc.Core, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at Microsoft.AspNetCore.Routing.Tree.TreeRouter+<RouteAsync>d__13.MoveNext (Microsoft.AspNetCore.Routing, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Routing.RouteCollection+<RouteAsync>d__10.MoveNext (Microsoft.AspNetCore.Routing, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Builder.RouterMiddleware+<Invoke>d__4.MoveNext (Microsoft.AspNetCore.Routing, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.AspNetCore.Authentication.AuthenticationMiddleware+<Invoke>d__6.MoveNext (Microsoft.AspNetCore.Authentication, Version=2.1.0.0, Culture=neutral, PublicKeyToken=adb9793829ddae60)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Intentional.Services.Utilities.Logging.AuthenticationErrorLoggingMiddleware+<Invoke>d__3.MoveNext (Microsoft.Intentional.Services.Utilities.Logging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Intentional.Services.Utilities.Logging.UserAgentLoggingMiddleware+<Invoke>d__6.MoveNext (Microsoft.Intentional.Services.Utilities.Logging, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Microsoft.Intentional.Services.Utilities.Logging.CorrelationId.CorrelationIdMiddleware+<Invoke>d__4.MoveNext (Microsoft.Intentional.Services.Utilities.Logging.CorrelationId, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null)

Any ideas what could be causing this?

[commonsensesoftware]

I believe this is related to #306. Prior to 2.3.0, the design internally mutated the property bag associated with an action. In edge cases, this could result in various concurrency problems ranging from a pegged CPU to the error you're seeing. This only happens - sometimes, which is why it has taken so long for this to be detected.

The latest package (v2.3.0) addresses this issue and pre-aggregates all of the API version information such that the property bag is fully read-only in the pipeline. My recommendation would be to update to 2.3.0. Let me know if this issue persists post 2.3.0+ or you encounter another issue.

Thanks.

[Tadimsky]

Ok, great! I remember seeing a thread around this one.
I'm trying it out now and will see if it still happens in our INT environment. It did only show up for a bit and then went away.

[combl]

I can confirm that this is fixed now. Got the same exception with 2.2.0 in my logs. It disappeared completly after upgrading to 2.3.0.

[commonsensesoftware]

Great! Is this issue safe to close out now?

[Tadimsky]

I haven't seen this either after the upgrade 🙂

[commonsensesoftware]

It seems that this issue has been mitigated. Feel free to re-open the issue or create a new issue if you continue to run into problems. Thanks.

[benaadams]

Added an issue to corefx so its easier to find information about this exception in search https://github.com/dotnet/corefx/issues/31186