[BUG] EventExecutor exception

[askpt]

Description

We are currently experiencing some flaky unit tests that we should tackle. Example output:

Unhandled exception. System.Threading.Channels.ChannelClosedException: The channel has been closed.
   at OpenFeature.EventExecutor.ProcessFeatureProviderEventsAsync(Object providerRef) in /_/src/OpenFeature/EventExecutor.cs:line 238
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()
The active test run was aborted. Reason: Test host process crashed : Unhandled exception. System.Threading.Channels.ChannelClosedException: The channel has been closed.
   at OpenFeature.EventExecutor.ProcessFeatureProviderEventsAsync(Object providerRef) in /_/src/OpenFeature/EventExecutor.cs:line 238
   at System.Threading.Tasks.Task.<>c.<ThrowAsync>b__128_1(Object state)
   at System.Threading.ThreadPoolWorkQueue.Dispatch()
   at System.Threading.PortableThreadPool.WorkerThread.WorkerThreadStart()


Test Run Aborted.
Total tests: Unknown
     Passed: 90
 Total time: 2.8570 Seconds
     6>_VSTestConsole:
         MSB4181: The "VSTestTask" task returned false but did not log an error.
     6>Done Building Project "/home/runner/work/dotnet-sdk/dotnet-sdk/test/OpenFeature.Tests/OpenFeature.Tests.csproj" (VSTest target(s)) -- FAILED.

As part of the debugging, we can try using the dotnet test—- blame to understand what is wrong.

Notes

• https://learn.microsoft.com/en-us/dotnet/core/tools/dotnet-test
• https://github.com/open-feature/dotnet-sdk/actions/runs/13074052418/job/36481801840

[kinyoklion]

This looks to be a bug in the EventExecutor rather than just a flaky test. Writing to a channel will throw if the channel has been closed.

There is a purple note about it here: https://learn.microsoft.com/en-us/dotnet/core/extensions/channels#bounding-strategies

There isn't any synchronization between shutdown and the event processing loop.

            while (await reader.WaitToReadAsync().ConfigureAwait(false))
            {
                if (!reader.TryRead(out var item))
                    continue;

//!!!!!!!!!!!!!!! If you are executing any of the following, and ShutDownAsync is called or a providerr is registered, then you will get an exception.
                switch (item)
                {
                    case ProviderEventPayload eventPayload:
                        this.UpdateProviderStatus(typedProviderRef, eventPayload);
                        await this.EventChannel.Writer.WriteAsync(new Event { Provider = typedProviderRef, EventPayload = eventPayload }).ConfigureAwait(false);
                        break;
                }
            }

Theoretically this could happen in real situations and not jsut a test.

It seems the typical pattern is for the producer to close the channel after it has exhausted things it wants to produce.

So, typically I think that writes and completing the channel would be happening either on the same thread, or would be synchronized.

In our case though I think we could just handle the exception from the WriteAsync.

[askpt]

This looks to be a bug in the EventExecutor rather than just a flaky test. Writing to a channel will throw if the channel has been closed.

There is a purple note about it here: https://learn.microsoft.com/en-us/dotnet/core/extensions/channels#bounding-strategies

There isn't any synchronization between shutdown and the event processing loop.

            while (await reader.WaitToReadAsync().ConfigureAwait(false))
            {
                if (!reader.TryRead(out var item))
                    continue;

//!!!!!!!!!!!!!!! If you are executing any of the following, and ShutDownAsync is called or a providerr is registered, then you will get an exception.
                switch (item)
                {
                    case ProviderEventPayload eventPayload:
                        this.UpdateProviderStatus(typedProviderRef, eventPayload);
                        await this.EventChannel.Writer.WriteAsync(new Event { Provider = typedProviderRef, EventPayload = eventPayload }).ConfigureAwait(false);
                        break;
                }
            }

Theoretically this could happen in real situations and not jsut a test.

It seems the typical pattern is for the producer to close the channel after it has exhausted things it wants to produce.

So, typically I think that writes and completing the channel would be happening either on the same thread, or would be synchronized.

In our case though I think we could just handle the exception from the WriteAsync.

@kinyoklion Thanks for investigating this. I changed the title and description to better align with your findings.

[askpt]

I created this PR #393 to start cleaning up and fixing the bug in the EventExecutor. I will fix the bug in the following PR.

[askpt]

@toddbaert, @kinyoklion, @beeme1mr It seems after PR #393 the build tests are now stabilized. Can we close this issue?