rate-limiting: propagate back-pressure from queue as HTTP 429's

[yaauie]

Adds a proactive handler that rejects new requests with HTTP 429's when the queue has been blocking for more than 10 consecutive seconds, allowing back- pressure to propagate in advance of filling up the connection backlog queue.

While I'd like to take the time to break out the supplied ExecutionObserver into a separate library, I think it is better to roll with what we have than to hold back.

[yaauie]

Rebased and targeted to 3.x branch (for LS 8.x series), will need a forward-port to 4.x/main

[mashhurs]

Overall implementation details looks good to me (will run on my local to test)!
Some questions I would like to clarify:

• We have thread pool with maxPendingRequests. Is there possibility IMessageHandler#onNewMessage take much time (>10s) but no pending requests in thread pool? If so, involving pending request int logic would also make sense to me.
• I wonder if we need to make hard coded 10s configurable as an expert-only config. For very intensive applications 10s may even be long as my understanding.

[yaauie]

• We have thread pool with maxPendingRequests. Is there possibility IMessageHandler#onNewMessage take much time (>10s) but no pending requests in thread pool? If so, involving pending request int logic would also make sense to me.

They are two different safeguards, and do not need to be intertwined.

If the queue is full and a write to the queue is blocking, even a single request made before the block is lifted should result in a 429. The prior 429 safeguard only kicked in if there was a sufficient quantity of pending requests, and will still kick in at that threshold of concurrent pending requests regardless of blockage status. This is as-desired

• I wonder if we need to make hard coded 10s configurable as an expert-only config. For very intensive applications 10s may even be long as my understanding.

We can make it configurable later, if necessary. I don't see an immediate reason to make it configurable now, especially as making the option intuitive for a user to configure would be very difficult. The 10s threshold is meant to make it reactive enough in a fully-blocked state without causing superfluous 429's for intermittent blockages, especially since most connecting clients (unfortunately) don't use http's Expect: 100-continue and therefore would need to retransmit data.

[mashhurs]

LGTM! Thank you so much for this, it is very helpful.