Skip to content

Report last connection error if request deadline is exceeded #601

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jul 1, 2022

Conversation

dnadoba
Copy link
Collaborator

@dnadoba dnadoba commented Jun 24, 2022

If the request deadline is exceeded, we should provide as much information as possible to let the user know the underlying reason why the deadline was exceeded. We should therefore report the last connection error instead of HTTPClientError.deadlineExceeded if the request was never executed because of a connection error.

Modification

Do not fail the request in the RequestBag if canceled. Instead, let the connection pool state machine fail the request as it has access to the last connection error.

Result

Instead of HTTPClientError.deadlineExceeded we now get a connection error if the request couldn't be scheduled because of a connection error.

If a user manually cancels a request, we currently still just report HTTPClientError.cancelled. However, we could now easily change the behavior and also report the last connection error if we decided to do so.

async/await does not use RequestBag but Transaction and still has the old behaviour. I will apply similar changes to Transaction in a follow up PR.

@dnadoba dnadoba added the 🔨 semver/patch No public API change. label Jun 24, 2022
@Lukasa
Copy link
Collaborator

Lukasa commented Jun 24, 2022

Your PR comment implies that if a user manually cancels we still report .cancelled, but the code seems to have functionality that diverges from that. Can you clarify what's happening there?

@dnadoba
Copy link
Collaborator Author

dnadoba commented Jun 27, 2022

It's a bit hidden but we only report HTTPClientError.cancelled if the user canceled the request manually. We currently only go into the new .cancelledWhileQueued state if the cancellation reason was .deadlineExceeded:

case .queued(let queuer) where reason == .deadlineExceeded:

I guess if we don't want to report connection errors if the user cancels the request manually I will refactor the code a bit and make it more obvious what is going on. The code is currently a bit too generic and doesn't make it obvious that we actually only report connection errors if the cancellation reason was .deadlineExceeded.

dnadoba added 2 commits June 27, 2022 13:27
… level errors if the deadline was exceeded and not if the user cancels the request manually
@dnadoba dnadoba requested a review from Lukasa June 27, 2022 16:20
@dnadoba
Copy link
Collaborator Author

dnadoba commented Jun 27, 2022

@Lukasa I have removed CancelationReason and instead added a new method/action deadlineExceeded() to make it more explicit that we only get the new behaviour for deadlineExceeded and not manual user cancelation.

@Lukasa Lukasa requested a review from fabianfett June 28, 2022 11:45
Copy link
Collaborator

@Lukasa Lukasa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, I think I'm happy but I'd like @fabianfett to review.

Copy link
Member

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall looks great. We should remove the .cancelRequestTimeout Action and make sure we handle the race condition correctly.

@@ -323,9 +323,10 @@ extension HTTPConnectionPool {

mutating func cancelRequest(_ requestID: Request.ID) -> Action {
// 1. check requests in queue
if self.requests.remove(requestID) != nil {
if let request = self.requests.remove(requestID) {
let error = self.lastConnectFailure ?? HTTPClientError.cancelled
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a comment here, why we look into the lastConnectFailure.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In theory we could check here if we are at max connections... In those cases we could issue a more explicit: deadlineExceededBecauseToQueueDepthToLong error.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure how useful this kind of error message would be because deadlineExceededBecauseToQueueDepthToLong may only be a symptom as other requests in front could not be scheduled because there was a connection failure.

@@ -31,6 +31,8 @@ extension RequestBag {
fileprivate enum State {
case initialized
case queued(HTTPRequestScheduler)
/// if the deadline was exceeded while in the `.queued(_:)` state, we wait until the request pool fails the request with a potential more descriptive error message if a connection failure has occured while the request was queued.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NIT: can we wrap the comment here? I think normally I wrapped at about 100 characters.

return .init(
request: .cancelRequestTimeout(requestID),
request: .failRequest(request, error, cancelTimeout: true),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since, we remove .cancelRequestTimeout here and in the http2 state machine, I think we can remove this action completely.

mutating func deadlineExceeded() -> DeadlineExceededAction {
switch self.state {
case .queued(let queuer):
self.state = .deadlineExceededWhileQueued
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would love a comment here, why we do this dance. Please also make explicit that we depend on the scheduler to fail this request. What happens, however if we think we are still queued, but we aren't actually? There might be a race here:

  1. request hit's request's deadline (thread a)
  2. connection becomes available (thread b)
  3. request is moved from pool queue to connection (thread b)
  4. request depends on pool to be cancelled (thread a), but pool ignores request since it is on the connection already

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this we def. need a test case as well.

Comment on lines 100 to 101
case .finished(error: .some), .deadlineExceededWhileQueued:
return false
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll need to fail the request here explicitly...

case .queued(let queuer):
self.state = .deadlineExceededWhileQueued
return .cancelScheduler(queuer)
default:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we make all the other cases explicit here?

Copy link
Member

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hupps. Wrong state selected before.

Copy link
Member

@fabianfett fabianfett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks awesome! Thanks!

@dnadoba dnadoba merged commit 14fa6d9 into swift-server:main Jul 1, 2022
@dnadoba dnadoba deleted the dn-deadline-connection-error branch July 1, 2022 08:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🔨 semver/patch No public API change.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants