Connections not being closed, even when server sets the "Connection: close" header

[0xpablo]

Hi there,

I'm trying to perform some requests to an embedded device in my local network (by IP address). I noticed that after a few requests, the device would stop responding (requests would time out). It looks like the device supports a limited number of concurrent connections / open sockets, so after going over the limit, requests time out.

The embedded device sets the Connection: close header, but looking inside the Active Connections panel in Xcode, I notice that the connections are not being closed:

The only workaround I found was to shutdown the HTTPClient used to make the request and create a new one for the next request.

I'm running macOS 11, and the issue happens on both release and debug builds. My code is very similar to the sample code on the README (I don't think I'm doing anything that would cause this issue).

Any ideas?

Thanks in advance!

---

Environment:

- AsyncHTTPClient commit hash: 947429beb4b3f7515bfe2f1d7bb721318cf30935
- Swift version: Apple Swift version 5.3.2 (swiftlang-1200.0.45 clang-1200.0.32.28) (Version 12.3 (12C33))
- OS: macOS 11.1

[Lukasa]

Do you have the full set of headers sent by the remote server? We do have code that should tear these down if Connection: close genuinely is being sent.

[0xpablo]

Sure! This is the output of a curl to that device:

curl -v  -X "POST" "http://192.168.1.116:3000/api/v1/hvac"  -d $'{
  "systemid": 1,
  "zoneid": 0
}'
Note: Unnecessary use of -X or --request, POST is already inferred.
*   Trying 192.168.1.116...
* TCP_NODELAY set
* Connected to 192.168.1.116 (192.168.1.116) port 3000 (#0)
> POST /api/v1/hvac HTTP/1.1
> Host: 192.168.1.116:3000
> User-Agent: curl/7.64.1
> Accept: */*
> Content-Type: application/json; charset=utf-8
> Content-Length: 34
> 
* upload completely sent off: 34 out of 34 bytes
< HTTP/1.1 200 OK
< Server: Airzone-Athome gateway
< Connection: close
< Access-Control-Allow-Origin: *
< Access-Control-Allow-Methods: PUT,POST,DELETE,PATCH
< Content-Type: text/html
< Content-Length: 1505
< 

[Lukasa]

Given that these are plaintext, can we get a packet capture of what HTTPClient is sending and receiving? HTTPClient sends different headers than curl, and so this may well be triggering different behaviour.

[weissi]

ping @0xpablo

[0xpablo]

Sorry for the delay @Lukasa and @weissi. I recorded the traffic like this: tcpdump dst 192.168.1.116 or src 192.168.1.116 -i en8 -w trace.pcap.

trace.pcap.zip

[Lukasa]

Yup, looks like something is going wrong here. I'll see if I can repro in an isolated environment.

[Lukasa]

And great, ok, this is a really nasty bug: we actually leak these connections. If I spin in a loop hitting a localhost server exhibiting the same behaviour as your server, we leak connections until eventually we exhaust the FD limit and crash.

This is normally not a problem because the server should be actually closing the connection after it sends Connection: close. The draft HTTP messaging RFC update expresses this well:

A server that sends a "close" connection option MUST initiate a close of the connection (see below) after it sends the response containing "close".

This server is failing to do that. We need to defend against that case, in any rate. The best system for us is to issue a socket closure ourselves eagerly. This addresses our own resource exhaustion issue, and also solves the problem here.

I'll throw a unit test together and get this fixed.

[0xpablo]

Great!!! Thanks for investigating and the quick fix, I will try pointing to that PR to confirm that the issue is fixed, and I will also try contacting the manufacturers of the embedded device to let them know that they should be closing the socket as well.