Should use HTTP keep alive to avoid port exhaustion

[awarecan]

Meta -

OS: Windows 7, 10
Selenium Version: 3.8.1
Browser: Chrome
Browser Version: 65.0.3325.181 (64-bit)
Node.js: 8.11.1

Expected Behavior -

During the test, TCP connection should be reused

Actual Behavior -

Each WebDriver command will create a new TCP connection to ChromeDriver

Steps to reproduce -

await browser.get('https://www.google.com/');
for (let i=0; i<50000; i++) {
  await browser.getCurrentUrl();
}

use netstat -na to monitor the number of TCP connection in TIME_WAIT state while you are running above code. You will see massive amount of TCP connections. If you are using a powerful box and has default Windows TCP/IP configuration, you will hit ephemeral port exhaustion very soon.

Note -

This was a well known issue related with ChromeDriver. However ChromeDriver has a dirty fix in 2.35 release. It requires client set Connection: keep-alive HTTP header, see the discussion in #4162 and #3457

Now, we finally have chance to resolve this issue in selenium-webdriver side by passing in an Agent with keepAlive option while creating HttpClient in createExecutor() function.

[p0deje]

Seems like we need to do this for all the bindings.

[barancev]

In Java I can't reproduce this issue with chromedriver, but the number of TIME_WAIT'ing ports grows if I use geckodriver.

[dmytro-verner]

@barancev possibly you have registry's HKLM\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters (in case of Windows) TcpTimedWaitDelay set to non-default value(the default one is 240 secs).

Setting this value to a lower bound(e.g. 30 secs) can help to circumvent the current issue.

[barancev]

It appears that OkHttp uses keep-alive by default, so it's not an issue with Java binding.

[p0deje]

Since there is no straightforward way to use keep-alive connections in standard Ruby library, I think we'll just stick to existing solution described in wiki. One needs to install net-http-persistent gem and use Persistent HTTP client. We might consider making it a runtime dependency and use by default, but that's subject for discussion.

[ariasuni]

Are there any known workaround for the Python bindings?

[lmtierney]

@ariasuni you can currently set keep_alive on a Remote webdriver instance. Currently it's not in the individual drivers, but you could always start up the driver binary yourself and use Remote to connect to it as a workaround. Which drivers do we know support keep alive? We can add it to those xtors.

selenium/py/selenium/webdriver/remote/webdriver.py

Line 127 in 2c2db8a

- keep_alive - Whether to configure remote_connection.RemoteConnection to use

[ariasuni]

Unfortunately, your solution is more a refactoring, and it would require me to modify a ton of production code which I can’t do.

It worked perfectly fine for me with 2.53.* but not any 3.*, thus I can’t upgrade and use recent versions of Firefox or Chromium.

[awarecan]

Python should support ChromeDriver keep-alive option already

selenium/py/selenium/webdriver/chrome/remote_connection.py

Line 23 in 9090f52

def __init__(self, remote_server_addr, keep_alive=True):

@ariasuni I don't know why @barancev removed ChromeDriver from my issue subject. As I explained in the beginning, this issue is ChromeDriver specific. If you have problem with Firefox, I suggest you open a new issue to tracking.

[ariasuni]

Well, I have this problem with PhantomJS but I guess that it won’t be resolved if it’s specific to this webdriver.

[secondcircle]

Anyone know if there is a fix for chromedriver with C#? I'm running into this same port exhaustion issue only with chromedriver.

[awarecan]

@secondcircle Could you double check your chromeDriver version? The keep-live feature is only available on 2.35 and after.

From the code I read, RemoteDriver should support keep-alive by default, and ChromeDriver just a simple extension of RemoteDriver

selenium/dotnet/src/webdriver/Remote/DriverServiceCommandExecutor.cs

Line 37 in 894b67a

public DriverServiceCommandExecutor(DriverService driverService, TimeSpan commandTimeout)

[barancev]

@awarecan I've expanded the issue from python to other bindings too. At the same time I've removed "browser"-label because we only can fix client bindings. But this will have effect only for the drivers that provide support for keep-alive. It seems that chromedriver supports it, so we can use this driver to test client bindings.

If python binding has implemented keep-alive support we can remove this label from this issue, but I want to get a confirmation from the python part of the team. The same for C#.

[awarecan]

@secondcircle I realized the dotNet binding issue now. I actually point to the right issue in the beginning of this thread, please check #4162 that is about dotNet binding and chromeDriver 2.35

@barancev I don't have time to check what happens in python and C# language binding, I bet #4162 points to the root cause, the chromeDriver 2.35's implementation may have issue with the HTTP 1.1 client, have to set Connection: keep-alive header explicitly to make it works

[awarecan]

To whom may have interesting, mozilla/geckodriver#713 is the discussion in geckodriver project

[lmtierney]

As stated, keep_alive is already used for Chrome in python, removing label.