Webdriver opens thousands of connections

[ArturSpirin]

Meta -

OS: Windows 7 64-Bit

Selenium Version:
Name: selenium
Version: 3.0.2
Summary: Python bindings for Selenium
Home-page: https://github.com/SeleniumHQ/selenium/
Author: UNKNOWN
Author-email: UNKNOWN
License: UNKNOWN
Location: c:\python27\lib\site-packages

Browser: Chrome driver

Browser Version: 55.0.2883.87 m (64-bit)

Expected Behavior -

Webdriver creates one connection and reuses it to send commands to the driver

Actual Behavior -

There is a new connection being created every time webdriver is doing something in the browser. The browser object does not get recreated, it just creates a new connection and carries on but eventually there are no more free ports available to create new connection and the program dies

Steps to reproduce -

Summary:
Driver object gets created and then being mapped to a thread that will run the tests. after that the thread is started and tests start running. As tests run and I do netstat -a -n | find /c "ip:port" the count of open connection on the port that webdriver started on is rapidly increasing and can go up into 16k range and then shortly after everything crashes.

This is the function that returns a driver object:

@staticmethod
    def get_driver(browser_id):
        """
        This method will return new instance of a webdriver depending on the browser_id that was passed to it
        @id browser_id
        """
        if browser_id == 0:
            return webdriver.PhantomJS()
        elif browser_id == 1:
            return webdriver.Chrome()
        elif browser_id == 2:
            return webdriver.Firefox()
        elif browser_id == 3:
            return webdriver.Opera()
        elif browser_id == 4:
            return webdriver.Ie()
        else:
            raise Exception("There is no support for browser_id: {}".format(browser_id))

This is the method that maps the thread to the driver:

@staticmethod
    def map(thread, driver, driver_id):
        """
        This method explicitly maps thread to driver object
        :param thread: thread object ex. threading.current_thread()
        :param driver: WebDriver instance of the object ex. webdriver.Chrome()
        :param driver_id: int from 0 to 4 (See @Runner#get_driver() for more documentation of browser/driver IDs)
        :return: None
        """
        Browser.__driver_map[thread] = {"driver": driver, "driver_id": driver_id}
        MyLogger.log.info("Mapped thread: {} to driver: {}.\n\tNew Map Object: {}".format(thread, driver,
                                                                                          Browser.__driver_map))

And finally this is the method that gets called to get the driver object in order to do stuff in the browser:

 @staticmethod
    def get_driver():
        """
        This method can be called from any thread without any arguments, as long as the thread was mapped to the driver
        during the initialization of the test cycle, the caller will receive proper driver object depending on the
        thread that it was called from
        :return:
        """
        thread = threading.current_thread()
        Browser.CALLS += 1
        MyLogger.log.debug(">>> Calls: {}".format(Browser.CALLS))
        return Browser.__driver_map[thread]["driver"]

[ragePowered]

@ArturSpirin I strongly believe that there is some architectural flaw in your code, cuz parallel testing works just fine. Please check how many times get_driver(browser_id) is invoked. Also you didn't mention version of your ChromeDriver

[ArturSpirin]

Hey, the chromedriver version is 2.27. I also tried 2.26 and same thing is happening. The driver object gets created once, and then its just retrieved from the map as needed.

The code seems to be straight forward I'm really confused why this is happening. I had the code reviewed by other people and they can't seem to find anything wrong with it neither.

[cgoldberg]

@ArturSpirin can you fix your formatting and indentation and post a minimal full program that displays this issue

[ragePowered]

@ArturSpirin, please provide info requested by @cgoldberg in a gist

[cgoldberg]

typically one quits the driver after each test and launches a new browser

[ArturSpirin]

This is the main class that gets called to start the tests. It has a static function that returns a driver object, and that object is being mapped to the thread that will run the test suite on that driver object.

import threading
import unittest
from optparse import OptionParser

import time
from selenium import webdriver

from flow_test.brightedge.client.util import logger
from new_flow_test.pages.BasePage import Page
from new_flow_test.test.TestInfo import TestInfo
from new_flow_test.utils.BrowserManager import Browser
from new_flow_test.utils.MyLogger import MyLogger
from new_flow_test.utils.UiObject import UiObject


class Runner:
    """
    Runner is the main entry for executing test cycles
    All of the class variables are intended for debugging purposes
    """

    SUITES = ["Setup Keywords"]
    THREADED = True
    BROWSER_ID = 1
    CADENCE = 0
    ENVIRONMENT = 1
    WAIT_THRESHOLD = 10
    LOG_LEVEL = 10
    UNITTEST_VERBOSITY = 2

    def __init__(self):
        """
        The init method is responsible for providing and parsing command line arguments
        Those arguments will be used to start appropriate test suites on specified browser
        """
        # Debug flag provides option to start the tests directly from the IDE
        debug = True

        if not debug:

            parser = OptionParser()

            parser.add_option("-s", "--suites", dest="suites",
                              help="List of suites to run. Ex. --suites 'suite_name1, suite_name2, suite_name3']. "
                                   "Suite names can be obtained from respective python scripts", type=str, default=[])

            parser.add_option("-t", "--threaded", dest="threaded", default=True,
                              help="If you want this cycle to be milti-threaded", type=str)

            parser.add_option("-b", "--browser_id", dest="browser_id", default=1,
                              help="Which Browser you want to use for testing. 0=PhantomJS, 1=Chrome, 2=FF, 3=Opera, "
                                   "4=IE", type=int)

            parser.add_option("-c", "--cadence_id", dest="cadance_id", help="Test cadence. Weekly=1, Daily=0", type=int,
                              default=1)

            parser.add_option("-e", "--environment_id", dest="environment_id",
                              help="Environment to use for testing. Production=0, Staging=1", type=int, default=1)

            parser.add_option("-w", "--wait_threshold", dest="wait_threshold",
                              help="Specifies the max timeout threshold, in seconds, that will be used to wait for "
                                   "web elements and AJAX calls", type=int, default=10)

            parser.add_option("-l", "--log_level", dest="log_level",
                              help="Specifies logging level. debug=10, info=20, warn=30, error=40, critical=50, ",
                              type=int, default=10)

            (self.options, args) = parser.parse_args()

            Runner.SUITES = list(self.options.suites.replace(" ", "").split(","))
            Runner.THREADED = bool(self.options.threaded)
            Runner.BROWSER_ID = self.options.browser_id

        TestInfo.CADENCE = Runner.CADENCE if debug else self.options.cadence_id
        Page.ENVIRONMENT = Runner.ENVIRONMENT if debug else self.options.environment_id
        UiObject.WAIT_THRESHOLD = Runner.WAIT_THRESHOLD if debug else self.options.wait_threshold
        Page.DOMAIN = "brightedge.com" if Runner.ENVIRONMENT == 0 else "staging.brightedge.com"

        self.start_test_cycle()

    @staticmethod
    def get_driver(browser_id):
        """
        This method will return new instance of a webdriver depending on the browser_id that was passed to it
        @id browser_id
        """
        if browser_id == 0:
            return webdriver.PhantomJS()
        elif browser_id == 1:
            return webdriver.Chrome()
        elif browser_id == 2:
            return webdriver.Firefox()
        elif browser_id == 3:
            return webdriver.Opera()
        elif browser_id == 4:
            return webdriver.Ie()
        else:
            raise Exception("There is no support for browser_id: {}".format(browser_id))

    def start_test_cycle(self):
        """
        This is a main method that is responsible for figuring out what tests to start and how to start them,
        it will rely on the cmd line arguments to do so.
        """
        from new_flow_test.test.primers.Feature import Feature
        if len(Runner.SUITES) == 0:
            MyLogger.log.info("No test suites were given")
            for suite in Feature.SUITES:
                MyLogger.log.debug("Adding suite: {} for testing".format(suite))
                Runner.SUITES.append(suite)

        if Runner.THREADED and len(Runner.SUITES) > 1:
            threads = []
            for name in Runner.SUITES:
                if name in Feature.SUITES:
                    MyLogger.log.info("Starting Suite: {} in new thread.".format(Feature.SUITES[name]))
                    test_suite = unittest.TestLoader().loadTestsFromTestCase(Feature.SUITES[name])
                    thread = threading.Thread(target=self.run_suite, args=(test_suite,))
                    Browser.map(thread, self.get_driver(Runner.BROWSER_ID), Runner.BROWSER_ID)
                    thread.start()
                    threads.append(thread)
                    time.sleep(5)
            for thread in threads:
                thread.join()
        else:
            for name in Runner.SUITES:
                if name in Feature.SUITES:
                    test_suite = unittest.TestLoader().loadTestsFromTestCase(Feature.SUITES[name])
                    Browser.map(threading.currentThread(), self.get_driver(Runner.BROWSER_ID), Runner.BROWSER_ID)
                    self.run_suite(test_suite)
    def run_suite(self, suite):
        unittest.TextTestRunner(verbosity=Runner.UNITTEST_VERBOSITY).run(suite)

Runner()

Here is the Browser class that contains the map of threads to driver objects. and it contains a static function that returns the driver object based on the thread that called it.

class Browser:
    """
    Browser class is responsible for controlling the physical browser and keep track of what threads have
    dibs on what browser object
    """

    CALLS = 0
    __driver_map = {}

    @staticmethod
    def get_driver():
        """
        This method can be called from any thread without any arguments, as long as the thread was mapped to the driver
        during the initialization of the test cycle, the caller will receive proper driver object depending on the
        thread that it was called from
        :return:
        """
        thread = threading.current_thread()
        Browser.CALLS += 1
        MyLogger.log.debug(">>> Calls: {}".format(Browser.CALLS))
        return Browser.__driver_map[thread]["driver"]

    @staticmethod
    def map(thread, driver, driver_id):
        """
        This method explicitly maps thread to driver object
        :param thread: thread object ex. threading.current_thread()
        :param driver: WebDriver instance of the object ex. webdriver.Chrome()
        :param driver_id: int from 0 to 4 (See @Runner#get_driver() for more documentation of browser/driver IDs)
        :return: None
        """
        Browser.__driver_map[thread] = {"driver": driver, "driver_id": driver_id}
        MyLogger.log.info("Mapped thread: {} to driver: {}.\n\tNew Map Object: {}".format(thread, driver,
                                                                                          Browser.__driver_map))

The Browser.get_driver() is then used everywhere else in the code to get the driver. Runner.get_driver() is not called anywhere else besides the line in the Runner class itself: Browser.map(thread, self.get_driver(Runner.BROWSER_ID), Runner.BROWSER_ID)

[ArturSpirin]

@ragePowered, @cgoldberg Any help would be really appreciated, I can't even finish running one test right now.
If you log the port in the .services for Service().command_line_args() like so:

print ">>> Port: {}".format(self.port)

... and then do:

netstat -a -n   | find /c "<port_number>"

...it will give you count for all of the connections on that port right now
It seems like every single operation in the browser spawns a new connection, yet browser object is the same.

[cgoldberg]

@ArturSpirin what is the actual error you get when you say "everything crashes"?

Also, why don't just use an existing parallel test runner like pytest xdist or nose parallel?

[ArturSpirin]

======================================================================
ERROR: test_add_kw_flow_existing_KWG (new_flow_test.test.suites.SetupKeywords.SetupKeywords)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\eclipse\workspace\qa\new_flow_test\test\suites\SetupKeywords.py", line 65, in test_add_kw_flow_existing_KWG
    SetupKeywords.BRIGHTEDGE.add_keywords_to_existing_group([keyword], 1)
  File "C:\eclipse\workspace\qa\new_flow_test\utils\BrightEdge.py", line 24, in add_keywords_to_existing_group
    StoryBuilder() \
  File "C:\eclipse\workspace\qa\new_flow_test\pages\podbased\base\tables\modules\AddKeywords.py", line 20, in click_add_keywords
    AddKeywords.ADD_KEYWORDS_BUTTON.click()
  File "C:\eclipse\workspace\qa\new_flow_test\utils\UiObject.py", line 95, in click
    UiObject.                          wait_for_ajax_calls(wait, driver)
  File "C:\eclipse\workspace\qa\new_flow_test\utils\UiObject.py", line 229, in wait_for_ajax_calls
    if driver.execute_script("return BrightedgeInternal.Requests.ready_for_testing"):
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 465, in execute_script
    'args': converted_args})['value']
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\webdriver.py", line 234, in execute
    response = self.command_executor.execute(driver_command, params)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 408, in execute
    return self._request(command_info[0], url, body=data)
  File "C:\Python27\lib\site-packages\selenium\webdriver\remote\remote_connection.py", line 439, in _request
    self._conn.request(method, parsed_url.path, body, headers)
  File "C:\Python27\lib\httplib.py", line 1053, in request
    self._send_request(method, url, body, headers)
  File "C:\Python27\lib\httplib.py", line 1093, in _send_request
    self.endheaders(body)
  File "C:\Python27\lib\httplib.py", line 1049, in endheaders
    self._send_output(message_body)
  File "C:\Python27\lib\httplib.py", line 893, in _send_output
    self.send(msg)
  File "C:\Python27\lib\httplib.py", line 855, in send
    self.connect()
  File "C:\Python27\lib\httplib.py", line 832, in connect
    self.timeout, self.source_address)
  File "C:\Python27\lib\socket.py", line 575, in create_connection
    raise err
error: [Errno 10048] Only one usage of each socket address (protocol/network address/port) is normally permitted

@cgoldberg That's the stack I get when "everything crashes"

[ArturSpirin]

@cgoldberg @ragePowered you guys still there.

Try this script. I get 13 open connection when this script is done running. Add more steps and watch the connections go up via netstat -a -n | find /c "<port_number>" If you have a more advanced script that check a bunch of stuff on the page, you can imagine those connection will spike pretty fast. Im getting 16k connectiong in about 1.5 minutes when my actual script runs.

driver = webdriver.Chrome()
driver.get("https://www.google.com/")
search_box = driver.find_element_by_id("lst-ib")
search_box.send_keys("testing webdriver")
search_button = driver.find_element_by_id("_fZl")
search_box.send_keys("testing webdriver")
search_button.click()
search_box.send_keys("testing webdriver")
search_button.click()
search_box.send_keys("testing webdriver")

[ArturSpirin]

@cgoldberg @ragePowered - Any updates, are you able to reproduce?

[ArturSpirin]

For anyone who is running into the same issue on Windows, I was able to "solve" it temporarily by setting TcpTimedWaitDelay value in HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters\ to 30 as decimal. Basically it terminates closed TCP connection after 30 seconds instead of 4 minutes (which is the default on Windows).

This is not a very good solution so I'm still waiting for the fix from the Selenium team. But at least for now I can run up to two test suites in parallel and keep the connections, in most cases, under 10k.

CC @cgoldberg , @ragePowered

[cgoldberg]

no idea... I haven't seen any details that actually point to an issue with selenium.

[ArturSpirin]

@cgoldberg run this script:

driver = webdriver.Chrome()
driver.get("https://www.google.com/")
search_box = driver.find_element_by_id("lst-ib")
search_box.send_keys("testing webdriver")
search_button = driver.find_element_by_id("_fZl")
search_box.send_keys("testing webdriver")
search_button.click()
search_box.send_keys("testing webdriver")
search_button.click()
search_box.send_keys("testing webdriver")

Once you run it, check the connection on the port selenium used to create the driver. There should be 1 connection. But you will find 13! Run netstat -a -n | find /c "<port_number>" to count the connections on the given port. Have you tried it? If you did not try it, of course you wont see any issues.

If if you just do driver = webdriver.Chrome() and check connections, there will be 5 tcp connection in a closed state!

[cgoldberg]

If if you just do driver = webdriver.Chrome() and check connections,
there will be 5 tcp connection in a closed state!

what's wrong with that?

[ArturSpirin]

selenium is spawning a TCP connection every time it does anything, it does not reuse any of its connections. That is whats wrong.

[cgoldberg]

selenium is spawning a TCP connection every time it does anything

correct. if you are running into ephemeral port exhaustion, you probably need to tune your TCP/IP stack.

[ArturSpirin]

So when you run the script on your system, you end up with 1 connection?

[ArturSpirin]

@cgoldberg @ragePowered I've tried the same script on a linux box, fresh install - same thing happening. Connections are not being reused.

[cgoldberg]

same thing happening.
Connections are not being reused.

@ArturSpirin , selenium will reuse connections if the driver it is talking to supports keep-alives. However, chromedriver doesn't support keep-alives, so a new TCP connection is spawned for each HTTP request sent to chromedriver server. Since selenium is already doing the right thing, there is nothing to be fixed in the selenium python bindings.

• https://bugs.chromium.org/p/chromedriver/issues/detail?id=1529
• https://chromium.googlesource.com/chromium/src/+/master/chrome/test/chromedriver/server/chromedriver_server.cc#116

this issue can be closed.

[lmtierney]

Thank you for looking into this @cgoldberg

[ArturSpirin]

Copy that, thanks for the update @cgoldberg