SeleniumBase Driver Initialization Fails with Gunicorn + Nginx, but Works Manually with Flask

[bhautik-mangukiya]

Description:

I am experiencing an issue where the SeleniumBase Driver initialization works perfectly when the code is executed manually using python3, or when Flask is run manually. However, when the Flask app is integrated with Gunicorn and Nginx for automatic startup, the driver initialization fails with the following error:

Error loading driver: Message: session not created: cannot connect to chrome at 127.0.0.1:9222
from chrome not reachable

The issue arises when trying to initialize the Driver or SB (SeleniumBase) with undetectable Chrome in headless mode.

---

Steps to Reproduce:

1. Create a Python script (test.py):

   from seleniumbase import Driver
   import logging
   
   logger = logging.getLogger(__name__)
   
   def main():
       try:
           driver = Driver(uc=True, headless=True, no_sandbox=True, block_images=True)
           logger.info('Driver defined')
           driver.get("https://example.com/")
           logger.info('Driver opened')
           title = driver.title
           driver.quit()
           return {"title": title}
       except Exception as e:
           return {"error": str(e)}
   
   if __name__ == "__main__":
       print(main())

2. Run the script manually:

   • Execute the script with python3 test.py.
   • The script runs as expected and fetches the webpage title.

3. Add the same functionality in a Flask app:

   from flask import Flask, request
   from seleniumbase import SB
   import logging
   
   logger = logging.getLogger(__name__)
   app = Flask(__name__)
   
   @app.route("/fetch", methods=["POST"])
   def fetch_title():
       data = request.json
       review_page_url = data.get('query')
       try:
           logger.info("Loading driver...")
           try:
               with SB(uc=True, xvfb=True, headless2=True) as sb:
                   logger.info("Display started successfully.")
                   sb.driver.get(review_page_url)
                   logger.info("Driver initialized successfully.")
                   title = sb.driver.title
           except Exception as e:
               logger.error(f"Error loading driver: {e}")
               return {"error": str(e)}, 500
   
           return {"title": title}
       except Exception as e:
           app.logger.error(f"Error loading driver: {e}")
           return {"error": str(e)}, 500
   
   if __name__ == "__main__":
       app.run()

4. Run Flask manually:

   • Start the Flask app with python3 app.py.
   • Send a POST request using curl or Postman to the /fetch endpoint.
   • The driver loads successfully, and the page title is returned.

5. Configure Flask with Gunicorn and Nginx:

   • Set up Gunicorn and Nginx to start the Flask app automatically (as per typical production setups).
   • Start the server and send a POST request to the /fetch endpoint.

6. Error Observed:

   • The request fails with the error:

     Error loading driver: Message: session not created: cannot connect to chrome at 127.0.0.1:9222
     from chrome not reachable
     

---

Observations:

• The driver works fine when the script is run manually or when Flask is started manually.
• The issue occurs only when Flask is served with Gunicorn and Nginx.

---

Environment Details:

• SeleniumBase Version: 4.33.12
• Flask Version: 3.1.0
• Gunicorn Version: 23.0.0
• Ubuntu Version: 22.04.4 LTS (Jammy Jellyfish)
• Nginx Version: 1.18.0
• Python Version: 3.10.12

---

What I Have Tried:

1. Verified that the environment variables are correctly set in the gunicorn.service file, including:
   • PATH
   • Chrome driver binary path
2. Added a virtual display (xvfb) for headless mode.
3. Used the uc=True parameter with SeleniumBase SB and Driver.
4. Increased the timeout duration.
5. Switched the Gunicorn worker class to gthread.
6. Verified that all required dependencies are installed.
7. Checked system resources (sufficient memory and CPU available).
8. Tested the code outside Gunicorn (both manually and with Flask), and it worked as expected.
9. I have used selenium for some websites which works as expected

---

Additional Context:

The issue seems related to Gunicorn's process management or Nginx's proxy setup interfering with the uc=True argument or the headless Chrome browser session. Further guidance on resolving this issue is appreciated.

---

[mdmintz]

Error loading driver: Message: session not created: cannot connect to chrome at 127.0.0.1:9222 from chrome not reachable

That usually means Chrome is not in the default location for the OS being used, so you must set it manually with binary_location. Eg: binary_location="/usr/bin/google-chrome". Figure out where Chrome is located, and then set it. Note that on Linux, you'll need to use either google-chrome-stable or google-chrome (but NOT chromium).

Other possible reasons may include port conflicts if multithreading, or missing Linux permissions.

Note that the point of xvfb is so that you don't have to use headless Chrome, which is detectable.

[bhautik-mangukiya]

is there any way to use the driver with multithreading on Linux?

[mdmintz]

UC Mode Video 2 covers multithreading: https://www.youtube.com/watch?v=2pTpBtaE7SQ

[bhautik-mangukiya]

Thanks.