Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[🐛 Bug]: Failed to get some website on Linux but success on Windows #15486

Closed
dongrixinyu opened this issue Mar 23, 2025 · 4 comments
Closed

[🐛 Bug]: Failed to get some website on Linux but success on Windows #15486

dongrixinyu opened this issue Mar 23, 2025 · 4 comments
Labels
I-defect Something is not working as intended J-logging Applied to issues where logging information would help troubleshoot

Comments

@dongrixinyu
Copy link

What happened?

I tried to use selenium to get some pages on Windows as below:

from selenium import webdriver

url = "https://vip.stock.finance.sina.com.cn/mkt/#stock_hs_up"
option = webdriver.ChromeOptions()
option.add_argument('headless')
driver = webdriver.Chrome(options=option)
driver.get(url)
width = driver.execute_script("return document.documentElement.scrollWidth")
height = driver.execute_script("return document.documentElement.scrollHeight")
driver.set_window_size(width, height)
driver.get_screenshot_as_file("webpage.png")

driver.get_screenshot_as_file(r"D:\webpage.png")
  • Chrome version: Google Chrome 134.0.6998.88
  • selenium version: 4.29.0

This snippet above achieves what I want.

but when I execute it on Linux as below, it failed.

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

url = "https://vip.stock.finance.sina.com.cn/mkt/#stock_hs_up"

option = webdriver.ChromeOptions()
option.add_argument('headless')

# here I need to set the correct binary path of chrome driver. It is a little bit different from code on Windows.
service = Service(executable_path='./chromedriver-linux64/chromedriver')

driver = webdriver.Chrome(options=option, service=service)
# driver = webdriver.Chrome()
driver.get(url)
width = driver.execute_script("return document.documentElement.scrollWidth")
height = driver.execute_script("return document.documentElement.scrollHeight")

driver.set_window_size(width, height)

driver.get_screenshot_as_file("webpage.png")
  • Chrome version: Google Chrome 134.0.6998.88
  • selenium version: 4.29.0
  • the network is OK

I failed to get the result.

Traceback (most recent call last):
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 1395, in getresponse
    response.begin()
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 286, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/socket.py", line 718, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cuichengyu/quantrading/test.py", line 17, in <module>
    driver.get(url)
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 454, in get
    self.execute(Command.GET, {"url": url})
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 427, in execute
    response = self.command_executor.execute(driver_command, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 404, in execute
    return self._request(command_info[0], url, body=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 428, in _request
    response = self._conn.request(method, url, body=body, headers=headers, timeout=self._client_config.timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/_request_methods.py", line 143, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=57885): Read timed out. (read timeout=120)

As you can see, it report timeout error, but when I request is by curl https://vip.stock.finance.sina.com.cn/mkt/#stock_hs_up, I get the result below immedietly, for sure the network is OK.

<script type="text/javascript" src="//n.sinaimg.cn/finance/stock/hq/src/addSima.js"></script>
<!--ima end-->
<script type="text/javascript" src="http://www.sinaimg.cn/unipro/pub/suda_m_v629.js"></script>
<script type="text/javascript">suds_init(1523,20.00,915,2);</script>
<script type="text/javascript" src="http://hq.sinajs.cn/?format=json&func=window.StandardBJTime=hq_json_sys_time;if%28typeof%28StandardBJTime_Callback%29==%27function%27%29StandardBJTime_Callback%28%29;&list=sys_time"></script>
<script type="text/javascript" src="http://finance.sina.com.cn/globaltimezone.js"></script>
<!--<script type="text/javascript" src="http://finance.sina.com.cn/basejs/MarketTS.js "></script>-->
<script type="text/javascript" src="//n.sinaimg.cn/finance/hqzxpclrr/MarketTS.js"></script>
<script type="text/javascript">
        (new MarketTS()).bind("stock",'nsq',function () { },function (__stringStatus,__allStatus)
        {
                document.getElementById("notice_us").innerHTML = __stringStatus;
                if(__allStatus[0] == 2)
                {
                        document.getElementById("notice_us").style.color = 'red';
                }
                else
                {
                        document.getElementById("notice_us").style.color = '#888';
                }
        });
</script>
<script charset="UTF-8" src="//n.sinaimg.cn/finance/hqcenter_right_ad/hqcenter_finapp_ad_0205.js"></script>
<!--<script charset="utf-8" src="http://n.sinaimg.cn/finance/xcapp/js/hqCenter_xcapp.js?ts=2.5"></script>-->
</html>

How can we reproduce the issue?

from selenium import webdriver
from selenium.webdriver.chrome.service import Service

url = "https://vip.stock.finance.sina.com.cn/mkt/#stock_hs_up"

option = webdriver.ChromeOptions()
option.add_argument('headless')

# here I need to set the correct binary path of chrome driver. It is a little bit different from code on Windows.
service = Service(executable_path='./chromedriver-linux64/chromedriver')

driver = webdriver.Chrome(options=option, service=service)
# driver = webdriver.Chrome()
driver.get(url)
width = driver.execute_script("return document.documentElement.scrollWidth")
height = driver.execute_script("return document.documentElement.scrollHeight")

driver.set_window_size(width, height)

driver.get_screenshot_as_file("webpage.png")


- OS: linux ubuntu
- Chrome version: Google Chrome 134.0.6998.88
- selenium version: 4.29.0
- the network is OK

Relevant log output

Traceback (most recent call last):
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 534, in _make_request
    response = conn.getresponse()
               ^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connection.py", line 516, in getresponse
    httplib_response = super().getresponse()
                       ^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 1395, in getresponse
    response.begin()
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 325, in begin
    version, status, reason = self._read_status()
                              ^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/http/client.py", line 286, in _read_status
    line = str(self.fp.readline(_MAXLINE + 1), "iso-8859-1")
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/socket.py", line 718, in readinto
    return self._sock.recv_into(b)
           ^^^^^^^^^^^^^^^^^^^^^^^
TimeoutError: timed out

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/cuichengyu/quantrading/test.py", line 17, in <module>
    driver.get(url)
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 454, in get
    self.execute(Command.GET, {"url": url})
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/webdriver.py", line 427, in execute
    response = self.command_executor.execute(driver_command, params)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 404, in execute
    return self._request(command_info[0], url, body=data)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/selenium/webdriver/remote/remote_connection.py", line 428, in _request
    response = self._conn.request(method, url, body=body, headers=headers, timeout=self._client_config.timeout)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/_request_methods.py", line 143, in request
    return self.request_encode_body(
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/_request_methods.py", line 278, in request_encode_body
    return self.urlopen(method, url, **extra_kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/poolmanager.py", line 443, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 841, in urlopen
    retries = retries.increment(
              ^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/util/retry.py", line 474, in increment
    raise reraise(type(error), error, _stacktrace)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/util/util.py", line 39, in reraise
    raise value
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 787, in urlopen
    response = self._make_request(
               ^^^^^^^^^^^^^^^^^^^
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 536, in _make_request
    self._raise_timeout(err=e, url=url, timeout_value=read_timeout)
  File "/home/cuichengyu/anaconda3/envs/quantrading/lib/python3.11/site-packages/urllib3/connectionpool.py", line 367, in _raise_timeout
    raise ReadTimeoutError(
urllib3.exceptions.ReadTimeoutError: HTTPConnectionPool(host='localhost', port=57885): Read timed out. (read timeout=120)

Operating System

Ubuntu

Selenium version

4.29.0

What are the browser(s) and version(s) where you see this issue?

Chrome 134.0.6998.88

What are the browser driver(s) and version(s) where you see this issue?

ChromeDriver 134.0.6998.88

Are you using Selenium Grid?

No response
@dongrixinyu dongrixinyu added I-defect Something is not working as intended A-needs-triaging A Selenium member will evaluate this soon! labels Mar 23, 2025
@github-actions GitHub Actions
Copy link

@dongrixinyu, thank you for creating this issue. We will troubleshoot it as soon as we can.

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

@titusfortner titusfortner added J-logging Applied to issues where logging information would help troubleshoot and removed A-needs-triaging A Selenium member will evaluate this soon! labels Mar 23, 2025
@github-actions GitHub Actions
Copy link

We need more information about this issue in order to troubleshoot.

Please turn on logging and re-run your code. Information on how to adjust logs for your language can be found in our
Troubleshooting documentation.

@github-actions GitHub Actions
Copy link

github-actions bot commented Apr 6, 2025

This issue was closed because we did not receive any additional information after 14 days.

@github-actions github-actions bot closed this as not planned Won't fix, can't repro, duplicate, stale Apr 6, 2025
@cgoldberg
Copy link
Contributor

@dongrixinyu if you want this issue investigated, please add debug logs and we will re-open it. The error you posted shows it is timing out communicating with chromedriver, not in reaching the remote URL.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
I-defect Something is not working as intended J-logging Applied to issues where logging information would help troubleshoot
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@titusfortner @cgoldberg @dongrixinyu