[🐛 Bug]: Selenium 3 new browser images #1444

cocorossello · 2021-11-03T15:14:53Z

What happened?

This is more a question than a issue. I see that images with chrome 95 are only being published with selenium 4.0.

I've migrated to selenium 4.0 in the client, however, we are still using selenium grid 3.0 in our selenium server. We run 80+ headless tests in parallel with chrome, it works with selenium grid 3 but it doesn't with 4.0, it's just too slow for some reason.

I would like to report on that issue, but I don't feel I can provide a lot of details, so I will stick to 3.0 until I see this issue fixed.

My question is if you will still provide selenium 3.0 images with new browsers, just to see what we can do.

Thanks.

Command used to start Selenium Grid with Docker

This is the docker compose file used for 4.0, but it's not relevant for the issue


version: "3"
services:
  selenium-hub4:
    image: selenium/hub:4.0.0-20211013
    container_name: selenium-hub4
    restart: unless-stopped
    ports:
      - "4442:4442"
      - "4443:4443"
      - "4444:4444"
    environment:
      - JAVA_OPTS=-Xmx10g  -XX:-ClassUnloading -XX:-ClassUnloadingWithConcurrentMark  -Dselenium.LOGGER.level=WARNING
      - SE_NODE_SESSION_TIMEOUT=200
  chrome-node4:
    image: selenium/node-chrome:4.0.0-20211013
    container_name: selenium-chrome-node4
    restart: unless-stopped
    shm_size: 16gb
    depends_on:
      - selenium-hub4
    environment:
      - SE_EVENT_BUS_HOST=selenium-hub4
      - SE_EVENT_BUS_PUBLISH_PORT=4442
      - SE_EVENT_BUS_SUBSCRIBE_PORT=4443
      - JAVA_OPTS=-Xmx10g  -XX:-ClassUnloading -XX:-ClassUnloadingWithConcurrentMark  -Dselenium.LOGGER.level=WARNING
      - SCREEN_WIDTH=1366
      - SCREEN_HEIGHT=768
      - SCREEN_DEPTH=24
      - SCREEN_DPI=74
      - START_XVFB=false
      - SE_NODE_MAX_SESSIONS=65
      - SE_NODE_OVERRIDE_MAX_SESSIONS=true
      - SE_SESSION_RETRY_INTERVAL=1
      - SE_NODE_SESSION_TIMEOUT=200
    ports:
      - "6900:5900"
    logging:
      options:
        max-size: 50m

This is the one used for selenium 3.0:

version: "3"
services:
  selenium-hub:
    image: selenium/hub:3.141.59-20210929
    container_name: selenium-hub
    restart: unless-stopped
    ports:
      - "5444:4444"
    privileged: true
    environment:
      GRID_BROWSER_TIMEOUT: 320
      GRID_TIMEOUT: 320
      NODE_BROWSER_TIMEOUT: 320
      NODE_TIMEOUT: 320
      JAVA_OPTS:  -Xmx10g  -XX:-ClassUnloading -XX:-ClassUnloadingWithConcurrentMark  -Dselenium.LOGGER.level=WARNING
      GRID_CLEAN_UP_CYCLE=60000
    logging:
      options:
        max-size: 50m
  chrome-node:
    image: selenium/node-chrome:3.141.59-20210929
    container_name: selenium-chrome-node
    restart: unless-stopped
    shm_size: 16gb
    tmpfs:
      - /tmp
    environment:
      REMOTE_HOST: http://chrome-node:5555
      HUB_HOST: selenium-hub
      NODE_MAX_INSTANCES: 10
      NODE_MAX_SESSION: 100
      GRID_BROWSER_TIMEOUT: 320
      GRID_TIMEOUT: 320
      NODE_BROWSER_TIMEOUT: 320
      NODE_TIMEOUT: 320
      START_XVFB: "false"
      DBUS_SESSION_BUS_ADDRESS: "/dev/null"
      JAVA_OPTS: -Xmx10g -XX:-ClassUnloading -XX:-ClassUnloadingWithConcurrentMark  -Dselenium.LOGGER.level=WARNING
    privileged: true
    extra_hosts:
      - "DEVELOPMENT:192.168.130.5"
      - "PREPRODUCTION:192.168.131.5"
      - "PRODUCTION:192.168.132.5"
      - "JENKINS:192.168.130.4"
    logging:
      options:
        max-size: 50m

Relevant log output

No relevant output

Operating System

Ubuntu 20.04 LTS

Docker Selenium version (tag)

3.141.59-20210929

diemol · 2021-11-03T15:25:38Z

We won't provide any more images for Grid 3, maintaing both sets of images was already challenging, also because Grid 4.0 is the stable version now.

I'll leave this issue open for a bit more in case more details are provided.

cocorossello · 2021-11-04T09:03:30Z

Ok, understood.

But threre is something weird with selenium 4 server.

I made this sample project:

https://github.com/cocorossello/selenium4-slow-driver-creation

It's just a simple example that opens a few drivers with selenium 3 server and selenium 4 server (both use selenium 4 as the client).

Selenium 4 is much slower, specially on subsequent invocations. This is probably not related to the docker images of course, but maybe selenium 4 activates some feature by default and this is what's making it much slower, I don't know. Do you have any cluse?


[INFO ] 2021-11-04 09:50:59.690 [Test worker] DriverCreationTest - Selenium 3 Opened in 622 ms
[INFO ] 2021-11-04 09:50:59.851 [Test worker] DriverCreationTest - Selenium 3 Opened in 97 ms
[INFO ] 2021-11-04 09:51:00.005 [Test worker] DriverCreationTest - Selenium 3 Opened in 94 ms

[INFO ] 2021-11-04 09:51:01.723 [Test worker] DriverCreationTest - Selenium 4 Opened in 1652 ms
[INFO ] 2021-11-04 09:51:06.715 [Test worker] DriverCreationTest - Selenium 4 Opened in 4918 ms
[INFO ] 2021-11-04 09:51:11.718 [Test worker] DriverCreationTest - Selenium 4 Opened in 4929 ms

stryczekk · 2021-11-10T11:27:56Z

I have similar observations.
I did some tests like @cocorossello and got similar times
v3 ~100ms
v4 ~5s
With such differences, v4 is currently unusable.

jllado · 2021-11-12T10:50:00Z

Same here 🙏

redxeagle · 2021-11-23T18:27:09Z

Same here. :(

andrewnicols · 2021-12-01T12:54:02Z

I believe we're seeing this too. Our first Selenium 4 job took one hour longer (four hours vs three) than under Selenium 3.141.59.

pawelusfr · 2021-12-01T16:02:21Z

My setup is 1 hub and 50 nodes (all of them running on separate EC2 instances) and I'm using Protractor with sharding set to 50 to run the tests.

What I have noticed is that when multiple concurrent session requests are hitting the hub it seems to be taking a long time to handle them. When I start the tests Protractor will send 50 session requests at almost the same time, but the nodes are just sitting there doing nothing for almost 2 minutes. Looking at the grid UI I saw the hub opening sessions and closing them and then opening new ones again.

Looking at hub logs I can see that e.g. for first 3 concurrent session requests distributor needs 5s to handle 1st, 7s to handle 2nd and 10s to handle 3rd request

This doesn't seem to be the case when there's only 1 session request (only 1 test running) and it starts right away.

One additional thing is that Selenium 4 hub uses double the CPU that Selenium 3 hub was using under the same load

ArminazK · 2022-01-03T09:58:32Z

Hi, I have been using selenium 4 images (just images, no GRID (no docker compose)), but faced really poor performance compared to selenium 3. Those are my major findings of selenium 4 docker images:

It needs way more resources: selenium 3 can handle 5 chrome instances at a time without any issues, but selenium 4 handles only 2 and executes scripts super slow
We do have 'multiremote' tests - not possible to execute those tests even on 2 browsers (on a same image), cause webdriver cannot operate with a browser which does not have focus (proved by connecting via vnc and bringing browser window from background)
Even running in parallel mode it takes minimum 2x time to execute same test on selenium 4 compared to selenium3

So we have now been using selenium 3 without any performance issues daily, but we would like to have support of the latest browsers...

Are you considering on still maintaining selenium3 images until once selenium 4 performance issues will be fixed?
Or there is a 'magic' setting to turn on for selenium 4?

Thanks a lot!

andrewnicols · 2022-01-05T13:28:15Z

Given there are a number of people experiencing some pretty serious issues with performance of Selenium 4, could I suggest that Selenium 3 images continue to be generated until these issues are investigated and resolved.

These issues do not seem to be limited to use of Grid, and seem to affect Standalone too. A 25-35% drop in performance is really quite serious and will have a massive impact on build times.

ejayaraman · 2022-01-06T11:17:30Z

We won't provide any more images for Grid 3, maintaing both sets of images was already challenging, also because Grid 4.0 is the stable version now.

I'll leave this issue open for a bit more in case more details are provided.

Hi @diemol. Apologies for tagging. I would like to get a confirmation if your above comments is still the case i.e is 3.141.59-20210929 the last docker image for Selenium 3? Knowing this will help us to prioritise our upgrade to Selenium 4. Thanks.

diemol · 2022-01-06T23:31:55Z

I appreciatte your downvotes in my comment above.

I do understand folks are just starting to try Grid 4 and finding things that could be improved. Before the holidays I found a couple of things that could be changed but there is literally a limilted amount of hours per day to work on this. So, maintaining this and also images for Grid 3 is way too much for me.

I will be checking this in the next 2 weeks. If you want to add more comments to this thread, please add concrete things that I can use to analyse ways to make more improvements (such as the initial comment from @cocorossello). Most of the other comments have zero actionable facts that I can analyse.

andrewnicols · 2022-01-11T03:41:42Z

Hi @diemol,

This issue is not limited to Hub. I've taken @cocorossello's gradle runner and run it against chrome standalone instead of hub, and these are my results:

Java version:

➜  ~ java --version
java 17.0.1 2021-10-19 LTS
Java(TM) SE Runtime Environment (build 17.0.1+12-LTS-39)
Java HotSpot(TM) 64-Bit Server VM (build 17.0.1+12-LTS-39, mixed mode, sharing)

Selenium versions:

3.141.59
4.1.1

Chrome

Versions:

Chrome 97.0.4692.71
ChromeDriver 97.0.4692.71 (adefa7837d02a07a604c1e6eff0b3a09422ab88d-refs/branch-heads/4692@{#1247})

Results

[INFO ] 2022-01-11 09:59:25.290 [Test worker] DriverCreationTest - Selenium 3 Opened in 1545 ms
[INFO ] 2022-01-11 09:59:26.307 [Test worker] DriverCreationTest - Selenium 3 Opened in 960 ms
[INFO ] 2022-01-11 09:59:26.959 [Test worker] DriverCreationTest - Selenium 3 Opened in 598 ms
[INFO ] 2022-01-11 09:59:29.133 [Test worker] DriverCreationTest - Selenium 4 Opened in 2117 ms
[INFO ] 2022-01-11 09:59:34.034 [Test worker] DriverCreationTest - Selenium 4 Opened in 4824 ms
[INFO ] 2022-01-11 09:59:39.008 [Test worker] DriverCreationTest - Selenium 4 Opened in 4862 ms

Firefox

Versions

➜  ~ geckodriver --version
geckodriver 0.30.0 (d372710b98a6 2021-09-16 10:29 +0300)
Firefox version: 95.0.2 (64-bit)

Results

[INFO ] 2022-01-11 10:01:32.462 [Test worker] DriverCreationTest - Selenium 3 Opened in 1663 ms
[INFO ] 2022-01-11 10:01:33.946 [Test worker] DriverCreationTest - Selenium 3 Opened in 1138 ms
[INFO ] 2022-01-11 10:01:35.313 [Test worker] DriverCreationTest - Selenium 3 Opened in 1127 ms
[INFO ] 2022-01-11 10:01:39.320 [Test worker] DriverCreationTest - Selenium 4 Opened in 3666 ms
[INFO ] 2022-01-11 10:01:44.630 [Test worker] DriverCreationTest - Selenium 4 Opened in 5016 ms
[INFO ] 2022-01-11 10:01:49.539 [Test worker] DriverCreationTest - Selenium 4 Opened in 4617 ms

Summary

I've run these tests quite a few times now, and the results are largely consistent. Selenium 4 is always approximately 4 times slower than Selenium 3.

In my case I've removed Docker from the equation (because the Selenium images do not work reliably on ARM) so this issue really should be in an issue on the main seleniumhq/selenium project instead.

I've also created a NodeJS performance tester to test the time to init and provide averages of each browser and selenium version. See https://github.com/andrewnicols/selenium-1444 for information.

I have raised SeleniumHQ/selenium#10242 to look at this issue as it is not related to do the docker images.

andrewnicols · 2022-01-11T07:12:54Z

@diemol , can I ask what the time cost is to continue maintaining the Selenium 3 images? Where does the additional cost come from? I thought that their generation was largely automated now?

diemol · 2022-01-17T15:39:50Z

Grid 3 images are not automated because they were being released from Travis, and when Travis limited our execution minutes, we were using Travis for Grid 3 and GitHub Actions for Grid 4.

As development was advancing, we did not feel the need to update the pipeline for Grid 3 since it was going away. That is why a Grid 3 release would be manual now.

andrewnicols · 2022-01-18T01:07:22Z

The upstream performance problem with starting a new session has now been resolved. See SeleniumHQ/selenium#10242 and SeleniumHQ/selenium@7848221 for more information.

I suspect that this will be available in Selenium 4.1.1. As a workaround, the --session-retry-interval 0 argument can be set

cocorossello · 2022-01-31T08:28:17Z

Thank you!

I can confirm that it works. Now times are something like:

[INFO ] 2022-01-31 09:26:16.982 [Test worker] DriverCreationTest - Selenium 3 Opened in 613 ms
ene 31, 2022 9:26:17 A. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2022-01-31 09:26:17.142 [Test worker] DriverCreationTest - Selenium 3 Opened in 96 ms
ene 31, 2022 9:26:17 A. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2022-01-31 09:26:17.296 [Test worker] DriverCreationTest - Selenium 3 Opened in 94 ms
ene 31, 2022 9:26:17 A. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2022-01-31 09:26:17.502 [Test worker] DriverCreationTest - Selenium 4 Opened in 143 ms
ene 31, 2022 9:26:17 A. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2022-01-31 09:26:17.714 [Test worker] DriverCreationTest - Selenium 4 Opened in 116 ms
ene 31, 2022 9:26:17 A. M. org.openqa.selenium.remote.ProtocolHandshake createSession
INFO: Detected dialect: W3C
[INFO ] 2022-01-31 09:26:17.945 [Test worker] DriverCreationTest - Selenium 4 Opened in 123 ms

(You still have to set - SE_SESSION_RETRY_INTERVAL=0 in the environment for this to work)

diemol · 2022-01-31T09:10:24Z

Thanks for the feedback @cocorossello!

I am pushing today a new release with SE_SESSION_RETRY_INTERVAL=0, so you do not need to set it up.

andrewnicols · 2022-01-31T14:25:01Z

Hi @diemol,

Is it worth holding back another 24-48 hours before releasing a new set of images? A new version of chrome will be released tomorrow (1 Feb).

In our case we need that version because it fixed a bug in chromedriver with scrolling elements into view before clicking on them.

diemol · 2022-01-31T15:30:35Z

No worries, I can do a new release tomorrow.
It was better to do separate releases to identify better possible issues and get more detailed feedback.

github-actions · 2022-03-03T00:16:01Z

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

cocorossello added the needs-triaging label Nov 3, 2021

diemol added R-awaiting-answer and removed needs-triaging labels Nov 3, 2021

This comment has been minimized.

Sign in to view

andrewnicols mentioned this issue Dec 1, 2021

[runner] Update to Chromedriver 96.0 custom moodlehq/moodle-ci-runner#66

Merged

diemol added S-needs-investigation and removed R-awaiting-answer labels Dec 15, 2021

andrewnicols mentioned this issue Jan 11, 2022

[🐛 Bug]: Performance of Selenium 4 is significantly slower than 3.141.59 SeleniumHQ/selenium#10242

Closed

diemol added S-done and removed S-needs-investigation labels Jan 30, 2022

diemol mentioned this issue Jan 30, 2022

Retry interval to 0 #1489

Merged

8 tasks

diemol closed this as completed in #1489 Jan 31, 2022

github-actions bot locked and limited conversation to collaborators Mar 3, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[🐛 Bug]: Selenium 3 new browser images #1444

[🐛 Bug]: Selenium 3 new browser images #1444

cocorossello commented Nov 3, 2021

diemol commented Nov 3, 2021

cocorossello commented Nov 4, 2021 •

edited

Loading

This comment has been minimized.

stryczekk commented Nov 10, 2021

jllado commented Nov 12, 2021

redxeagle commented Nov 23, 2021

andrewnicols commented Dec 1, 2021

pawelusfr commented Dec 1, 2021

ArminazK commented Jan 3, 2022 •

edited

Loading

andrewnicols commented Jan 5, 2022

ejayaraman commented Jan 6, 2022

diemol commented Jan 6, 2022

andrewnicols commented Jan 11, 2022

andrewnicols commented Jan 11, 2022

diemol commented Jan 17, 2022

andrewnicols commented Jan 18, 2022

cocorossello commented Jan 31, 2022

diemol commented Jan 31, 2022

andrewnicols commented Jan 31, 2022

diemol commented Jan 31, 2022

github-actions bot commented Mar 3, 2022

[🐛 Bug]: Selenium 3 new browser images #1444

[🐛 Bug]: Selenium 3 new browser images #1444

Comments

cocorossello commented Nov 3, 2021

What happened?

Command used to start Selenium Grid with Docker

Relevant log output

Operating System

Docker Selenium version (tag)

diemol commented Nov 3, 2021

cocorossello commented Nov 4, 2021 • edited Loading

This comment has been minimized.

stryczekk commented Nov 10, 2021

jllado commented Nov 12, 2021

redxeagle commented Nov 23, 2021

andrewnicols commented Dec 1, 2021

pawelusfr commented Dec 1, 2021

ArminazK commented Jan 3, 2022 • edited Loading

andrewnicols commented Jan 5, 2022

ejayaraman commented Jan 6, 2022

diemol commented Jan 6, 2022

andrewnicols commented Jan 11, 2022

Java version:

Selenium versions:

Chrome

Versions:

Results

Firefox

Versions

Results

Summary

andrewnicols commented Jan 11, 2022

diemol commented Jan 17, 2022

andrewnicols commented Jan 18, 2022

cocorossello commented Jan 31, 2022

diemol commented Jan 31, 2022

andrewnicols commented Jan 31, 2022

diemol commented Jan 31, 2022

github-actions bot commented Mar 3, 2022

cocorossello commented Nov 4, 2021 •

edited

Loading

ArminazK commented Jan 3, 2022 •

edited

Loading