[🚀 Feature]: Expand CI Coverage

[titusfortner]

Feature and motivation

We should have better CI coverage before we do releases.

Update: Here's a spreadsheet with the current coverage
It's worth noting that not all Java tests run all the browsers, they have to be explicitly turned on per class.

@p0deje is adding Windows + Ruby support - #11295
And also working on improving caching so we can reduce our total execution times to support adding more coverage.
Selenium Manager should make this much easier now that we don't have to fuss with drivers in the CI setup.

Things we're missing include:

• No MacOS tests in any language
• No Edge + IE Mode tests in any language (low priority)
• No .NET tests at all (likely needs work from Bazel Build technical debt #10465)
• No Chrome Beta/Dev tests in any language

Specific Recommendations

• Remove firefox-dev from Java; from @whimboo: (Done in 8f8bf8e)

devedition comes basically before beta builds, but with starting around b3 they are basically equal in features except that the devedition has lesser restrictions; so you don't necessarily have to run dev edition

• Add Chrome Beta tests
• Java/.NET run more than one test in Windows/Mac
• Add more coverage in Java/Python so it isn't just Ruby testing everything

[github-actions]

@titusfortner, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then I-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[github-actions]

This issue is stale because it has been open 280 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[titusfortner]

We have .NET tests now
Looks like I added a duplicate of this specifically for Mac tests — #12397 (Ruby implementation #12556)

I think we need to have a discussion about what things which bindings should be testing. let's discuss next TLC meeting.

[titusfortner]

So, RBE is doing this:

  //dotnet/...  //java/... //javascript/atoms/... //javascript/webdriver/... //py/...

But it is all on Linux and I'm not sure which all browsers are included for the different bindings. I don't understand why Java is limiting things the way it does, and I'm not sure if it is still testing pre-release browsers (which I don't actually want it to).

RBE is great, but seems limiting still? Should we be doing a CI run with Windows & Mac tests?
Can we run beta/dev browser tests in a separate job, because a failing test from an unreleased browser version is good to be aware of, but shouldn't fail the suite.

At any rate, we still need to discuss what we want here.

[diemol]

It is more important to run tests on beta and dev browsers because we can catch things there, like the issue with Firefox 121.

I would put more effort into doing nightly releases and run the code examples we have in the docs with those nightly releases in Linux, macOS, and Windows.

[titusfortner]

I feel like failing beta is good for us to raise issues with Google and mozilla but shouldn't keep us from releasing. I guess as long as we can set guards separately for bugs...

Agreed we should get nightlies for everything, but coverage in docs examples is going to be pretty light for a while still.

My point with this is that we don't have an overall strategy, it's been a bunch of small decisions over time by many people as to what to do.
And RBE job is still kind of opaque to me as to what all it is actually testing, so I need to figure that out, anyway.

[p0deje]

I think as soon as Simon finishes upgrading to Bazel 7, we can try to build a matrix for each binding with:

• RBE for Linux builds with pinned browsers
• GHA for Windows/macOS (and pinned browsers too?)

Additionally, we can see we can hook tools like Dependabot/Renovate to automatically upgrade pinned browsers, including beta/dev versions.

[titusfortner]

Well, we do need integration tests for Selenium Manager, so it can't all be pinned browsers.
I would have known a month ago that .NET build wasn't working for Selenium Manager if the CI didn't require pinned browsers

[p0deje]

Well, we do need integration tests for Selenium Manager, so it can't all be pinned browsers.

I suppose we can have few of them that are explicitly checking integration in each bindings.

[p0deje]

We now have Firefox, Firefox beta, Chrome and Edge running for Ruby on RBE. I believe we can shrink the current CI and to only things that are not tested on RBE (Safari, Selenium-Manager related tests, etc)

[github-actions]

This issue is stale because it has been open 280 days with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 14 days with no activity.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. Please open a new issue for related bugs.