[Core] [JUnit5] Eclipse JUnit runner: Unable to start Cucumber

[nicerloop]

Describe the bug
The Eclipse JUnit runner is unable to start Cucumber

java.lang.IllegalArgumentException: Unable to load org.eclipse.jdt.internal.junit.runner.junit3.JUnit3Listener
at io.cucumber.core.resource.ClasspathScanner.lambda$null$5(ClasspathScanner.java:98)

To Reproduce
Steps to reproduce the behavior:

1. Create a simple maven project with java 1.8, io.cucumber:cucumber-junit-platform-engine:5.0.0-RC2 and io.cucumber:cucumber-java:5.0.0-RC2
2. Create a simple RunCucumberTest class and a simple feature with a scenario
3. Select project and Run As > JUnit Test
4. See error in failure trace and console

Expected behavior
The scenario is correctly run

Context & Motivation
I want to use cucumber 5 and junit 5 with maven and eclipse

Screenshots
no screenshot procided

Your Environment

• Versions used: cucumber 5.0.0-RC2
• Operating System and version: windows 10
• Build tool: eclipse 4.13 2019-09

Additional context
Why do core/src/main/java/io/cucumber/core/resource/ClasspathScanner.java throw an exception on line 98 when encountering unloadable class? In fact, just ignoring the ClassNotFoundException or NoClassDefFoundError should make everyone happy?

[mpkorstanje]

java.lang.IllegalArgumentException: Unable to load org.eclipse.jdt.internal.junit.runner.junit3.JUnit3Listener

JUnit 3? Are you sure this is not an eclipse problem?

I want to use cucumber 5 and junit 5 with maven and eclipse

I ran into a few issues with JUnit 5 in RC2. You may want to wait until RC3.

[nicerloop]

It seems eclipse throws a bunch of jars from its runners in the classpath, including a JUnit 3 listener, which has a dependency on JUnit3 classes, which are missing, and cause the exception.

My current interrest is in having cucumber and pitest work together through junit5-platform - that is done with corrections on the pitest side; now working on having all that inside eclipse.

I agree eclipse might be a bit cleaner with its classpath building - but junit5 allows running junit4 which allows running junit3.

But in any case, instantiating random classes present in the classpath faces the risk of missing dependent classes, and ClassNotFoundException or NoClassDefFoundError should not stop the classpath scan. Why throw an exception and not just ignore the unloadable class?

[mpkorstanje]

My current interrest is in having cucumber and pitest work together through junit5-platform - that is done with corrections on the pitest side; now working on having all that inside eclipse.

Ah cool! In that case you can definitely use RC2.

I agree eclipse might be a bit cleaner with its classpath building - but junit5 allows running junit4 which allows running junit3.

Fair enough. I'll see what I can do there.

As a work around can you add junit-platform.properties with the package of your glue in it?

cucumber.glue=com.example.app

[mpkorstanje]

Why throw an exception and not just ignore the unloadable class?

This was a trade off I had to think about. There are a few different opposing requirements and assumptions:

1. Cucumber should work out of the box with minimal configuration.
2. Cucumber should not quietly ignore problems
3. The classpath is safe by default

Unfortunately #3 only applies to a limited set of projects and #1 causes us to scan the root package for step definitions by default . So we can either throw an error, ignore the class loading error or require that a default is configured.

Now my assumption is that in most use cases the step definitions compile and have their dependencies available. So ignoring the class load error should be acceptable - it probably wasn't a step definition. However should there be class loading errors in the step definitions it will be much harder to debug these because Cucumber will quietly ignore them. So it's really good when it works and absolutely terrible when it fails. Not the most ideal failure mode.

Depending on feed back I may reconsider this in the future.

[nicerloop]

As a work around can you add junit-platform.properties with the package of your glue in it?

cucumber.glue=com.example.app

For maven surefire as well as for eclipse junit runner, I had to add a simple RunCucumberTest annotated class. This could be used to define the root package for the glue as well as for the feature. In that case, no further configuration would be needed, in the same way it is done with junit4.

So we can either throw an error, ignore the class loading error or require that a default is configured.

This would be a sane safe default for class path scanning, and would not need to abandon your seemingly opposing requirements: this part of the class path should be safe to scan, and can be identified with no additional configuration besides the marker class; class loading errors can still be loudly notified with an exception.

[mpkorstanje]

That's not a bad intuition. But it will not work.

Cucumber is build with the assumption that features are in the direct siblings of the cucumber engine in the test hierarchy. This means when a the same feature file is discovered by different selectors it will create the same test tree regardless of the selector that is used.

By either making the runner class or the default configuration it provides part of the test hierarchy this invariant no longer holds. You can find all sorts examples of ill behavior this will cause by taking a class selector and combining it with a file or uri selector.

[nicerloop]

Leaving the feature discovery aside, could the marker annotated class be used to define the root package for glue? Would that create other problems I don't see?

[mpkorstanje]

Those were the problems I was talking about. In JUnit4 discovery and configuration of tests were combined. Basically the ParentRunner class did all the work. With JUnit 5 the JUnit Platform separated discovery and configuration. So if we were to use marker classes we'd have to add their information to the discovered tests.