Expected exception: org.junit.runners.model.InitializationError

[lshikhva]

Hi,

We're run apache gump (https://svn.apache.org/repos/asf/gump/trunk/) to test a new JDK 8 early access releases from https://jdk8.java.net/download.html
Came across a new failure staring with b94:
populate-dist:
[copy] Copying 53 files to /home/dtftest/gump/results/junit/gump-test/doc
[copy] Copying 1 file to /home/dtftest/gump/results/junit/gump-test
[copy] Copying 1 file to /home/dtftest/gump/results/junit/gump-test
[copy] Copying 1 file to /home/dtftest/gump/results/junit/gump-test
[copy] Copying 1 file to /home/dtftest/gump/results/junit/gump-test

dist:
[java] JUnit version 4.12
[java] ....................................................................................................................................................................................................................................................................................................................................................I.I............................................E.....................................................................................................................................................................................................................................................................................................................................................................................................
[java] Time: 8.442
[java] There was 1 failure:
[java] 1) inaccessibleBaseClassIsCaughtAtValidation(org.junit.tests.validation.InaccessibleBaseClassTest)
[java] java.lang.AssertionError: Expected exception: org.junit.runners.model.InitializationError
[java] at org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:33)
[java] at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:274)
[java] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:76)
[java] at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:55)
[java] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:239)
[java] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:64)
[java] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:237)
[java] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:54)
[java] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:228)
[java] at org.junit.runners.ParentRunner.run(ParentRunner.java:312)
[java] at org.junit.runners.Suite.runChild(Suite.java:128)
[java] at org.junit.runners.Suite.runChild(Suite.java:27)
[java] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:239)
[java] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:64)
[java] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:237)
[java] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:54)
[java] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:228)
[java] at org.junit.runners.ParentRunner.run(ParentRunner.java:312)
[java] at org.junit.runners.Suite.runChild(Suite.java:128)
[java] at org.junit.runners.Suite.runChild(Suite.java:27)
[java] at org.junit.runners.ParentRunner$3.run(ParentRunner.java:239)
[java] at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:64)
[java] at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:237)
[java] at org.junit.runners.ParentRunner.access$000(ParentRunner.java:54)
[java] at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:228)
[java] at org.junit.runners.ParentRunner.run(ParentRunner.java:312)
[java] at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
[java] at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
[java] at org.junit.runner.JUnitCore.runMain(JUnitCore.java:77)
[java] at org.junit.runner.JUnitCore.main(JUnitCore.java:36)
[java]
[java] FAILURES!!!
[java] Tests run: 774, Failures: 1
[java]

BUILD FAILED
/home/dtftest/gump/results/junit/build.xml:211: The following error occurred while executing this line:
/home/dtftest/gump/results/junit/build.xml:181: Java returned: 1

Total time: 28 seconds

[dsaff]

Has there been an intentional change in JDK 8 that allows reflective calls like getDeclaredMethods to succeed on classes that would otherwise be inaccessible, i.e., package-private classes in another package?

[stefanbirkner]

The behaviour of java.lang.reflect.Method.getAnnotations() has changed. You can check https://github.com/stefanbirkner/oraclejdk8-method-getannotations-bug for the difference.

I created a bug report: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=9008626 (Will be available when it is accepted.)

[dsaff]

@stefanbirkner,

That appears to be a rather old bug. Is there a way to workaround it so that JUnit works both in its presence and in its absence? Thanks.

[stefanbirkner]

@dsaff I try to find a solution. Unfortunately I still don't have feedback from Oracle about the issue.

[Tibor17]

As I understood you found that java.lang.reflect.Method.getAnnotations() does not return empty array in Java 8 unlike in Java 5 in this test. BlockJUnit4ClassRunner#describeChild() is using such call.
I cannot say that Java 8 is buggy in this case. As I have read the Javadoc it really looks like these two have different purpose:
Sub.class.getMethod("a").getAnnotations() Sub.class.getMethod("a").getDeclaredAnnotations()
The call getDeclaredAnnotations says that ignores inheritance, which means that the opposite call getAnnotations should use the inheritance; otherwise two methods with the same behavior would not make sense. The Sun Microsystems was known by long term bug fixing. I guess this is the case.

Take a look into JDK 6 and you will see the implementation of Method -> AccessibleObject
public Annotation[] getAnnotations() { return getDeclaredAnnotations(); }
I am not sure if JDK 8 is aware of @inherited, but @test is not annotated by @inherited.
So I am not sure if such kind of inheritance mentioned in getDeclaredAnnotations is all about Joshua Bloch's @inherited.

Is there a fix for this, I'm unable to build junit from source, without skipping the test phase.

[stefanbirkner]

I did some more research on this issue. The difference between Java 7 and Java 8 is that JUnit with Java 7 uses the FrameworkMethod

public void org.junit.tests.validation.anotherpackage.Super.a()

whereas JUnit with Java 8 uses the FrameworkMethod

public void org.junit.tests.validation.anotherpackage.Sub.a()

[stefanbirkner]

It seems as if JDK 8 generates a bridge method Sub.a().

javap output for class Sub compiled with JDK 7:

Compiled from "Sub.java"
public class org.junit.tests.validation.anotherpackage.Sub extends org.junit.tests.validation.anotherpackage.Super {
  public org.junit.tests.validation.anotherpackage.Sub();
}

javap output for class Sub compiled with JDK 8:

Compiled from "Sub.java"
public class org.junit.tests.validation.anotherpackage.Sub extends org.junit.tests.validation.anotherpackage.Super {
  public org.junit.tests.validation.anotherpackage.Sub();
  public void a();
}

[stefanbirkner]

There are two solutions for solving this issue. Remove the validation

if (!Modifier.isPublic(getDeclaringClass().getModifiers())) {
  errors.add(new Exception("Class " + getDeclaringClass().getName() + " should be public"));
}

from FrameworkMethod or don't let TestClass provide bridge methods and synthetic methods. See stefanbirkner/junit@69de4c2

I recommend removing the validation, but I'm not sure whether this could cause problems with security managers. @junit-team/junit-committers Which solution are you preferring?

[kcooney]

For those missing context, the source file for Sub.java is here:

https://github.com/junit-team/junit/blob/master/src/test/java/org/junit/tests/validation/anotherpackage/Sub.java

It looks like in this case a bridge method might be created because the base class is not public, but the method is public, and the subclass is public.

I'm curious what will happen in JDK7 and JDK8 if we turn off the validation check, ignore the bridge method and try to call the test method via the Method object defined in Super on an instance of `Sub'