assertion failed: ClassBType.info not yet assigned when using -opt:l:inline and Spark 2.4.0 on Scala 2.12

[dieu]

Hello,

We are maintaining the open source project https://github.com/twitter/algebird, and try to upgrade algebird-spark module to new spark version 2.4.0 and scala version 2.12.7 (https://github.com/twitter/algebird/tree/apanasenko/spark_2.4.0). But we are hit a problem (works fine on 2.11.12):

https://travis-ci.org/twitter/algebird/jobs/451666066#L2031

java.lang.ArrayIndexOutOfBoundsException: 15859
	at scala.tools.asm.ClassReader.readUTF(ClassReader.java:2624)
	at scala.tools.asm.ClassReader.readUTF8(ClassReader.java:2596)
	at scala.tools.nsc.backend.jvm.opt.InlineInfoAttribute.nextUTF8$1(InlineInfoAttribute.scala:95)
	at scala.tools.nsc.backend.jvm.opt.InlineInfoAttribute.$anonfun$read$1(InlineInfoAttribute.scala:116)
	at scala.tools.nsc.backend.jvm.opt.InlineInfoAttribute.$anonfun$read$1$adapted(InlineInfoAttribute.scala:114)
	at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234)
	at scala.collection.immutable.Range.foreach(Range.scala:156)
	at scala.collection.TraversableLike.map(TraversableLike.scala:234)
	at scala.collection.TraversableLike.map$(TraversableLike.scala:227)
	at scala.collection.AbstractTraversable.map(Traversable.scala:104)
	at scala.tools.nsc.backend.jvm.opt.InlineInfoAttribute.read(InlineInfoAttribute.scala:114)
	at scala.tools.nsc.backend.jvm.opt.InlineInfoAttribute.read(InlineInfoAttribute.scala:33)
	at scala.tools.asm.ClassReader.readAttribute(ClassReader.java:2458)
	at scala.tools.asm.ClassReader.accept(ClassReader.java:610)
	at scala.tools.nsc.backend.jvm.opt.ByteCodeRepository.$anonfun$parseClass$1(ByteCodeRepository.scala:259)
	at scala.tools.nsc.backend.jvm.opt.ByteCodeRepository.parseClass(ByteCodeRepository.scala:250)
	at scala.tools.nsc.backend.jvm.opt.ByteCodeRepository.$anonfun$parsedClassNode$1(ByteCodeRepository.scala:65)
	at scala.collection.mutable.MapLike.getOrElseUpdate(MapLike.scala:206)
	at scala.collection.mutable.MapLike.getOrElseUpdate$(MapLike.scala:203)
	at scala.collection.mutable.AbstractMap.getOrElseUpdate(Map.scala:80)
	at scala.tools.nsc.backend.jvm.opt.ByteCodeRepository.parsedClassNode(ByteCodeRepository.scala:65)

and

java.lang.AssertionError: assertion failed: ClassBType.info not yet assigned: Lorg/apache/spark/rdd/RDD;
	at scala.tools.nsc.backend.jvm.BTypes$ClassBType.info(BTypes.scala:629)
	at scala.tools.nsc.backend.jvm.BTypes$ClassBType.isNestedClass(BTypes.scala:681)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$Collector.getClassIfNested(BackendUtils.scala:332)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$NestedClassesCollector.visitInternalName(BackendUtils.scala:657)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$NestedClassesCollector.visitDescriptor(BackendUtils.scala:686)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$NestedClassesCollector.$anonfun$visit$2(BackendUtils.scala:618)
	at scala.tools.nsc.backend.jvm.analysis.BackendUtils$NestedClassesCollector.$anonfun$visit$2$adapted(BackendUtils.scala:617)
	at scala.collection.Iterator.foreach(Iterator.scala:944)
	at scala.collection.Iterator.foreach$(Iterator.scala:944)
	at scala.collection.AbstractIterator.foreach(Iterator.scala:1432)
	at scala.collection.IterableLike.foreach(IterableLike.scala:71)

To reproduce results, you need to do:

$ git clone [email protected]:twitter/algebird.git
$ cd algebird/
$  git checkout apanasenko/spark_2.4.0
$ ./sbt -212 ";project algebird-spark; compile"

a problem reprdocible even if I delete all code in algebird-spark module and keep only one file with:

class AlgebirdRDD[T](val rdd: RDD[T])

[smarter]

Workaround: remove the inline related settings (-opt:l:inline and -opt-inline-from:com.twitter.algebird.**) from the scalacOptions of the algebird-spark project. (Unrelated but why do you compile with -Ydebug ? That's an internal compiler flag used for debugging, not something to turn on by default in your build)

[smarter]

Unrelated but why do you compile with -Ydebug ?

Oh I guess that's necessary to get stack traces out of backend failures, my bad (though I don't see why the backend hides its stack traces like this).

[dieu]

Workaround: remove the inline related settings (-opt:l:inline and -opt-inline-from:com.twitter.algebird.**) from the scalacOptions of the algebird-spark project. (Unrelated but why do you compile with -Ydebug ? That's an internal compiler flag used for debugging, not something to turn on by default in your build)

thanks a lot, that's help (note: -optimize also trigger this behavior).

Unrelated but why do you compile with -Ydebug ?

Oh I guess that's necessary to get stack traces out of backend failures, my bad (though I don't see why the backend hides its stack traces like this).

yep, only to see full stack trace and give more info for somebody who will dig inside.

[smarter]

/cc @lrytz since this seems to be an optimizer issue.

[lrytz]

There's something wrong with the ScalaInlineInfo attribute in the classfiles in http://repo1.maven.org/maven2/org/apache/spark/spark-core_2.12/2.4.0, this much I can tell already. Not sure why, I'll build it locally. Does the spark build do any post-processing to generated classfiles?

[Jasper-M]

It shades a few libraries.

[lrytz]

Yeah, so the problem is post-processing in the maven build, I guess it's the shade plugin.

I checked out https://github.com/apache/spark v2.4.0 and ran

• ./dev/change-scala-version.sh 2.12
• ./build/mvn -Pscala-2.12 -DskipTests package

Comparing compiled and packaged classfiles

• javap -v -cp core/target/scala-2.12/classes org.apache.spark.internal.Logging | tr -dC '[:print:]\t\n'
• javap -v -cp core/target/spark-core_2.12-2.4.0.jar org.apache.spark.internal.Logging | tr -dC '[:print:]\t\n'

shows that the constant pool entries are re-ordered.

The ScalaInlineInfo attribute has references to method names and descriptors in the constant pool, it stores some additional information for methods (e.g. if the method is annotated @inline). These references are not updated, as the shade plugin doesn't know the attribute.

Ideas...? cc @retronym

[lrytz]

I guess the best solution is to add support to the maven shade plugin (https://github.com/apache/maven-shade-plugin).

[lrytz]

Or maybe the shade plugin has hooks / extension points. Looking at that.

[Jasper-M]

If at all possible, perhaps you could enter all class names that are used as String constants (if they're not already there...) and refer to those constants in the Scala specific sections. I believe String constants are also transformed by the shade plugin. But that would probably be a pretty big overhaul, if it's even possible.

I guess the best solution is to add support to the maven shade plugin (https://github.com/apache/maven-shade-plugin).

Does the sbt assembly plugin know how to update the ScalaSignature and ScalaInlineInfo stuff?

[lrytz]

ScalaSignature doesn't have any references to the constant pool, so there's no problem. But using sbt-assembly with shading (which seems to use Jar Jar) likely has the same issue for ScalaInlineInfo.

[lrytz]

Maven shade uses asm:

https://github.com/apache/maven-shade-plugin/blob/master/src/main/java/org/apache/maven/plugins/shade/DefaultShader.java#L469

A comment in ASM predicted what's happening here

This may corrupt it if this value contains references to the constant pool

https://gitlab.ow2.org/asm/asm/blob/ASM_7_0/asm/src/main/java/org/objectweb/asm/ClassReader.java#L401-404

Passing Array(InlineInfoAttributePrototype) to accept would fix it. Not sure what's the best way fowrard. The shade plugin could accept a list of prototypes, and we could release a copy of our class with a prototype as a separate library.

We could of course encode full strings in ScalaInlineInfo instead of relying on the constant pool, but that would affect classfile size. Or we could encode the information in the ScalaSignature where the name strings also exist (though not name-mangled / flattened strings). But the goal of the exercise was to avoid having to run the unpickler.

[heuermh]

Hello @dieu @lrytz Have you submitted an issue to the Apache Spark JIRA? We're running into a similar issue (see reference link above).

[dieu]

Hello @dieu @lrytz Have you submitted an issue to the Apache Spark JIRA? We're running into a similar issue (see reference link above).

Nope, but disabling of -opt-inline works well for us.

[SethTisue]

@sadhen if you comment here, we can assign you the ticket (for credit/glory)

[da-liii]

@SethTisue Do we need to port the PR to 2.13.x?

[SethTisue]

It will be forward-merged to 2.13.x in time for 2.13.5. We merge everything forward every so often. (If your PR merges cleanly, you won't have to do anything. If the merge is tricky, you might asked to submit a forward port. In this case, the change is small enough that I don't expect you'll need to do anything further.)