Output .tasty files in Pickler phase

[nicolasstucki]

This is an upstream of some changes in #4767 with some additional optimizations.

[smarter]

I wonder if emitting tasty earlier may mess up zinc's logic for deleting/restoring output files.

@sjrd In Scala.js, is it possible for a .sjsir file to be emitted without a corresponding .class file (because compilation failed before .class file generation) ? If so then I guess we're good

[sjrd]

In Scala.js, is it possible for a .sjsir file to be emitted without a corresponding .class file (because compilation failed before .class file generation) ? If so then I guess we're good

In theory, it can happen, if one of the very last phases of scalac reports an error (or crashes). However, AFAIK there is only one compile error that can happen, and it is "JVM bytecode limit exceeded", which are very rare. I have never checked what happened to incremental compilation in that scenario. It may very well be broken. I don't care much for Scala.js because it's rare. If you emit as early as pickler, you should definitely investigate.

[allanrenucci]

This will probably mess up incremental compilation. The TastyFileManager relies on the generated class files to manage the generated TASTY files

[nicolasstucki]

@allanrenucci the CI is not kicking in. Is there something wrong with it or is it just this PR?

[nicolasstucki]

@allanrenucci, never mind. I had to manually approve in the CI as I modified .drone.yml.

[smarter]

The same way that we signal to sbt that we're emitting a .class file using a callback (https://github.com/lampepfl/dotty/blob/master/compiler/src/dotty/tools/backend/jvm/GenBCode.scala#L274-L280) we might need to signal that we're emitting a .tasty file. This might require adding a new callback to sbt.

[allanrenucci]

We might not have to update the plugin or sbt after all. If we produce some TASTY files and the compilation fails after pickler, we probably just need to delete the files produced. Zinc should be able to restore the overwritten files. Files that may be overwritten are deleted by Zinc before compilation and restored in case of failure.

Out of curiosity, why do we want to output TASTY files in Pickler phase?

[smarter]

Out of curiosity, why do we want to output TASTY files in Pickler phase?

It's useful for parallel compilation in a project (#4767) and could also be used by the build tool to parallelize across projects (start compilation of dependent projects after tasty has been output instead of waiting for the project to finish compiling). Another usecase is quicker feedback when working on code: instead of doing sbt ~compile you could have sbt ~check that only runs until tasty is output (we can't stop earlier for separate compilation to work).

[allanrenucci]

Closing with label revisit. We need to investigate how this interacts with incremental compilation