Add SemanticDB support - cont. (#1508)

* Add SemanticDB support

* Semanticdb output works

* fixup to make work with scalaopt phase

* fixed tests

* added tests

* fix

* Add Scalac3 semanticdb options

* Fix scala3 failures

* fix formating

* fix scala3 support; add scala3 test

* clean up dead code

* bump, to retrigger PR checks

* bump scalameta; refactor version string

* more scala 3.1, refactor version string

* update hashes

* bump scalafmt

* Revert "bump, to retrigger PR checks"

This reverts commit 110442d.

* Revert "Revert "bump, to retrigger PR checks""

This reverts commit 39b10b2.

* Revert "bump scalafmt"

This reverts commit cb07f15.

* Revert "update hashes"

This reverts commit 993aa30.

* Revert "more scala 3.1, refactor version string"

This reverts commit 166c1f0.

* Revert "bump scalameta; refactor version string"

This reverts commit 8fcbbc9.

* Revert "Bump jmh version: 1.20 -> 1.36 (#1466)"

This reverts commit 1e4df56.

* Bump jmh version: 1.20 -> 1.36 (#1466)

Also bump versions of its dependencies:
* org.ow2.asm:asm: 6.1.1 -> 9.0
* net.sf.jopt-simple:jopt-simple: 4.6 -> 5.0.4

* add to predefined outputs

* revert predefined output

* include semanticdb in class jar

* fix merge

* fix tests

* lint

* move to phases

* fix bug

* force rebuild

* lint fix

* Semanticdb to use phase, and other improvements

* semanticdb: addressed pr review comments
- fixed semanticdb example
- added prefix '_' to semanticdb output path
- renamed certain vars to not prepend with "scala_"
- changed depsprovider var name to be plural instead of singular

---------

Co-authored-by: Ólafur Páll Geirsson <[email protected]>
Co-authored-by: Matt Peterson <[email protected]>
Co-authored-by: Aish <[email protected]>
Co-authored-by: Aish Fenton <[email protected]>
Co-authored-by: Dmitry Komanov <[email protected]>

Author: 6 people

README.md

@@ -283,3 +283,4 @@ Here's a (non-exhaustive) list of companies that use `rules_scala` in production
 * [Twitter](https://twitter.com/)
 * [VSCO](https://vsco.co)
 * [Wix](https://www.wix.com/)
+

docs/scala_toolchain.md

@@ -79,7 +79,8 @@ scala_register_toolchains()
           Allows to configure dependencies lists by configuring <code>DepInfo</code> provider targets. 
           Currently supported dep ids: <code>scala_compile_classpath</code>, 
           <code>scala_library_classpath</code>, <code>scala_macro_classpath</code>, <code>scala_xml</code>, 
-          <code>parser_combinators</code>.     
+          <code>parser_combinators</code>,     
+          <code>semanticdb</code>
         </p>
       </td>
     </tr>
@@ -143,6 +144,27 @@ scala_register_toolchains()
           List of target prefixes included for unused deps analysis. Exclude patetrns with '-'
         </p>
       </td>
+    </tr>
+        <tr>
+      <td><code>enable_semanticdb</code></td>
+      <td>
+        <p><code>Boolean; optional (default False)</code></p>
+        <p>
+          Enables semanticdb output. 
+        </p>
+      </td>
+    </tr>
+            <tr>
+      <td><code>semanticdb_bundle_in_jar</code></td>
+      <td>
+        <p><code>Boolean; optional (default False)</code></p>
+        <p>
+          When False, *.semanticdb files are added to the filesystem in a directory.
+        </p>
+        <p>
+          When True, *.semanticdb files will be bundled inside the jar file.
+        </p>
+      </td>
     </tr>
   </tbody>
 </table>

examples/semanticdb/BUILD

@@ -0,0 +1,28 @@
+load("@io_bazel_rules_scala//scala:scala_toolchain.bzl", "scala_toolchain")
+load("@io_bazel_rules_scala//scala:scala.bzl", "scala_binary", "scala_library")
+
+scala_toolchain(
+    name = "semanticdb_toolchain_impl",
+    enable_semanticdb = True,
+    semanticdb_bundle_in_jar = False,
+    visibility = ["//visibility:public"],
+)
+
+toolchain(
+    name = "semanticdb_toolchain",
+    toolchain = "semanticdb_toolchain_impl",
+    toolchain_type = "@io_bazel_rules_scala//scala:toolchain_type",
+    visibility = ["//visibility:public"],
+)
+
+scala_library(
+    name = "hello_lib",
+    srcs = ["Foo.scala"],
+)
+
+scala_binary(
+    name = "hello",
+    srcs = ["Main.scala"],
+    main_class = "main",
+    deps = [":hello_lib"],
+)

examples/semanticdb/Foo.scala

@@ -0,0 +1,3 @@
+class Foo(){
+  def sayHello: String = "Hello!!"
+}

examples/semanticdb/Main.scala

@@ -0,0 +1,7 @@
+
+object main{
+  def main(args: Array[String]): Unit = {
+    val foo = new Foo()
+    println(foo.sayHello)
+  }
+}

examples/semanticdb/README.md

@@ -0,0 +1,13 @@
+# scala_rules Semanticdb example
+
+This example demonstrates using an aspect to access the semanticdb info for one or more targets. In this example, an aspect is used to generate a json file that contains the semanticdb info that could be consumed by a consumer such as an IDE. 
+
+In this example, note that a scala_toolchain with enable_semanticdb=True is setup in the BUILD file.
+
+This command can be used to run the aspect (and not run the full build) 
+
+```
+bazel build //...  --aspects aspect.bzl%semanticdb_info_aspect --output_groups=json_output_file
+```
+
+The semanticdb_info.json file will be created for each target, and contains the semanticdb info for the target.

examples/semanticdb/WORKSPACE

@@ -0,0 +1,45 @@
+workspace(name = "specs2_junit_repositories")
+
+load("@bazel_tools//tools/build_defs/repo:http.bzl", "http_archive")
+
+skylib_version = "1.4.1"
+
+http_archive(
+    name = "bazel_skylib",
+    sha256 = "b8a1527901774180afc798aeb28c4634bdccf19c4d98e7bdd1ce79d1fe9aaad7",
+    type = "tar.gz",
+    urls = [
+        "https://mirror.bazel.build/github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib-{}.tar.gz".format(skylib_version, skylib_version),
+        "https://github.com/bazelbuild/bazel-skylib/releases/download/{}/bazel-skylib-{}.tar.gz".format(skylib_version, skylib_version),
+    ],
+)
+
+local_repository(
+    name = "io_bazel_rules_scala",
+    path = "../..",
+)
+
+load("@io_bazel_rules_scala//:scala_config.bzl", "scala_config")
+
+scala_config(scala_version = "2.13.6")
+
+load(
+    "@io_bazel_rules_scala//scala:scala.bzl",
+    "rules_scala_setup",
+    "rules_scala_toolchain_deps_repositories",
+)
+
+rules_scala_setup()
+
+rules_scala_toolchain_deps_repositories(fetch_sources = True)
+
+load("@rules_proto//proto:repositories.bzl", "rules_proto_dependencies", "rules_proto_toolchains")
+
+rules_proto_dependencies()
+
+rules_proto_toolchains()
+
+#Register and use the custom toolchain that has semanticdb enabled
+register_toolchains(
+    "//:semanticdb_toolchain",
+)

examples/semanticdb/aspect.bzl

@@ -0,0 +1,24 @@
+#This aspect is an example of exposing semanticdb information for each target into a json file.
+# An IDE could use a json file like this to consume the semanticdb data for each target.
+
+load("@io_bazel_rules_scala//scala:semanticdb_provider.bzl", "SemanticdbInfo")
+
+def semanticdb_info_aspect_impl(target, ctx):
+    if SemanticdbInfo in target:
+        output_struct = struct(
+            target_label = str(target.label),
+            semanticdb_target_root = target[SemanticdbInfo].target_root,
+            semanticdb_pluginjar = target[SemanticdbInfo].plugin_jar,
+        )
+
+        json_output_file = ctx.actions.declare_file("%s_semanticdb_info.json" % target.label.name)
+        ctx.actions.write(json_output_file, json.encode_indent(output_struct))
+
+        return [OutputGroupInfo(json_output_file = depset([json_output_file]))]
+    return []
+
+semanticdb_info_aspect = aspect(
+    implementation = semanticdb_info_aspect_impl,
+    attr_aspects = ["deps"],
+    toolchains = ["@io_bazel_rules_scala//scala:toolchain_type"],
+)

scala/BUILD

@@ -34,6 +34,8 @@ _PARSER_COMBINATORS_DEPS = ["@io_bazel_rules_scala_scala_parser_combinators"]
 
 _SCALA_XML_DEPS = ["@io_bazel_rules_scala_scala_xml"]
 
+_SEMANTICDB_DEPS = ["@org_scalameta_semanticdb_scalac"] if SCALA_MAJOR_VERSION.startswith("2") else []
+
 setup_scala_toolchain(
     name = "default_toolchain",
     scala_compile_classpath = _SCALA_COMPILE_CLASSPATH_DEPS,
@@ -108,3 +110,10 @@ declare_deps_provider(
     visibility = ["//visibility:public"],
     deps = _PARSER_COMBINATORS_DEPS,
 )
+
+declare_deps_provider(
+    name = "semanticdb_provider",
+    deps_id = "semanticdb",
+    visibility = ["//visibility:public"],
+    deps = _SEMANTICDB_DEPS,
+)

scala/private/macros/scala_repositories.bzl

@@ -128,6 +128,7 @@ ARTIFACT_IDS = [
     "io_bazel_rules_scala_scala_reflect",
     "io_bazel_rules_scala_scala_xml",
     "io_bazel_rules_scala_scala_parser_combinators",
+    "org_scalameta_semanticdb_scalac",
 ] if SCALA_MAJOR_VERSION.startswith("2") else [
     "io_bazel_rules_scala_scala_library",
     "io_bazel_rules_scala_scala_compiler",

scala/private/macros/setup_scala_toolchain.bzl

@@ -8,13 +8,16 @@ def setup_scala_toolchain(
         scala_macro_classpath,
         scala_xml_deps = None,
         parser_combinators_deps = None,
+        semanticdb_deps = None,
+        enable_semanticdb = False,
         visibility = ["//visibility:public"],
         **kwargs):
     scala_xml_provider = "%s_scala_xml_provider" % name
     parser_combinators_provider = "%s_parser_combinators_provider" % name
     scala_compile_classpath_provider = "%s_scala_compile_classpath_provider" % name
     scala_library_classpath_provider = "%s_scala_library_classpath_provider" % name
     scala_macro_classpath_provider = "%s_scala_macro_classpath_provider" % name
+    semanticdb_deps_provider = "%s_semanticdb_deps_provider" % name
 
     declare_deps_provider(
         name = scala_compile_classpath_provider,
@@ -57,15 +60,31 @@ def setup_scala_toolchain(
     else:
         parser_combinators_provider = "@io_bazel_rules_scala//scala:parser_combinators_provider"
 
+    dep_providers = [
+        scala_xml_provider,
+        parser_combinators_provider,
+        scala_compile_classpath_provider,
+        scala_library_classpath_provider,
+        scala_macro_classpath_provider,
+    ]
+
+    if enable_semanticdb == True:
+        if semanticdb_deps != None:
+            declare_deps_provider(
+                name = semanticdb_deps_provider,
+                deps_id = "semanticdb",
+                deps = [semanticdb_deps],
+                visibility = visibility,
+            )
+
+            dep_providers.append(semanticdb_deps_provider)
+        else:
+            dep_providers.append("@io_bazel_rules_scala//scala:semanticdb_provider")
+
     scala_toolchain(
         name = "%s_impl" % name,
-        dep_providers = [
-            scala_xml_provider,
-            parser_combinators_provider,
-            scala_compile_classpath_provider,
-            scala_library_classpath_provider,
-            scala_macro_classpath_provider,
-        ],
+        dep_providers = dep_providers,
+        enable_semanticdb = enable_semanticdb,
         visibility = visibility,
         **kwargs
     )

scala/private/phases/phase_compile.bzl

@@ -129,10 +129,18 @@ def _phase_compile(
     jars2labels = p.collect_jars.jars2labels.jars_to_labels
     deps_providers = p.collect_jars.deps_providers
     default_classpath = p.scalac_provider.default_classpath
+    plugins = ctx.attr.plugins
+    additional_outputs = []
+    scalacopts = p.scalacopts
+
+    if (hasattr(p, "semanticdb")):
+        scalacopts += p.semanticdb.scalacopts
+        plugins = plugins + p.semanticdb.plugin
+        additional_outputs += p.semanticdb.outputs
 
     out = _compile_or_empty(
         ctx,
-        p.scalacopts,
+        scalacopts,
         manifest,
         jars,
         srcjars,
@@ -144,6 +152,8 @@ def _phase_compile(
         deps_providers,
         default_classpath,
         unused_dependency_checker_ignored_targets,
+        plugins,
+        additional_outputs,
     )
 
     # TODO: simplify the return values and use provider
@@ -169,7 +179,9 @@ def _compile_or_empty(
         dependency_info,
         deps_providers,
         default_classpath,
-        unused_dependency_checker_ignored_targets):
+        unused_dependency_checker_ignored_targets,
+        plugins,
+        additional_outputs):
     # We assume that if a srcjar is present, it is not empty
     if len(ctx.files.srcs) + len(srcjars.to_list()) == 0:
         _build_nosrc_jar(ctx)
@@ -200,7 +212,7 @@ def _compile_or_empty(
             jars,
             all_srcjars,
             transitive_compile_jars,
-            ctx.attr.plugins,
+            plugins,
             ctx.attr.resource_strip_prefix,
             ctx.files.resources,
             ctx.files.resource_jars,
@@ -212,6 +224,7 @@ def _compile_or_empty(
             ctx.executable._scalac,
             dependency_info,
             unused_dependency_checker_ignored_targets,
+            additional_outputs,
         )
 
         # build ijar if needed

scala/private/phases/phase_semanticdb.bzl

@@ -0,0 +1,78 @@
+load(
+    "@io_bazel_rules_scala//scala/private/toolchain_deps:toolchain_deps.bzl",
+    "find_deps_info_on",
+)
+load("@io_bazel_rules_scala//scala:semanticdb_provider.bzl", "SemanticdbInfo")
+load("@io_bazel_rules_scala_config//:config.bzl", "SCALA_MAJOR_VERSION")
+load("@bazel_skylib//lib:paths.bzl", "paths")
+
+def phase_semanticdb(ctx, p):
+    #semanticdb_bundle_in_jar feature: enables bundling the semanticdb files within the output jar.
+
+    #Scala 2: Uses the semanticdb compiler plugin. Will output semanticdb files into the specified 'targetroot' which defaults to be under the '_scalac/classes' dir. When targetroot is under the _scalac/classes dir scalac bundles the *.semanticdb files into the jar.
+
+    #Scala3: Semanticdb is built into scalac. Currently, if semanticdb-target is used, the semanticdb files are written and not bundled, otherwise, the semanticdb files are not written as files and only available inside the jar.
+
+    toolchain = ctx.toolchains["@io_bazel_rules_scala//scala:toolchain_type"]
+    toolchain_type_label = "@io_bazel_rules_scala//scala:toolchain_type"
+
+    if toolchain.enable_semanticdb == True:
+        scalacopts = []
+        semanticdb_deps = []
+        output_files = []
+        plugin_jar_path = ""
+
+        target_output_path = paths.dirname(ctx.outputs.jar.path)
+
+        semanticdb_intpath = "_scalac/" + ctx.label.name + "/classes" if toolchain.semanticdb_bundle_in_jar == True else "_semanticdb/" + ctx.label.name
+
+        semanticdb_target_root = "%s/%s" % (target_output_path, semanticdb_intpath)
+
+        #declare all the semanticdb files
+        if (not toolchain.semanticdb_bundle_in_jar):
+            semanticdb_outpath = "META-INF/semanticdb"
+
+            for currSrc in ctx.files.srcs:
+                if currSrc.extension == "scala":
+                    outputfilename = "%s/%s/%s.semanticdb" % (semanticdb_intpath, semanticdb_outpath, currSrc.path)
+                    output_files.append(ctx.actions.declare_file(outputfilename))
+
+        if SCALA_MAJOR_VERSION.startswith("2"):
+            semanticdb_deps = find_deps_info_on(ctx, toolchain_type_label, "semanticdb").deps
+
+            if len(semanticdb_deps) == 0:
+                fail("semanticdb enabled, but semanticdb plugin jar not specified in scala_toolchain")
+            if len(semanticdb_deps) != 1:
+                fail("more than one semanticdb plugin jar was specified in scala_toolchain. Expect a single semanticdb plugin jar")
+
+            plugin_jar_path = semanticdb_deps[0][JavaInfo].java_outputs[0].class_jar.path
+
+            scalacopts += [
+                #note: Xplugin parameter handled in scalacworker,
+                "-Yrangepos",
+                "-P:semanticdb:failures:error",
+                "-P:semanticdb:targetroot:" + semanticdb_target_root,
+            ]
+        else:
+            #Note: In Scala3, semanticdb is built-in to compiler, so no need for plugin
+
+            scalacopts.append("-Ysemanticdb")
+
+            if toolchain.semanticdb_bundle_in_jar == False:
+                scalacopts.append("-semanticdb-target:" + semanticdb_target_root)
+
+        semanticdb_provider = SemanticdbInfo(
+            semanticdb_enabled = True,
+            target_root = None if toolchain.semanticdb_bundle_in_jar else semanticdb_target_root,
+            is_bundled_in_jar = toolchain.semanticdb_bundle_in_jar,
+            plugin_jar = plugin_jar_path,
+        )
+
+        return struct(
+            scalacopts = scalacopts,
+            plugin = semanticdb_deps,
+            outputs = output_files,
+            external_providers = {"SemanticdbInfo": semanticdb_provider},
+        )
+
+    return None