Add toolchain options API to WORKSPACE and Bzlmod (#1730)

* Add toolchain options API to WORKSPACE and Bzlmod

Updates `scala_toolchains()` to accept either boolean or dict arguments
for specific toolchains, and updates `//scala/extensions:deps.bzl` to
generate them from tag classes. Part of #1482.

Notable qualities:

- Adds toolchain options support to the `scala_toolchains()` parameters
  `scalafmt`, `scala_proto`, and `twitter_scrooge`, and to the
  `scalafmt` tag class.

- Eliminates the `scalafmt_default_config`, `scala_proto_options`, and
  `twitter_scrooge_deps` option parameters from `scala_toolchains()`.

- Provides uniform, strict evaluation and validation of toolchain
  options passed to `scala_toolchains()`.

- Configures enabled toolchains using root module settings or the
  default toolchain settings only.

- Introduces the shared `TOOLCHAIN_DEFAULTS` dict in
  `//scala/private:toolchains_defaults.bzl` to aggregate individual
  `TOOLCHAIN_DEFAULTS` macro parameter dicts.

This change also:

- Replaces the non-dev dependency `scala_deps.scala()` tag instantiation
  in `MODULE.bazel` with `dev_deps.scala()`.

- Renames the `options` parameter of the `scala_deps.scala_proto` tag
  class to `default_gen_opts` to match `setup_scala_proto_toolchain()`.

- Introduces `_stringify_args()` to easily pass all toolchain macro args
  compiled from `scala_toolchains_repo()` attributes through to the
  generated `BUILD` file.

- Extracts `_DEFAULT_TOOLCHAINS_REPO_NAME` and removes the
  `scala_toolchains_repo()` macro.

- Includes docstrings for the new private implementation functions, and
  updates all other docstrings, `README.md`, and other relevant
  documentation accordingly.

---

Inspired by @simuons's suggestion to replace toolchain macros with a
module extension in:

- #1722 (comment)

Though a full replacement is a ways off, this is a step in that
direction that surfaced several immediate improvements.

First, extensibility and maintainability:

- The new implementation enables adding options support for other
  toolchains in the future while maintaining backward compatibility, for
  both the `WORKSPACE` and Bzlmod APIs. Adding this support will only
  require a minor release, not a major one.

- The `scala_deps` module extension implementation is easier to read,
  since all toolchains now share the `_toolchain_settings` mechanism.

Next, improved consistency of the API and implementation:

- Toolchain options parameters should present all the same parameters as
  the macros to which they correspond, ensured by the
  `TOOLCHAIN_DEFAULTS` mechanism. This is to make it easier for users
  and maintainers to see the direct relationship between these separate
  sets of parameters. (They can also define additional parameters to
  those required by the macro, like `default_config` from the `scalafmt`
  options.)

  This principle drove the renaming of the `scala_deps.scala_proto` tag
  class parameter from `options` to `default_gen_opts`. It also inspired
  updating `scala_toolchains_repo()` to pass toolchain options through
  `_stringify_args()` to generate `BUILD` macro arguments.

- The consolidated `TOOLCHAIN_DEFAULTS` dict reduces duplication between
  the `scala/extensions/deps.bzl` and `scala/toolchains.bzl` files. It
  ensures consistency between tag class `attr` default values for Bzlmod
  users and the `scala_toolchains()` default parameter values for
  `WORKSPACE` users.

  The `TOOLCHAINS_DEFAULTS` dicts corresponding to each toolchain macro
  do duplicate the information in the macro argument lists. However, the
  duplicated values in this case are physically adjacent to one another,
  minimizing the risk of drift.

- Extracting `_DEFAULT_TOOLCHAINS_REPO_NAME` is a step towards enabling
  customized repositories based on the builtin toolchains, while
  specifying different options. This extraction revealed the fact that
  the `scala_toolchains_repo()` macro was no longer necessary, since
  only `scala_toolchains()` ever called it.

Finally, fixes for the following design bugs:

- Previously, `scala_deps.scala_proto(options = [...])` compiled the
  list of `default_gen_opts` from all tag instances in the module graph.
  This might've been convenient, but didn't generalize to other options
  for other toolchains. In particular, it differed from the previous
  `toolchains`, `scalafmt`, and `twitter_scrooge` tag class behavior.

  The new semantics are unambiguous, consistent, and apply to all
  toolchains equally; they do not show a preference for any one
  toolchain over the others. They also maintain the existing `scalafmt`
  and `twitter_scrooge` tag class semantics, but now using a generic
  mechanism, simplifying the implementation.

- Instantating `scala_deps.scala()` was a bug left over from the
  decision in #1722 _not_ to enable the builtin Scala toolchain by
  default under Bzlmod.

  The previous `scala_deps.toolchains()` tag class had a default `scala
  = True` parameter. The user could set `scala = False` to disable the
  builtin Scala toolchain. After replacing `toolchains()` with
  individual tag classes, the documented behavior was that the user must
  enable the builtin Scala toolchain by instantiating
  `scala_deps.scala()`.

  By instantiating `scala_deps.scala()` in our own `MODULE.bazel` file,
  we ensured that `rules_scala` would always instantiate the builtin Scala
  toolchain. While relatively harmless, it violated the intention of
  allowing the user to avoid instantiating the toolchain altogether.

* Update documentation for toolchain option dicts

Touched up documentation for the `scalafmt` and `scala_proto`
toolchains.

* Replace Scalafmt default_config alias with symlink

This avoids the need for the user to use `exports_files` so
`@rules_scala_toolchains//scalafmt:config` can access the config file.
Essentially restores the API from before #1725, but still fixes the same
bug as #1725.

* Update phase_scalafmt.md, check scalafmt conf path

Updates the Scalafmt documentation to reflect the current API. Adds a
check to `scala_toolchains_repo` to `fail` if the Scalafmt
`default_config` file doesn't exist.

The previous commit doesn't actually restore the exact pre-#1725 API.
It eliminates the `exports_files` requirement, but still requires a
`Label` or a relative path string, not an optional `.scalafmt.conf` in
the root directory.

After experimenting a bit and thinking this through, dropping support
for an optional `.scalafmt.conf` provides the most robust and reliable
interface. Specifically, supporting it requires detecting whether it
actually exists before falling back to the default. Having users
explicitly specify their own config seems a small burden to impose for
a more straightforward and correct implementation.

At the same time, I saw the opportunity to provide the user with
explicit feedback if the specified config file doesn't exist. Hence the
new check and `fail()` message.

Also renamed the generated `.scalafmt.conf` file in
`@rules_scala_toolchains//scalafmt` to `scalafmt.conf`. No need for it
to be a hidden file in that context.

* Update the `scala_deps` module extension docstring

The `scala_deps` docstring now more accurately describes the behavior
implemented by `_toolchain_settings()`.

* Fix `scala_proto` param in test_version/WORKSPACE

Caught this after doing an `--enable_workspace --noenable_bzlmod` build.

Author: mbland

BUILD

@@ -1 +0,0 @@
-exports_files([".scalafmt.conf"])

MODULE.bazel

@@ -95,7 +95,6 @@ use_repo(
     "rules_scala_toolchains",
     "scala_compiler_sources",
 )
-scala_deps.scala()
 
 # Register some of our testing toolchains first when building our repo.
 register_toolchains(
@@ -119,6 +118,7 @@ dev_deps = use_extension(
     "scala_deps",
     dev_dependency = True,
 )
+dev_deps.scala()
 dev_deps.jmh()
 dev_deps.junit()
 dev_deps.scala_proto()

README.md

@@ -930,19 +930,36 @@ parameter list, which is almost in complete correspondence with parameters from
 the previous macros. The `WORKSPACE` files in this repository also provide many
 examples.
 
-### Replacing toolchain registration macros in `WORKSPACE`
+### Replacing toolchain registration macros
 
 Almost all `rules_scala` toolchains configured using `scala_toolchains()` are
-automatically registered by `scala_register_toolchains()`. There are two
-toolchain macro replacements that require special handling.
+automatically registered by `scala_register_toolchains()`. The same is true for
+toolchains configured using the `scala_deps` module extension under Bzlmod.
+There are two toolchain macro replacements that require special handling.
 
 The first is replacing `scala_proto_register_enable_all_options_toolchain()`
-with the following `scala_toolchains()` parameters:
+with the following:
 
 ```py
+# MODULE.bazel
+
+scala_deps.scala_proto(
+    "default_gen_opts" = [
+        "flat_package",
+        "grpc",
+        "single_line_to_proto_string",
+    ],
+)
+
+# WORKSPACE
 scala_toolchains(
-    scala_proto = True,
-    scala_proto_options = [],
+    scala_proto = {
+        "default_gen_opts": [
+            "flat_package",
+            "grpc",
+            "single_line_to_proto_string",
+        ],
+    },
 )
 ```
 

docs/phase_scalafmt.md

@@ -71,25 +71,21 @@ bazel run <TARGET>.format-test
 
 to check the format (without modifying source code).
 
-The extension provides a default configuration, but there are 2 ways to use
-a custom configuration:
-
-- Put `.scalafmt.conf` at the root of your workspace
-- Pass `.scalafmt.conf` in via `scala_toolchains`:
-
-    ```py
-    # MODULE.bazel
-    scala_deps.scalafmt(
-        default_config = "//path/to/my/custom:scalafmt.conf",
-    )
-
-    # WORKSPACE
-    scala_toolchains(
-        # Other toolchains settings...
-        scalafmt = True,
-        scalafmt_default_config = "//path/to/my/custom:scalafmt.conf",
-    )
-    ```
+The extension provides a default configuration. To use a custom configuration,
+pass its path or target Label to the toolchain configuration:
+
+```py
+# MODULE.bazel
+scala_deps.scalafmt(
+    default_config = "path/to/my/custom/scalafmt.conf",
+)
+
+# WORKSPACE
+scala_toolchains(
+    # Other toolchains settings...
+    scalafmt = {"default_config": "path/to/my/custom/scalafmt.conf"},
+)
+```
 
 When using Scala 3, you must append `runner.dialect = scala3` to
 `.scalafmt.conf`.

docs/scala_proto_library.md

@@ -1,22 +1,40 @@
 # scala_proto_library
 
-To use this rule, you'll first need to add the following to your `WORKSPACE` file,
-which adds a few dependencies needed for ScalaPB:
+To use this rule, add the following configuration, which adds the dependencies
+needed for ScalaPB:
 
 ```py
-scala_toolchains(
-    # Other toolchains settings...
-    scala_proto = True,
-    scala_proto_options = [
+# MODULE.bazel
+scala_deps.scala_proto(
+    "default_gen_opts" = [
         "grpc",
         "flat_package",
         "scala3_sources",
     ],
 )
+```
+
+```py
+# WORKSPACE
+scala_toolchains(
+    # Other toolchains settings...
+    scala_proto = {
+        "default_gen_opts": [
+            "grpc",
+            "flat_package",
+            "scala3_sources",
+        ],
+    },
+)
 
 scala_register_toolchains()
 ```
 
+See the __scalapbc__ column of the [__ScalaPB: SBT Settings > Additional options
+to the generator__](
+https://scalapb.github.io/docs/sbt-settings#additional-options-to-the-generator
+) table for `default_gen_opts` values.
+
 Then you can import `scala_proto_library` in any `BUILD` file like this:
 
 ```py

scala/extensions/deps.bzl

@@ -14,8 +14,8 @@ Provides the `scala_deps` module extension with the following tag classes:
 - `twitter_scrooge`
 - `jmh`
 
-For documentation, see the `_tag_classes` dict, and the `_<TAG>_attrs` dict
-corresponding to each `<TAG>` listed above.
+For documentation, see the `_{general,toolchain}_tag_classes` dicts and the
+`_<TAG>_attrs` dict corresponding to each `<TAG>` listed above.
 
 See the `scala/private/macros/bzlmod.bzl` docstring for a description of
 the defaults, attrs, and tag class dictionaries pattern employed here.
@@ -27,6 +27,7 @@ load(
     "root_module_tags",
     "single_tag_values",
 )
+load("//scala/private:toolchain_defaults.bzl", "TOOLCHAIN_DEFAULTS")
 load("//scala:scala_cross_version.bzl", "default_maven_server_urls")
 load("//scala:toolchains.bzl", "scala_toolchains")
 
@@ -89,35 +90,26 @@ _compiler_srcjar_attrs = {
     "integrity": attr.string(),
 }
 
-_scalafmt_defaults = {
-    "default_config": "//:.scalafmt.conf",
-}
+_scalafmt_defaults = TOOLCHAIN_DEFAULTS["scalafmt"]
 
 _scalafmt_attrs = {
     "default_config": attr.label(
         default = _scalafmt_defaults["default_config"],
         doc = "The default config file for Scalafmt targets",
+        allow_single_file = True,
     ),
 }
 
-_scala_proto_defaults = {
-    "options": [],
-}
+_scala_proto_defaults = TOOLCHAIN_DEFAULTS["scala_proto"]
 
 _scala_proto_attrs = {
-    "options": attr.string_list(
-        default = _scala_proto_defaults["options"],
+    "default_gen_opts": attr.string_list(
+        default = _scala_proto_defaults["default_gen_opts"],
         doc = "Protobuf options, like 'scala3_sources' or 'grpc'",
     ),
 }
 
-_twitter_scrooge_defaults = {
-    "libthrift": None,
-    "scrooge_core": None,
-    "scrooge_generator": None,
-    "util_core": None,
-    "util_logging": None,
-}
+_twitter_scrooge_defaults = TOOLCHAIN_DEFAULTS["twitter_scrooge"]
 
 _twitter_scrooge_attrs = {
     k: attr.label(default = v)
@@ -186,39 +178,51 @@ _toolchain_tag_classes = {
     ),
 }
 
-_tag_classes = _general_tag_classes | _toolchain_tag_classes
+def _toolchain_settings(module_ctx, tags, tc_names, toolchain_defaults):
+    """Configures all builtin toolchains enabled throughout the module graph.
+
+    Configures toolchain options for enabled toolchains that support them based
+    on the root module's settings for each toolchain. In other words, it uses:
 
-def _toolchains(mctx):
-    result = {k: False for k in _toolchain_tag_classes}
+    - the root module's tag class settings, if present; and
+    - the default tag class settings otherwise.
 
-    for mod in mctx.modules:
-        values = {tc: len(getattr(mod.tags, tc)) != 0 for tc in result}
+    This avoids trying to reconcile different toolchain settings across the
+    module graph. Non root modules that require specific settings should either:
 
-        if mod.is_root:
-            return values
+    - publish their required toolchain settings, or
+    - define and register a custom toolchain instead.
 
-        # Don't overwrite `True` values with `False` from another tag.
-        result.update({k: v for k, v in values.items() if v})
+    Args:
+        module_ctx: the module context object
+        tags: a tags object, presumably the result of `root_module_tags()`
+        tc_names: names of all supported toolchains
+        toolchain_defaults: a dict of `{toolchain_name: default options dict}`
 
-    return result
+    Returns:
+        a dict of `{toolchain_name: bool or options dict}` to pass as keyword
+            arguments to `scala_toolchains()`
+    """
+    toolchains = {k: False for k in tc_names}
 
-def _scala_proto_options(mctx):
-    result = {}
+    for mod in module_ctx.modules:
+        values = {tc: len(getattr(mod.tags, tc)) != 0 for tc in toolchains}
 
-    for mod in mctx.modules:
-        for tag in mod.tags.scala_proto:
-            result.update({opt: True for opt in tag.options})
+        # Don't overwrite True values with False from another tag.
+        toolchains.update({k: v for k, v in values.items() if v})
 
-    return sorted(result.keys())
+    for tc, defaults in toolchain_defaults.items():
+        if toolchains[tc]:
+            values = single_tag_values(module_ctx, getattr(tags, tc), defaults)
+            toolchains[tc] = {k: v for k, v in values.items() if v != None}
+
+    return toolchains
+
+_tag_classes = _general_tag_classes | _toolchain_tag_classes
 
 def _scala_deps_impl(module_ctx):
     tags = root_module_tags(module_ctx, _tag_classes.keys())
-    scalafmt = single_tag_values(module_ctx, tags.scalafmt, _scalafmt_defaults)
-    scrooge_deps = single_tag_values(
-        module_ctx,
-        tags.twitter_scrooge,
-        _twitter_scrooge_defaults,
-    )
+    tc_names = [tc for tc in _toolchain_tag_classes]
 
     scala_toolchains(
         overridden_artifacts = repeated_tag_values(
@@ -229,13 +233,9 @@ def _scala_deps_impl(module_ctx):
             tags.compiler_srcjar,
             _compiler_srcjar_attrs,
         ),
-        scala_proto_options = _scala_proto_options(module_ctx),
-        # `None` breaks the `attr.string_dict` in `scala_toolchains_repo`.
-        twitter_scrooge_deps = {k: v for k, v in scrooge_deps.items() if v},
         **(
             single_tag_values(module_ctx, tags.settings, _settings_defaults) |
-            {"scalafmt_%s" % k: v for k, v in scalafmt.items()} |
-            _toolchains(module_ctx)
+            _toolchain_settings(module_ctx, tags, tc_names, TOOLCHAIN_DEFAULTS)
         )
     )
 
@@ -244,23 +244,26 @@ scala_deps = module_extension(
     tag_classes = _tag_classes,
     doc = """Selects and configures builtin toolchains.
 
-If the root module explicitly uses the extension, it assumes responsibility for
-selecting all required toolchains by insantiating the corresponding tag classes:
+Modules throughout the dependency graph can enable a builtin toolchain by
+instantiating its corresponding tag class. The root module controls the
+configuration of all toolchains it directly enables. Any other builtin
+toolchain required by other modules will use that toolchain's default
+configuration values.
 
 ```py
 scala_deps = use_extension(
     "@rules_scala//scala/extensions:deps.bzl",
     "scala_deps",
 )
 scala_deps.scala()
-scala_deps.scala_proto()
+scala_deps.scala_proto(default_gen_opts = ["grpc", "scala3_sources"])
 
 dev_deps = use_extension(
     "@rules_scala//scala/extensions:deps.bzl",
     "scala_deps",
     dev_dependency = True,
 )
-dev_deps.scalafmt()
+dev_deps.scalafmt(default_config = "path/to/scalafmt.conf")
 dev_deps.scalatest()
 
 # And so on...

scala/private/toolchain_defaults.bzl

@@ -0,0 +1,20 @@
+"""Gathers defaults for toolchain macros in one place.
+
+Used by both //scala:toolchains.bzl and //scala/extensions:deps.bzl.
+"""
+
+load(
+    "//scala/scalafmt/toolchain:setup_scalafmt_toolchain.bzl",
+    _scalafmt = "TOOLCHAIN_DEFAULTS",
+)
+load("//scala_proto:toolchains.bzl", _scala_proto = "TOOLCHAIN_DEFAULTS")
+load(
+    "//twitter_scrooge/toolchain:toolchain.bzl",
+    _twitter_scrooge = "TOOLCHAIN_DEFAULTS",
+)
+
+TOOLCHAIN_DEFAULTS = {
+    "scalafmt": _scalafmt,
+    "scala_proto": _scala_proto,
+    "twitter_scrooge": _twitter_scrooge,
+}

scala/scalafmt/toolchain/setup_scalafmt_toolchain.bzl

@@ -8,6 +8,13 @@ load("//scala:providers.bzl", "declare_deps_provider")
 load("//scala:scala_cross_version.bzl", "version_suffix")
 load("@rules_scala_config//:config.bzl", "SCALA_VERSIONS")
 
+TOOLCHAIN_DEFAULTS = {
+    # Used by `scala_toolchains{,_repo}` to generate
+    # `@rules_scala_toolchains//scalafmt:config`, the default config for
+    # `ext_scalafmt` from `phase_scalafmt_ext.bzl`.
+    "default_config": Label("//:.scalafmt.conf"),
+}
+
 def setup_scalafmt_toolchain(
         name,
         scalafmt_classpath,