From 3d131f6df6f0e3c762a07beb266ebf4657e9dfe7 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:35:31 +0800 Subject: [PATCH 1/8] . --- content/reference-package-objects.md | 242 +++++++++++++++++++++++++++ 1 file changed, 242 insertions(+) create mode 100644 content/reference-package-objects.md diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md new file mode 100644 index 0000000..0a921b7 --- /dev/null +++ b/content/reference-package-objects.md @@ -0,0 +1,242 @@ +--- +layout: sip +permalink: /sips/:title.html +stage: implementation +status: under-review +title: SIP-XX - Reference-able Package Objects +--- + +**By: Li Haoyi** + +## History + +| Date | Version | +|---------------|--------------------| +| Dec 14th 2024 | Initial Draft | + +## Summary + +This proposal is to allow the following: + +```scala +package a +package object b + +val z = a.b // Currently fails with "package is not a value" +``` + + +Currently the workaround is to use a `.package` suffix: + +```scala +val z = a.b.`package` +``` + +This proposal is to make it such that given `a.b`, if `b` is a `package` +containing a `package object`, expands to `a.b.package` automatically + + +One limitation with `package object`s is that we cannot currently assign them to +values: `a.b` fails to compile when `b` is a `package object`, even though it succeeds when +`b` is a normal `object`. The workaround is to call `a.b.package`, which is ugly and +non-obvious, or to use a normal `object`, which is not always possible. There is no other +way to refer to the `package object b` in the example above. + +This proposal is to allow `a.b` to automatically expand into `a.b.package` when `b` is a +`package object`. Such usage will simplify the language, simplify IDE support for the +language, and generally make things more uniform and regular. + + +Prior Discussion can be found [here](https://contributors.scala-lang.org/t/pre-sip-reference-able-package-objects/6939) + +## Motivation + +Although package objects have been discussed [being dropped](https://docs.scala-lang.org/scala3/reference/dropped-features/package-objects.html) +in Scala 3, no concrete plans have been made as to how to do so, and we argue that they +are sufficiently useful that keeping them around is preferably to dropping them. + +### Package Entrypoints + +`package object`s are the natural "entry point" of a package. While top-level declarations +reduce their need somewhat, they do not replace it: `package object`s are still necessary +for adding package-level documentation or having the package-level API inherit from traits +or classes. Other languages have equivalent constructs (`module-info.java` or `__init__.py`) +that fulfil the same need, so it's not just a quirk of the Scala language. The +[Acyclic Plugin](https://github.com/com-lihaoyi/acyclic) uses package objects as a place to +put package-level annotations in source code to apply package-level semantics in the compiler +plugin. + +### Package API Facades + +Many libraries use package objects to expose the "facade" of the package hierarchy: + +- Mill uses `package object`s to expose the build definitions within each `package`, and +- each one is an instance of `mill.Module` + +- Requests-Scala uses a `package object` to represent the default `requests.BaseSession` +- instance with the default configuration for people to use + +- PPrint uses a `package object` to expose the `pprint.log` and other APIs for people to use +- directly, as a default instance of `PPrinter` + +- OS-Lib uses a `package object` to expose the primary API of the `os.*` operations + +None of these use cases can be satisfied by normal `object`s or by top-level declarations, +due to the necessity of documentation and inheritance. They need to be `package object`s. + +However, the fact that you cannot easily pass around these default instances as values e.g. +`val x: PPrinter = pprint` without calling `pprint.package` is a source of friction. + +### Uniform Semantics + +This source of friction is not just for humans, but for tools as well. For example, IntelliJ +needs a special case and special handling in the Scala plugin specifically to support this irregularity: + +* Original irregularity https://github.com/JetBrains/intellij-scala/blob/idea242.x/scala/scala-impl/src/org/jetbrains/plugins/scala/lang/psi/impl/expr/ScReferenceExpressionImpl.scala#L198 + +* Special casing to support Mill, which allows references to package objects https://github.com/JetBrains/intellij-scala/pull/672 + + + +## Limitations + +* `a.b` only expands to `a.b.package` when used "standalone", i.e. not when part of a + larger select chain `a.b.c` or equivalent postfix expression `a.b c`, prefix + expression `!a.b`, or infix expression `a.b c d`. + +* `a.b` expands to `a.b.package` of the type `a.b.package.type`, and only contains the + contents of the `package object`. It does not contain other things in the + `package a.b` that are outside of the `package object` + +Both these requirements are necessary for backwards compatibility, and anyway do not impact +the main goal of removing the irregularity between `package object`s and normal `object`s +and providing some non-hacky way to refer to the `package object` values + + +## User Alternatives + +The two main alternatives now are to use `.package` suffixes, e.g. in Mill writing: + +```scala +def moduleDeps = Seq(foo.`package`, bar.`package`, qux.baz.`package`) +``` + +Or to use normal `object`s. Notably, normal `object`s do not allow `package`s of the +same name, which leads to contortions. e.g. Rather than: + +```scala +package object foo extends _root_.foo.bar.Qux{ + val bar = 1 +} +``` +```scala +package foo.bar +class Qux +``` + +We need to move the `package foo` contents into `package foo2` to avoid conflicts with +`object foo`, and then we need to add back aliases to all the declarations in `foo2` to make +them available in `foo`: + +```scala +object foo extends foo2.bar.Qux{ + val bar = 1 + object bar{ + type Qux = foo2.bar.Qux + } +} +``` +```scala +package foo2.bar +class Qux +``` + +Both of these workarounds are awkward and non-idiomatic, but are necessary due to current +limitations in referencing `package object`s directly + +Notably, normal `object`s are not a replacement for `package object`s, because only +`package object`s allow the package contents to be defined in other files. Normal `object`s +would require that the package contents be all defined in a single file in the `object` body, +or scattered into other files as `trait`s in _different_ `package`s and mixed into the +`object`, both of which are messy and sub-optimal. + +It's possible to have a convention _"the `object` named `foo` is always going to be the +primary entrypoint for a package"_, but that is just a poor-man's `package object` with worse +syntax and less standardization. + +## Implementation Alternatives + +* We could make `a.b` where `b` is a `package` refer to the entire `package b` namespace, not + just the `package object`. This cannot in general work due to the JVM's _open packages_ and + separate compilation: while `package object`s can only exist in one file present in one + compilation run, JVM `package`s can contain arbitrary sets of classes from different compilation + runs. Thus it is impossible in general to define a "complete" API for a JVM `package` for us to + generate an object to refer to. + +* Using Scala 3 [Top Level Definitions](https://docs.scala-lang.org/scala3/book/taste-toplevel-definitions.html) + is one possible alternative to `package object`s, but they fall short on many use cases: + * Top-level definitions cannot generate objects that inherit from classes or traits, which + is necessary in many use cases: Mill (needs them to inherit `mill.Module`), Requests- + Scala (needs it to inherit from `requests.BaseSession`), etc. + * Top-level definitions can be defined in multiple files, so suffer from the issue that + it is at any point in time impossible to know the "entire" API of a `package` provided + by top-level definitions + * Top-level definitions do not provide a natural "package entrypoint" to the `package` source folder, + to provide package-level documentation, annotations, etc.. We could provide another `.scala` + file that we specify by-convention to be the "package entrypoint", but we already have + `package.scala` and it does the job just fine + +## Implementation & Testing + + +Mill since version 0.12.0 already emulates this proposed behavior in Scala 2 using source-code +mangling hacks, with custom support in IntelliJ. It works great and does what it was intended +to do (allow passing around `package object`s as values without having to call `.package` every time) + +We have a prototype Scala3 implementation here: + +* https://github.com/scala/scala3/pull/22011 + +The necessary IntelliJ changes have been made below: + +* https://github.com/JetBrains/intellij-scala/pull/672 + +These IntelliJ changes are currently guarded to only apply to `.mill` files, but the +guard can easily be removed to make it apply to any Scala files. In fact, implementing +this proposal would involve _removing_ a considerable amount of special casing from +the Intellij-Scala plugin, resulting in the code analysis for looking up references in +the Scala language to become much more regular and straightforward: + +```diff +lihaoyi intellij-scala$ git diff +diff --git a/scala/scala-impl/src/org/jetbrains/plugins/scala/lang/psi/impl/expr/ScReferenceExpressionImpl.scala b/scala/scala-impl/src/org/jetbrains/plugins/scala/lang/psi/impl/expr/ScReferenceExpressionImpl.scala +index b820dff8c3..29ba15bcdd 100644 +--- a/scala/scala-impl/src/org/jetbrains/plugins/scala/lang/psi/impl/expr/ScReferenceExpressionImpl.scala ++++ b/scala/scala-impl/src/org/jetbrains/plugins/scala/lang/psi/impl/expr/ScReferenceExpressionImpl.scala +@@ -182,24 +182,7 @@ class ScReferenceExpressionImpl(node: ASTNode) extends ScReferenceImpl(node) wit + }) + + override def getKinds(incomplete: Boolean, completion: Boolean = false): _root_.org.jetbrains.plugins.scala.lang.resolve.ResolveTargets.ValueSet = { +- val context = getContext +- context match { +- case _ if completion => +- StdKinds.refExprQualRef // SCL-3092 +- case _: ScReferenceExpression => +- StdKinds.refExprQualRef +- case postf: ScPostfixExpr if this == postf.operation || this == postf.getBaseExpr => +- StdKinds.refExprQualRef +- case pref: ScPrefixExpr if this == pref.operation || this == pref.getBaseExpr => +- StdKinds.refExprQualRef +- case inf: ScInfixExpr if this == inf.operation || this == inf.getBaseExpr => +- StdKinds.refExprQualRef +- case _ => +- // Mill files allow direct references to package +- // objects, even though normal .scala files do not +- if (this.containingScalaFile.exists(_.isMillFile)) StdKinds.refExprQualRef +- else StdKinds.refExprLastRef +- } ++ StdKinds.refExprQualRef + } + + override def multiType: Array[TypeResult] = { +``` \ No newline at end of file From d442b2f7f749b897ce96bf02b731c849c0094855 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:36:51 +0800 Subject: [PATCH 2/8] . --- content/reference-package-objects.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 0a921b7..24a6d38 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -42,8 +42,8 @@ values: `a.b` fails to compile when `b` is a `package object`, even though it su non-obvious, or to use a normal `object`, which is not always possible. There is no other way to refer to the `package object b` in the example above. -This proposal is to allow `a.b` to automatically expand into `a.b.package` when `b` is a -`package object`. Such usage will simplify the language, simplify IDE support for the +Allowing `a.b` to automatically expand into `a.b.package` when `b` is a +`package object` will simplify the language, simplify IDE support for the language, and generally make things more uniform and regular. From 932098bf9b781d5f845e21c9829d547610361bc5 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:37:48 +0800 Subject: [PATCH 3/8] . --- content/reference-package-objects.md | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 24a6d38..289c386 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -60,11 +60,12 @@ are sufficiently useful that keeping them around is preferably to dropping them. `package object`s are the natural "entry point" of a package. While top-level declarations reduce their need somewhat, they do not replace it: `package object`s are still necessary for adding package-level documentation or having the package-level API inherit from traits -or classes. Other languages have equivalent constructs (`module-info.java` or `__init__.py`) -that fulfil the same need, so it's not just a quirk of the Scala language. The -[Acyclic Plugin](https://github.com/com-lihaoyi/acyclic) uses package objects as a place to -put package-level annotations in source code to apply package-level semantics in the compiler -plugin. +or classes. For example the [Acyclic Plugin](https://github.com/com-lihaoyi/acyclic) uses package +objects as a place to put package-level annotations in source code to apply package-level +semantics in the compiler plugin. + +Other languages have equivalent constructs (`module-info.java` or `__init__.py`) +that fulfil the same need, so it's not just a quirk of the Scala language. ### Package API Facades From b221c798c8ca575677301440540017240f8aa392 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:38:02 +0800 Subject: [PATCH 4/8] . --- content/reference-package-objects.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 289c386..917e3ac 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -72,13 +72,13 @@ that fulfil the same need, so it's not just a quirk of the Scala language. Many libraries use package objects to expose the "facade" of the package hierarchy: - Mill uses `package object`s to expose the build definitions within each `package`, and -- each one is an instance of `mill.Module` + each one is an instance of `mill.Module` - Requests-Scala uses a `package object` to represent the default `requests.BaseSession` -- instance with the default configuration for people to use + instance with the default configuration for people to use - PPrint uses a `package object` to expose the `pprint.log` and other APIs for people to use -- directly, as a default instance of `PPrinter` + directly, as a default instance of `PPrinter` - OS-Lib uses a `package object` to expose the primary API of the `os.*` operations From f98adee2e2a9c66ca4c0d03129b1aad0282b2f12 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:40:10 +0800 Subject: [PATCH 5/8] . --- content/reference-package-objects.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 917e3ac..a546eae 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -97,7 +97,11 @@ needs a special case and special handling in the Scala plugin specifically to su * Special casing to support Mill, which allows references to package objects https://github.com/JetBrains/intellij-scala/pull/672 - +The fact that it is impossible to refer to the `package object` without using a `.package` suffix +is a wart: `.package` is an implementation/encoding detail, and so should not be a necessary part +of the user-facing language. We can refer to all other Scala definitions and objects without +leaking implementation/encoding details, and it would be more uniform to allow that for +`package object`s as well. ## Limitations From c287b4e6e07b8c3b66dcb20c8f7944aabead2be6 Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:40:43 +0800 Subject: [PATCH 6/8] . --- content/reference-package-objects.md | 30 +++++++++++++++------------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index a546eae..111e962 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -103,20 +103,6 @@ of the user-facing language. We can refer to all other Scala definitions and obj leaking implementation/encoding details, and it would be more uniform to allow that for `package object`s as well. -## Limitations - -* `a.b` only expands to `a.b.package` when used "standalone", i.e. not when part of a - larger select chain `a.b.c` or equivalent postfix expression `a.b c`, prefix - expression `!a.b`, or infix expression `a.b c d`. - -* `a.b` expands to `a.b.package` of the type `a.b.package.type`, and only contains the - contents of the `package object`. It does not contain other things in the - `package a.b` that are outside of the `package object` - -Both these requirements are necessary for backwards compatibility, and anyway do not impact -the main goal of removing the irregularity between `package object`s and normal `object`s -and providing some non-hacky way to refer to the `package object` values - ## User Alternatives @@ -191,6 +177,22 @@ syntax and less standardization. file that we specify by-convention to be the "package entrypoint", but we already have `package.scala` and it does the job just fine + +## Limitations + +* `a.b` only expands to `a.b.package` when used "standalone", i.e. not when part of a + larger select chain `a.b.c` or equivalent postfix expression `a.b c`, prefix + expression `!a.b`, or infix expression `a.b c d`. + +* `a.b` expands to `a.b.package` of the type `a.b.package.type`, and only contains the + contents of the `package object`. It does not contain other things in the + `package a.b` that are outside of the `package object` + +Both these requirements are necessary for backwards compatibility, and anyway do not impact +the main goal of removing the irregularity between `package object`s and normal `object`s +and providing some non-hacky way to refer to the `package object` values + + ## Implementation & Testing From 3cf958bbfc528fe697e45775faf331428c047c5c Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 14 Dec 2024 09:46:14 +0800 Subject: [PATCH 7/8] . --- content/reference-package-objects.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 111e962..7caa32c 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -208,6 +208,10 @@ The necessary IntelliJ changes have been made below: * https://github.com/JetBrains/intellij-scala/pull/672 +With IntelliJ-side discussion: + +* https://youtrack.jetbrains.com/issue/SCL-23198/Direct-references-to-package-objects-should-be-allowed-in-.mill-files + These IntelliJ changes are currently guarded to only apply to `.mill` files, but the guard can easily be removed to make it apply to any Scala files. In fact, implementing this proposal would involve _removing_ a considerable amount of special casing from From 98722944fb1e6c2467caa55acc94cd27a09dab0e Mon Sep 17 00:00:00 2001 From: Li Haoyi Date: Sat, 25 Jan 2025 09:10:44 +0800 Subject: [PATCH 8/8] Update reference-package-objects.md --- content/reference-package-objects.md | 20 ++++++++++---------- 1 file changed, 10 insertions(+), 10 deletions(-) diff --git a/content/reference-package-objects.md b/content/reference-package-objects.md index 7caa32c..1bb3041 100644 --- a/content/reference-package-objects.md +++ b/content/reference-package-objects.md @@ -180,18 +180,18 @@ syntax and less standardization. ## Limitations -* `a.b` only expands to `a.b.package` when used "standalone", i.e. not when part of a - larger select chain `a.b.c` or equivalent postfix expression `a.b c`, prefix - expression `!a.b`, or infix expression `a.b c d`. +* With this proposal, `a.b.c` can be refactored to `val x = a.b; x.c` only when `c` is declared inside + the `a.b` package object. This is slightly more irregular than the status quo, which disallows such + a refactoring at any time. In general, a package with a package object no longer behaves the same + as a package without. -* `a.b` expands to `a.b.package` of the type `a.b.package.type`, and only contains the - contents of the `package object`. It does not contain other things in the - `package a.b` that are outside of the `package object` +## Open Questions -Both these requirements are necessary for backwards compatibility, and anyway do not impact -the main goal of removing the irregularity between `package object`s and normal `object`s -and providing some non-hacky way to refer to the `package object` values +There are some open questions that can be resolved during experimentation +* Should package objects be usable as singleton type prefixes, e.g. `type foo == scala.type`? +* Should package objects participate in `foo() -> foo.apply()` desugaring, e.g. `_root_.pprint(124)`? +* ## Implementation & Testing @@ -250,4 +250,4 @@ index b820dff8c3..29ba15bcdd 100644 } override def multiType: Array[TypeResult] = { -``` \ No newline at end of file +```