With expressions

[munificent]

This strawman addresses #961 and partially addresses #314. It's just a strawman right now, so I'm putting it right in the issue here. If it gets traction, I'll move this text to a real proposal document.

Motivation

The #1 open feature request for Dart is data classes. This is a Kotlin feature that gives you a lightweight way to define a class with instance fields and value semantics. You give the class a name and some fields and it implicitly a constructor, equals(), hashCode(), destructuring support, and copy(). (Scala's case classes are similar.)

This proposal relates to the last method, copy(). Several languages that lean towards immutability have language or library features to support "functional update". This means creating a new object that has all of the same state as an existing object except with a few user-specified field values replaced. In Kotlin, it looks like:

data class Name(val first: String, val last: String?)

fun main() {
  var tom = Name("Tom", "Petty")
  var jones = tom.copy(last = "Jones") // <--
  println(jones)
}

On the marked line, we create a new Name object that has the same state as Name("Tom", "Petty") except that the last name has been replaced with "Jones". (You might think code like this is odd, but it's not unusual.)

You could imagine a similar API in Dart, here implemented manually:

class Name {
  final String first;
  final String? last;

  Name(this.first, this.last);

  Name copy({String? first, String? last}) =>
      Name(first ?? this.first, last ?? this.last);
}

main() {
  var tom = Name("Tom", "Petty");
  var jones = tom.copy(last: "Jones"); // <--
  print(jones);
}

This works fine. But note that the last field is nullable. Perhaps you want to create a new Name by removing a last name. This works fine in Kotlin:

var elvis = Name("Elvis", "Costello")
var theKing = elvis.copy(last = null)
println(theKing) // "Name(first=Elvis, last=null)"

But the corresponding Dart version does not:

var elvis = Name("Elvis", "Costello");
var theKing = elvis.copy(last: null);
print(theKing);

Here, the resulting object still has last name "Costello". The problem is that in Dart, there is no way to tell if a parameter was passed or not. All you can determine was whether the default value was used. In cases where the parameter is nullable and the default is also null, there's no way to distinguish "passed null" from "didn't pass a parameter".

We are working on adding macros to Dart to let packages automate generating methods like ==(), hashCode, etc., but macros don't help here. Macros can still only express the semantics that Dart gives you. Even with a very powerful macro language, you can't metaprogram a copyWith() method that handles update of nullable fields correctly.

This proposal sidesteps the issue by adding an expression syntax to directly express functional update. It is modeled after the with expression in C# 9.0, but mapped to Dart syntax and generalized to work with any class.

With expressions

A with expression looks exactly like a regular method call to a method named with:

class Name {
  final String first;
  final String? last;

  Name({this.first, this.last});
}

main() {
  var tom = Name(first: "Tom", last: "Petty");

  var jones = tom.with(last: "Jones"); // <--

  print(jones);
}

The grammar is simply:

selector ::= '!'
  | assignableSelector
  | argumentPart
  | (`.` `?.`) `with` arguments // <-- Added.

The semantics are a straightforward static desugaring to a constructor call:

1. Determine the static type T of the left-hand receiver expression.

2. Look for an unnamed constructor on that class. It is a compile-time error if T is not a class type with an unnamed constructor.

3. Evaluate the receiver expression to a fresh variable r.

4. Generate a call to T's unnamed constructor. Any arguments in the parenthesized argument list after with are passed to directly to it.

5. Then, for each named parameters in the constructor's parameter list:

   1. If an explicit argument with that name is passed, pass that.

   2. Else, if T defines a getter with the same name, invoke the getter on r and pass the result to that parameter. It is a compile-time error if the type of the getter is not assignable to the parameter's type.

6. It is a compile-time error if any required positional or named parameters do not end up with an argument.

7. If the token before with is ?., then wrap the entire invocation in:

   (r == null ? null : <generated constructor call>)

The result is an invocation of the receiver type's unnamed constructor. That produces a new instance of the same type. Any explicit parameters are passed to the constructor. Any absent named parameters that can be filled in by accessing matching fields on the original object are.

The problem with distinguishing "passed null" from "didn't pass a parameter" is addressed by the "If an explicit argument..." clause. We use the static presence of the argument to determine whether to inherit the existing object's value, and not the argument's value at runtime.

This desugaring only relies on the getters and constructor parameter list, so it works with any class whose shape matches. Most classes initialize their fields using initializing formals (this. parameters), so any class that does so using named constructor parameters can use this syntax.

The desugaring does not rely on any properties of the class that are not already part of its static API, so using with() on a class you don't control does not affect how the class's API can be evolved over time. (For example, the behavior does not depend on whether a particular constructor parameter happens to be an initializing formal or not.)

If records are added to Dart, we can easily extend with to support them, since they already support named fields and named getters.

The main limitation is that this syntax does not work well with classes whose constructor takes positional parameters. From my investigation, that's about half of the existing Dart classes in the wild. But my suspicion is that this pattern is most useful for classes that:

• Are immutable.
• Have a fairly large number of fields.
• Where it's meaningful to create an instance from various subsets of those fields.

Classes that fit those constraints are conveniently the classes most likely to use named parameters in their constructor. Also, much of the benefit of this feature is not having to write the argument names when constructing a new instance. Constructors with only positional arguments are already fairly terse.

Null-aware with

The proposal also supports a null-aware form:

Person? findPerson(String name) => ...

var jones = find("Tom")?.with(last: "Jones");
print(jones);

This is particularly handy, because the code you have to write in the absence of that is more cumbersome since you may need to cache the intermediate computation of the receiver:

Person? findPerson(String name) => ...

var maybeTom = find("Tom");
Person? jones;
if (maybeTom != null) {
  jones = Person(first: maybeTom.first, last: maybeTom.last);
}
print(jones);

Limitations

Positional parameters

This syntax can be used with positional parameters. It just means that the arguments must also be positional. This of somewhat limited use, but it does let you more easily copy objects whose constructor takes positional and named parameters.

Note that it does not fill in missing positional parameters:

var today = DateTime.now();
var past = today.with(1940, 5);

The resulting past value does not have the same date, hour, minutes, etc. as today. Those optional positional parameters are simply omitted. This is unfortunate, and probably a footgun for the proposal. But filling those parameters in implicitly based on their name would require making the name of a positional parameter meaningful in an API, which is currently not the case in Dart.

Named constructors

The proposed syntax does not allow calling a named constructor to copy the object. We could possibly extend it to allow:

var elvis = Name("Elvis", "Costello");
var theKing = elvis.with.mononym(last: null);
print(jones);

That looks fairly strange to me, but could work.

Alternatives

No . before with

We could eliminate the leading . if we want the syntax to be more visually distinct:

var jones = tom with(last: "Jones");

I think this looks nice. Extending it to support named constructors looks more natural to me:

var theKing = elvis with.mononym(last: null);

However, it makes it harder to come up with a null-aware form that looks reasonable. Also the precedence of with relative to the receiver and any surrounding expression is less clear.

[Levi-Lesches]

The problem with distinguishing "passed null" from "didn't pass a parameter" is addressed by the "If an explicit argument..." clause. We use the static presence of the argument to determine whether to inherit the existing object's value, and not the argument's value at runtime.

So it seems the problem is the general idea of passing null vs not passing anything.

There are other issues, like #577, #140, and #137. Specifically #577 (credits to @jodinathan) and these two comments from @tatumizer and @lrhn introduce the idea of using the already-reserved default keyword to denote "no parameter". Combine that with non-const default parameters, you have a solution that has several benefits over with.

1. It doesn't have the limitations you listed
2. It can work for any method, not just copyWith
3. It can work for any top-level function, where objects aren't relevant
4. One can programmatically omit an argument depending on a condition.

It would look something like this:

class User {
  final String username;
  final String? email;
  const User({required this.username, this.email});

  // assuming non-const default parameters are allowed, this gives us what we want
  // now, passing in null makes the value null, and omitting the value entirely gives the default.
  User copyWith({String? username = this.username, String? email = this.email}) => User(
    username: username, 
    email: email,
  );
}

void main() {
  // create a reference user
  final alice = User(username: "alice", email: "alice@gmail.com");

  // regular copyWith material, nothing novel here
  final bob = alice.copyWith(username: "bob", email: "bob@gmail.com");

  // we're able to remove the email by explicitly passing null
  final aliceUnverified = alice.copyWith(email: null); 

  // we're able to choose whether to keep the email or not 
  bool isVerified = getAuthStatus();
  final aliceSynced = alice.copyWith(email: isVerified ? default : null);
}

[lrhn]

The problem of default values comes when all type-valid arguments are also valid and reasonable values, so there is no value left over to be the sentinel default value.

An approach to that could be to allow parameters to have types different from the local variable they introduce. Example:

void foo(int x as num) { ... }

would declare a void Function(int) function with a local variable of type num (always intiially assigned an int, possibly even promoted to int).
The default value would then only need to be assignable to the local variable, so you could do:

void foo(int? x as num? = double.nan) { ... }

and accept any int or null, and still detect that no parameter was passed.
(The feature is also useful in the other direction: void bar(Object? x as int) { ... } which will throw at run-time if the argument isn't actually an int, like we do for covariant instance method parameters).

(I still want to move to a parameter passing strategy where you cannot distinguish passing null from not passing an argument, and then there will be no way to write a with that accepts overriding with null.)

[Levi-Lesches]

(I still want to move to a parameter passing strategy where you cannot distinguish passing null from not passing an argument, and then there will be no way to write a with that accepts overriding with null.)

Can you elaborate on this? How then will the copyWith problem, where the difference between not passing an argument and passing in null is crucial, be solved?

[lrhn]

(It won't be solved if you can't distinguish passing null from passing nothing. It'll be impossible. That's something I'm personally willing to sacrifice for a much nice parameter passing protocol, one where you can always forward parameters to another function without knowing its default values, because you can always pass null to mean "no argument". I think that's a bigger problem than copyWith . But that's a separate problem, and different from the with proposal, which could behave differently than a function call.)

[jakemac53]

1. Determine the static type T of the left-hand receiver expression.

I am concerned about this being based on the static type. I may have a function that wants to change the name of a Person, and if somebody passes me a Manager I don't want to end up converting that to a Person and dropping all the extra information contained in Manager. The macro/codegen approaches all avoid this problem by just creating copyWith methods specialized to each class (the subclasses are only adding non-optional params).

2. Else, if T defines a getter with the same name, invoke the getter on r and pass the result to that parameter. It is a compile-time error if the type of the getter is not assignable to the parameter's type.

I also share the general concern @tatumizer was I think trying to express about this part. In Dart constructor parameters don't have a 1:1 relationship with fields. For lots of classes these with expressions will just be totally useless (or even dangerous because they will not do what is expected).

I would somehow want this feature to be restricted to classes that meet the definition of a "data class" - basically classes where every field is initialized with an initializing formal in the unnamed constructor, and there are no additional parameters.

[jakemac53]

As a more meta comment, I think I would rather see us solve this problem by solving the more general problem around null versus undefined (in particular for parameters). I don't think we want to go the route of adding undefined as an actual value, but some other solution to the problem might exist?

[jodinathan]

@jakemac53 instead of undefined, what about the default keyword in #577?

[jodinathan]

I find the foo(x: if (cond) 42) idea less hacky and more consistent (rhymes with "if" in collections). Though these two things are not equivalent in general, in the context of "with" they are.

This is interesting, however, it seems that omitting a parameter is something that the compiler can do at compile time. For this to work the default value of a parameter must be known at runtime because you may need it or not depending on the condition result.

I like the default keyword and foo(x: if (cond) 42) being a syntax sugar for its use, ie:

foo(if (cond) x: 42);

// desugars to

foo(x: cond ? 42 : default);

// then you can easily use it with multiple parameters
foo(if (cond) x: 42, if (otherCond) y: 123);

// desugars to 
foo(x: cond ? 42 : default, y: otherCond ? 123 : default);

The point is that the default keyword is important for you to state to the compiler that you need to know the default value of the parameter.

[jakemac53]

@jakemac53 instead of undefined, what about the default keyword in #577?

I don't think that solves the copyWith use case, unfortunately. Or at least it isn't clear to me how.

[stereotype441]

The desugaring does not rely on any properties of the class that are not already part of its static API, so using with() on a class you don't control does not affect how the class's API can be evolved over time.

I can't decide if I'm nit-picking here, but I don't think this reasoning is correct. Consider this (admittedly bizarre) API evolution:

Before:

class C {
  int x;
  C({this.x});
}

After:

class C {
  int x;
  int? y;
  C({this.x, String? y}) : y = y == null ? null : int.parse(y);
}

If no callers use with, this is a non-breaking change*, because any calls to C's constructor that worked before the change will also work after it. But if there's a call site that does this:

f(C c) => c.with(x: 0);

then that suddenly becomes a compile error because int? is not assignable to String?

I realize that this is a totally bizarre example, so maybe we don't care. I just wanted to raise awareness of it.

*Okay, actually I lied here. Technically, any change to a constructor signature is a breaking change because of constructor tearoffs. So maybe my whole argument is moot? I dunno.

[stereotype441]

I spent a while thinking about whether I could have used this feature in the implementation of flow analysis. It uses a lot of immutable data structures and updates them in a very "with"-like way, so it seems like a good candidate. For example, check out this code from the VariableModel class:

class VariableModel<Variable extends Object, Type extends Object> {
  final List<Type>? promotedTypes;
  final List<Type> tested;
  final bool assigned;
  final bool unassigned;
  final SsaNode<Variable, Type>? ssaNode;
  final NonPromotionHistory<Type>? nonPromotionHistory;
  final Map<String, VariableModel<Variable, Type>> properties;

  VariableModel(
      {required this.promotedTypes,
      required this.tested,
      required this.assigned,
      required this.unassigned,
      required this.ssaNode,
      this.nonPromotionHistory,
      this.properties = const {}}) { ... }

  VariableModel<Variable, Type> write(...) {
    ...
    return new VariableModel<Variable, Type>( // (1)
        promotedTypes: promotedTypes,
        tested: tested,
        assigned: assigned,
        unassigned: unassigned,
        ssaNode: newSsaNode);
    ...
  }
  ...
}

This proposal would let me replace the statement at (1) with:

    return this.with(
        ssaNode: newSsaNode,
        nonPromotionHistory: null,
        properties: const {});

Which is actually a lot better than what it looked like before. Note that not only did the use of with allow me to drop the promotedTypes, tested, assigned, and unassigned arguments, but it also forced me to explicitly state that I want to set nonPromotionHistory to null and properties to const {}. It's that last part that gets me really excited, because previously, the fact that those two fields were changing was really non-obvious, and now it's totally clear.

[Levi-Lesches]

@tatumizer:

WIth "with", default value is supposed to be copied over from the old instance. In a constructor call, "default" would mean a constant value defined in the initializer.

To distinguish between "omitted value" and "copying over from the old instance", simply set the default in the signature to this.X, so that omitting a value defaults to the instance's current value (depends on the ability to set non-const default parameter values).

@jakemac53:

I don't think that solves the copyWith use case, unfortunately. Or at least it isn't clear to me how.

From my comment above, combining default with non-const default parameters does solve this isssue. See my example there how to implement a User.copyWith.

[jakemac53]

From my comment above, combining deafult with non-const default parameters does solve this isssue. See my example there how to implement a User.copyWith.

True, it would. I don't know what the reasons are for making default values be const so I have no idea as to the feasibility of that part.