Clarification for non-portable behavior for in-place Python operations

[oleksandr-pavlyk]

The wording for in-place operators may be more explicit to warn users that in-place operations, e.g., x1 += x2, where Type Promotion Rules require x1 + x2 to have data type different from data type of x1 are implementation defined.

Present wording hints at it:

For example, after in-place addition x1 += x2, the modified array x1 must always equal the result of the equivalent binary arithmetic operation x1 = x1 + x2.

but states that result of the in-place operation must "equal" the result of the out-of-place operation and the equality may hold true for arrays of different data types.

[asmeurer]

Did we agree that it should always be implementation defined if the types are different, or should x1 += x2 be OK if the type promoted type is the same as x1.dtype (i.e., x2 upcasts to x1)?

[kgryte]

should x1 += x2 be OK if the type promoted type is the same as x1.dtype (i.e., x2 upcasts to x1)?

@asmeurer I believe that is what @oleksandr-pavlyk was getting at in the OP. IMO, it should be okay to allow type promotion to x1.dtype and we can be more explicit in stating that the relationship x1 = x1 + x2 must hold provided the constraints of "limited" type promotion are satisfied.

[asmeurer]

So there are two cases:

Case 1: x1 + x2 promotes to x1.dtype

x1 = asarray([0], dtype=int64)
x2 = asarray([0], dtype=int32)
x1 += x2

Case 2: x1 + x2 promotes to x2.dtype

x1 = asarray([0], dtype=int32)
x2 = asarray([0], dtype=int64)
x1 += x2

I think @oleksandr-pavlyk was asking about case 2:

... where Type Promotion Rules require x1 + x2 to have data type different from data type of x1 ...

In my opinion, this is actually spelled out already https://data-apis.org/array-api/latest/API_specification/array_object.html#in-place-operators:

An in-place operation must not change the data type or shape of the in-place array as a result of Type Promotion Rules or Broadcasting.

In other words, case 2 is currently required to error (which is stronger than implementation defined).

Case 1 is perhaps more ambiguous whether it is required or not. My reading of the current text is that it is. If we want to make it implementation defined, we should explicitly state that. I'm not aware of any reasons why it would be a problem, though.

[ndgrigorian]

In other words, case 2 is currently required to error (which is stronger than implementation defined).

Does this make current NumPy behavior non-compliant, then?

Because in NumPy

In [26]: x_np = np.arange(10, dtype="i4")

In [27]: x_np += np.ones(10, dtype="i8")

In [28]: x_np
Out[28]: array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10], dtype=int32)

it simply casts the second array into the type of the first array per same_kind casting.

And per the spec

An in-place operation must have the same behavior (including special cases) as its respective binary (i.e., two operand, non-assignment) operation. For example, after in-place addition x1 += x2, the modified array x1 must always equal the result of the equivalent binary arithmetic operation x1 = x1 + x2.

This seems like a very strong condition implying that x1 += x2 of this case should be disallowed, and may be an even stronger condition than what you've quoted, because the data type of x1 (the in-place array) does not change.

[asmeurer]

I think the x1 = x1 + x2 sentence is not really intended to be saying anything about the data type of x1. It's only saying that whatever behavior is specified in __add__ (and thus add()) also applies to the += operator, i.e., all the special cases, nan behaviors, and so on. The wording should probably be improved since this isn't clear.

The sentence before that, which I quoted, "An in-place operation must not change the data type or shape of the in-place array..." does indeed imply that NumPy is currently noncompliant, because it uses must and not should. We could potentially loosen this to be implementation defined. I guess one question is whether the NumPy team agrees that this is nonideal behavior and should be deprecated. I certainly find the behavior surprising (it really is an in-place change of dtype: even views of x1 are updated to the promoted dtype).

Also we should check if other libraries allow this. This is the behavior in PyTorch:

>>> x1 = torch.asarray([0], dtype=torch.int32)
>>> x2 = torch.asarray([1], dtype=torch.int64)
>>> x1 += x2
>>> x1
tensor([1], dtype=torch.int32)

That actually could arguably be within what the spec says, because it didn't change the dtype of x1. So we should be clear whether this is actually OK, or whether it should be an error (or implementation defined).

[lucascolley]

I think the x1 = x1 + x2 sentence is not really intended to be saying anything about the data type of x1. It's only saying that whatever behavior is specified in add (and thus add()) also applies to the += operator, i.e., all the special cases, nan behaviors, and so on. The wording should probably be improved since this isn't clear.

Yes, it would be good to clarify there that the dtype may be different, due to:

An in-place operation must not change the data type or shape of the in-place array as a result of Type Promotion Rules or Broadcasting.

---

The sentence before that, which I quoted, "An in-place operation must not change the data type or shape of the in-place array..." does indeed imply that NumPy is currently noncompliant, because it uses must and not should.

Can you give an example of NumPy's non-compliance with that? The example in #828 (comment) looks to be doing exactly what is specified:

In [1]: import numpy as np

In [2]: x = np.arange(10, dtype=np.int32)

In [3]: y = np.ones(10, dtype=np.int64)

In [4]: (x + y).dtype
Out[4]: dtype('int64')

In [5]: x += y

In [6]: x.dtype
Out[6]: dtype('int32')

[jakevdp]

Does this mean that libraries which don't explicitly define __iadd__ are out of compliance? JAX is an example: __iadd__ is undefined, so it falls back to the __add__ behavior, which differs from NumPy's behavior above:

In [1]: import jax

In [2]: jax.config.update('jax_enable_x64', True)

In [3]: import jax.numpy as jnp

In [4]: x = jnp.arange(10, dtype=jnp.int32)

In [5]: y = jnp.ones(10, dtype=jnp.int64)

In [6]: (x + y).dtype
Out[6]: dtype('int64')

In [7]: x += y

In [8]: x.dtype  # type is promoted, unlike NumPy
Out[9]: dtype('int64')

The reason is that in Python when __iadd__ is not defined, x += y falls back to x = x + y. JAX cannot define __iadd__ because its arrays are immutable.

[lucascolley]

that does look like non-compliance currently, yes

[jakevdp]

So effectively then, the array API specification is saying "compilant libraries must overload __iadd__", because the behavior when falling back to a compilant implementation of __add__ is non-compliant.

[rgommers]

Hmm, that's an issue that wasn't thought about before I think. On the one hand, requiring that x += y doesn't upcast x makes perfect sense to me as a design rule, because (a) it's what the user would most likely expect when reading that code (they're called "in-place operators" after all, so changing dtype and memory footprint is unexpected), and (b) if we leave it free then the dtype becomes unpredictable, depending on whether __iadd__ was defined or not.

On the other hand, no one brought up this problem and intended to mark JAX as non-compliant here. The Python language was also not really designed with this in mind, casting rules like we need for numerical libraries weren't really a consideration, even the simpler numerical tower for Python's scalars came later (and is now considered a mistake).

Perhaps the easy way out is to allow both, and warn that only equal dtypes result in fully portable behavior and the rest is implementation-defined?

I guess one question is whether the NumPy team agrees that this is nonideal behavior and should be deprecated. I certainly find the behavior surprising (it really is an in-place change of dtype: even views of x1 are updated to the promoted dtype).

I do agree that that downcasting numpy does is bad. it uses 'same_kind' casting, and that probably should be 'safe' instead:

>>> x = np.arange(3)
>>> x += np.array([0, 1, 2.5])
...
UFuncTypeError: Cannot cast ufunc 'add' output from dtype('float64') to dtype('int64') with casting rule 'same_kind'

[asmeurer]

The JAX behavior isn't an "in-place" change, though, which is what the current standard text talks about. This is just the Python variable x being updated to point to a new object (it will have a different id()).

An in-place operation must not change the data type or shape of the in-place array as a result of Type Promotion Rules or Broadcasting.