Code.string_to_quoted's :column option changed behavior in Elixir 1.16 and onwards

[doorgan]

Elixir and Erlang/OTP versions

Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.15.8 (compiled with Erlang/OTP 24)

---

Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.16.0 (compiled with Erlang/OTP 24)

Operating system

macOs Sequoia 15.3.2

Current behavior

I think there is a bug introduced in elixir 1.16 when passing :column to Code.string_to_quoted

In elixir 1.15 it would change the starting column of the code, and then it would reset on subsequent newlines:

dorgan@Lucass-MacBook-Pro:~/dev/sourceror/ > asdf local elixir 1.15
dorgan@Lucass-MacBook-Pro:~/dev/sourceror/ > iex
Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Interactive Elixir (1.15.8) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> string =
...(1)> """
...(1)> def foo do
...(1)>   :ok
...(1)> end
...(1)> """
"def foo do\n  :ok\nend\n"
iex(2)> Code.string_to_quoted!(string, column: 20, token_metadata: true, columns: true)
{:def,
 [do: [line: 1, column: 28], end: [line: 3, column: 1], line: 1, column: 20],
 [{:foo, [line: 1, column: 24], nil}, [do: :ok]]}

But since elixir 1.16 it treats column as an indentation value, notice that now the end metadata column is shifted too:

dorgan@Lucass-MacBook-Pro:~/dev/sourceror/ > asdf local elixir 1.16
dorgan@Lucass-MacBook-Pro:~/dev/sourceror/ > iex
Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Interactive Elixir (1.16.0) - press Ctrl+C to exit (type h() ENTER for help)
iex(1)> string =
...(1)> """
...(1)> def foo do
...(1)>   :ok
...(1)> end
...(1)> """
"def foo do\n  :ok\nend\n"
iex(2)> Code.string_to_quoted!(string, column: 20, token_metadata: true, columns: true)
{:def,
 [do: [line: 1, column: 28], end: [line: 3, column: 20], line: 1, column: 20],
 [{:foo, [line: 1, column: 24], nil}, [do: :ok]]}

Expected behavior

I'm not sure which of the behaviors is the intended one, but since this isn't documented in the 1.16 release changelog I think the expected behavior is the one prior to that version: that :column sets the starting column and then get reset, instead of behaving as an indentation value.

[josevalim]

I will do some digging on the root cause here. It may have been an intentional change as I can see both approaches being useful. I assume you need the original approach? Because you are parsing regular code, just starting later?

[doorgan]

I assume you need the original approach? Because you are parsing regular code, just starting later?

Yep, although I can work with any of the approaches if I know which one is the intended one.

The original approach is the one I need, but I can achieve the same result in >=1.16 by padding the string start with empty spaces:

string = String.duplicate(" ", opts[:column] - 1) <> string
Code.string_to_quoted(string, line: opts[:line], columns: true)

The use case for this is extracting a substring from the original source code, and parse it while retaining positional metadata form the original code

[josevalim]

Confirm this is a bug that we need to address.