Miscalculated previous eol counts in , and ; eol tokens

[doorgan]

Elixir and Erlang/OTP versions

Erlang/OTP 26 [erts-14.2.2] [source] [64-bit] [smp:8:8] [ds:8:8:10] [async-threads:1] [jit]

Elixir 1.19.0-dev (3a2ce69) (compiled with Erlang/OTP 26)

Operating system

Any

Current behavior

There is a bug in the way the tokenizer counts end of lines after a , or a; that is noticeable when retrieving comments

Given this code string:

%{
  # 1. one

  # 1. two
  # 1. three
  one: one,
  one: after_one,
  one: after_one do
    :ok
  end,

  # 2. one

  # 2. two
  # 2. three
  # two,

  # 3. one

  # 3. two
  # 3. three
  three: three # final

  # 4. one

  # 4. two
  # 4. three
  # four
}

Running Code.string_to_quoted_with_comments!/1 on it produces this list of comments:

[
   %{line: 1, text: "# 1. one", column: 9, next_eol_count: 2, previous_eol_count: 1},
   %{line: 3, text: "# 1. two", column: 9, next_eol_count: 1, previous_eol_count: 2},
   %{line: 4, text: "# 1. three", column: 9, next_eol_count: 1, previous_eol_count: 1},
   %{line: 12, text: "# 2. one", column: 11, next_eol_count: 2, previous_eol_count: 2},
   %{line: 14, text: "# 2. two", column: 11, next_eol_count: 1, previous_eol_count: 4},
   %{line: 15, text: "# 2. three", column: 11, next_eol_count: 1, previous_eol_count: 5},
   %{line: 16, text: "# two,", column: 11, next_eol_count: 2, previous_eol_count: 6},
   %{line: 18, text: "# 3. one", column: 11, next_eol_count: 2, previous_eol_count: 8},
   %{line: 20, text: "# 3. two", column: 11, next_eol_count: 1, previous_eol_count: 10},
   %{line: 21, text: "# 3. three", column: 11, next_eol_count: 1, previous_eol_count: 11},
   %{line: 22, text: "# final", column: 17, next_eol_count: 2, previous_eol_count: 0},
   %{line: 24, text: "# 4. one", column: 11, next_eol_count: 2, previous_eol_count: 2},
   %{line: 26, text: "# 4. two", column: 11, next_eol_count: 1, previous_eol_count: 2},
   %{line: 27, text: "# 4. three", column: 11, next_eol_count: 1, previous_eol_count: 1},
   %{line: 28, text: "# four", column: 11, next_eol_count: 1, previous_eol_count: 1}
 ]

Notice that any comment after the end, gets a previous_eol_count higher than 2, when there is at most 2 newlines after any of the comments in the snippet.

Inspecting the eol tokens in the Code.previous_eol_count/1 private function I see this:

{:eol, {1, 7, 1}}
{:eol, {1, 7, 2}}
{:eol, {1, 7, 1}}
{:",", {10, 10, 2}}
{:",", {10, 10, 4}}
{:",", {10, 10, 5}}
{:",", {10, 10, 6}}
{:",", {10, 10, 8}}
{:",", {10, 10, 10}}
{:",", {10, 10, 11}}
{:eol, {22, 20, 2}}
{:eol, {22, 20, 2}}
{:eol, {22, 20, 1}}
{:eol, {22, 20, 1}}

Where the last element of the three tuple is the count of newlines before the token. Only the :"," tokens show the wrong numbers, and I can reproduce the issue with a simple block with a ; after each line

Expected behavior

EOL counts for the :"," and :";" tokens should match the eol count behavior of the :eol token.
In the above example, the following counts are expected:

{:eol, {1, 7, 1}}
{:eol, {1, 7, 2}}
{:eol, {1, 7, 1}}
{:",", {10, 10, 2}}
{:",", {10, 10, 2}}
{:",", {10, 10, 1}}
{:",", {10, 10, 1}}
{:",", {10, 10, 2}}
{:",", {10, 10, 2}}
{:",", {10, 10, 1}}
{:eol, {22, 20, 2}}
{:eol, {22, 20, 2}}
{:eol, {22, 20, 1}}
{:eol, {22, 20, 1}}

[josevalim]

After looking at this, it seems that we are not considering that a comment itselt does not reset the previous_eol_count. Although that is obviously a bug, because we are considering comments for next_eol_count.