Change OUTDENT tokens to be positioned at the end of the previous token

This commit adds another post-processing step after normal lexing that sets the
locationData on all OUTDENT tokens to be at the last character of the previous
token. This does feel like a little bit of a hack. Ideally the location data
would be set correctly in the first place and not in a post-processing step, but
I tried that and some temporary intermediate tokens were causing problems, so I
decided to set the location data once those intermediate tokens were removed.
Also, having this as a separate processing step makes it more robust and
isolated.

This fixes the problem in decaffeinate/decaffeinate#371 .
In that issue, the CoffeeScript tokens had three OUTDENT tokens in a row, and
the last two overlapped with the `]`. Since at least one of those OUTDENT tokens
was considered part of the function body, the function expression had an ending
position just after the end of the `]`.

OUTDENT tokens are sort of a weird case in the lexer anyway, since they often
don't correspond to an actual location in the source code. It seems like the
code in `lexer.coffee` makes an attempt at finding a good place for them, but in
some cases, it has a bad result. This seems hard to avoid in the general case.
For example, in this code:
```coffee
[->
  a]
```
There must be an OUTDENT between the `a` and the `]`, but CoffeeScript tokens
have an inclusive start and end, so they must always be at least one character
wide (I think). In this case, the lexer was choosing the `]` as the location,
and the parser ended up generating correct location data, I believe because
it ignores the outermost INDENT and OUTDENT tokens. However, with multiple
OUTDENT tokens in a row, the parser ends up producing location data that is
wrong.

It seems to me like there isn't a solid answer to "what location do OUTDENT
tokens have", since it hasn't mattered much, but for this commit, I'm defining
it: they always have the location of the last character of the previous token.
This should hopefully be fairly safe because tokens are still in the same order
relative to each other. Also, it's worth noting that this makes the start
location for OUTDENT tokens awkward. However, OUTDENT tokens are always used to
mark the end of something, so their `last_line` and `last_column` values are
always what matter when determining AST node bounds, so it is most important for
those to be correct.

Author: alangpierce

lib/coffee-script/rewriter.js

@@ -33,6 +33,7 @@ class exports.Rewriter
     @tagPostfixConditionals()
     @addImplicitBracesAndParens()
     @addLocationDataToGeneratedTokens()
+    @fixOutdentLocationData()
     @tokens
 
   # Rewrite the token stream, looking one token ahead and behind.
@@ -368,6 +369,20 @@ class exports.Rewriter
         last_column:  column
       return 1
 
+  # OUTDENT tokens should always be positioned at the last character of the
+  # previous token, so that AST nodes ending in an OUTDENT token end up with a
+  # location corresponding to the last "real" token under the node.
+  fixOutdentLocationData: ->
+    @scanTokens (token, i, tokens) ->
+      return 1 unless token[0] is 'OUTDENT'
+      prevLocationData = tokens[i - 1][2]
+      token[2] =
+        first_line:   prevLocationData.last_line
+        first_column: prevLocationData.last_column
+        last_line:    prevLocationData.last_line
+        last_column:  prevLocationData.last_column
+      return 1
+
   # Because our grammar is LALR(1), it can't handle some single-line
   # expressions that lack ending delimiters. The **Rewriter** adds the implicit
   # blocks, so it doesn't need to. To keep the grammar clean and tidy, trailing

src/rewriter.coffee

@@ -469,6 +469,24 @@ test "Verify tokens have locations that are in order", ->
         ok token[2].first_column >= lastToken[2].last_column
     lastToken = token
 
+test "Verify OUTDENT tokens are located at the end of the previous token", ->
+  source = '''
+    SomeArr = [ ->
+      if something
+        lol =
+          count: 500
+    ]
+  '''
+  tokens = CoffeeScript.tokens source
+  [..., number, curly, outdent1, outdent2, outdent3, bracket, terminator] = tokens
+  eq number[0], 'NUMBER'
+  for outdent in [outdent1, outdent2, outdent3]
+    eq outdent[0], 'OUTDENT'
+    eq outdent[2].first_line, number[2].last_line
+    eq outdent[2].first_column, number[2].last_column
+    eq outdent[2].last_line, number[2].last_line
+    eq outdent[2].last_column, number[2].last_column
+
 test "Verify all tokens get a location", ->
   doesNotThrow ->
     tokens = CoffeeScript.tokens testScript