[gyb] Python 2 or 3 compatible Generate Your Boilerplate

[RLovelett]

EDIT: I've edited it since I first submitted this pull-request. I've added all the other patches I found necessary to get the gyb script to run on either Python 2 or 3 (as opposed to the single patch I submitted in the first place). As result this comment has been reformatted. My goal was to write fairly explicit commit messages that explain what each commit does on its own. They may be worth reviewing to figure out why each change was made.

[RLovelett]

I've gone ahead and put all of my Python 2/3 code on this branch. At this point I am able to compile, with these patches, using either Python 2.7.11 (OS X) or 3.5.1 (Arch Linux). 🎉

I've deleted my gyb-python-3-wip branch because it no longer is a work in progress. If you want me to split up these commits into multiple pulls just let me know.

[RLovelett]

Hopefully this should help solve some of the issues that were left by #83 and #90.

[frewsxcv]

You should make this:

for line in (line_bytes.decode('utf-8') for line_bytes in f):

so that it utilizes generators instead of building up a list unnecessarily

https://www.python.org/dev/peps/pep-0289/

[RLovelett]

That was a good suggestion. I've never written Python before and I hadn't noticed the difference between the parenthesis and square brackets.

Though in the end I actually found there is a codecs module that does what I am looking for explicitly and I've refactored the code to make use of that instead.

[dabrahams]

The uses of partial don’t read very well. Would we be better off with a lambda?

[frewsxcv]

How about:

sourceLinesIter = iter(sourceLines)
try:
    for tokenKind, tokenText, tokenStart, (tokenEndLine, tokenEndCol), lineText \
        in tokenize.generate_tokens(next(sourceLinesIter)):

[dabrahams]

LGTM

[RLovelett]

That would be fine except the result of next(sourceLinesIter) is not callable. Taken from tokenize.generate_tokens docs

... requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects (see section File Objects) ...

Notice that in the original code next was sent to tokenize.generate_tokens not next(), i.e., something that is callable. @frewsxcv suggestion is the equivalent of sending next() and not next. Hence why I went with the partial function application.

Testing

I first applied this patch:

diff --git a/utils/gyb.py b/utils/gyb.py
index ab422f1..68a8280 100755
--- a/utils/gyb.py
+++ b/utils/gyb.py
@@ -307,9 +307,10 @@ def splitGybLines(sourceLines):
     unmatchedIndents = []

     dedents = 0
+    sourceLinesIter = iter(sourceLines)
     try:
         for tokenKind, tokenText, tokenStart, (tokenEndLine, tokenEndCol), lineText \
-            in tokenize.generate_tokens(partial(next, sourceLines.__iter__())):
+            in tokenize.generate_tokens(next(sourceLinesIter)):

             if tokenKind in (tokenize.COMMENT, tokenize.ENDMARKER):
                 continue

Then I ran this from the utils directory.

$ python3
>>> from gyb import *
>>> src = splitLines('''\
... if x:
...     print x
... if y: # trailing comment
...     print z
...     if z: # another comment\
... ''')
>>> splitGybLines(src)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/Users/lovelett/Source/swift/utils/gyb.py", line 313, in splitGybLines
  in tokenize.generate_tokens(next(sourceLinesIter)):
File "/usr/local/Cellar/python3/3.5.1/Frameworks/Python.framework/Versions/3.5/lib/python3.5/tokenize.py", line 514, in _tokenize
  line = readline()
TypeError: 'str' object is not callable

Expected:

$ python3
>>> from gyb import *
>>> src = splitLines('''\
... if x:
...     print x
... if y: # trailing comment
...     print z
...     if z: # another comment\
... ''')
>>> splitGybLines(src)
[3, 5]

[frewsxcv]

Blah, you're right. Forgot that generate_tokens wants a callable. You might be able to get by with:

tokenize.generate_tokens(sourceLinesIter.next)

...since next is a method on an iterator, though I'm not entirely sure. Otherwise, I guess just stick with a lambda or partial

[RLovelett]

@frewsxcv unfortunately next only exists on an iterator in Python 2. PEP 3114 renamed the next function to __next__ in Python 3 and got rid of next entirely. Or put another way, it works as expected in Python 2 and fails with the runtime error AttributeError: 'list_iterator' object has no attribute 'next' if you run the same code in Python 3.

Like I said before somewhere else, this is the first Python code I've ever written or worked with. I say that to say please forgive my ignorance when asking what may be a stupid question: what is a lambda? Is that some sort of Python keyword?

[frewsxcv]

lambda is a keyword to create an anonymous function in Python. Here's a short demo:

>>> some_string = "hello!"
>>> some_string
'hello!'
>>> some_function = lambda: some_string
>>> some_function()
'hello!'
>>> another_function = lambda s: some_string + s
>>> another_function(" world!")
'hello! world!'

So in this particular case, you'd probably want to do something like this:

tokenize.generate_tokens(lambda: next(sourceLinesIter))

[RLovelett]

I've gone ahead and switched over to the lambda syntax (see: dc70d6d06119b72f3733da56fccb3c2d66082d8a). As a newbie to Python this feels like a lateral move but I defer to those who have been there before me.

@frewsxcv thanks for showing me some Python! 🍻

[RLovelett]

@dabrahams I've updated (7700b3b) per your comments.

[dabrahams]

@RLovelett Thanks!