Adding Python 3 support #10

asmeurer · 2019-04-05T20:50:30Z

Fixes #2

This is the same as #9, but reopened so that Travis works and it hides the commits that are already in master.

TODOs:

Fix all test and doctest failures in Python 3
Fix all warnings from Python 3
Make sure all import files are updated for Python 3 and this is tested
Make sure all files have the proper __future__ imports.
Update py command line examples
Check for any other additional ast node types in Python 3
Check for bugs on some Python 3 codebases

Most of the changes were unnecessary, such as converting iterators into lists, or adding a dependency on futurize.

Most of the unchosen changes were the unnecessary addition of futurize as a dependency.

It's needed for Python 2/3 compatibility, but isn't included with six.

The exceptions are __init__.py, because __all__ is set to [], and in _py.py, launch_ipython_with_autoimporter is undefined, but I do not know what it should actually be (no such function is defined anywhere in pyflyby).

With the exception of those in test_autoimp.py, which are undefined names intentionally. If it was desired to make pyflakes pass 100% on all test files those functions could be created using exec().

This breaks pyflakes testing because there are variables that appear to be unused. When Python 2 support is dropped, these strings can become f-strings.

The code can only run in Python 2, so revert the incorrect change from futurize. This fixes 'py console' in Python 2.

This reverts some futurize changes. The tests are only relevant in Python 2, because they deal with oldstyle classes.

This reverts an incorrect futurize change, and makes it so it should work properly in both Python 2 and 3.

asmeurer · 2019-04-05T20:58:29Z

I guess Travis doesn't run on draft pull requests? That would make the feature rather useless.

This allows the sorted() calls in autoimp to work in Python 3.

…thon 3

asmeurer · 2019-04-05T22:49:14Z

The b64decode examples in the auto_eval doctest aren't very good, because it returns bytes in Python 3, producing different output (b'hello' vs. hello). Should I add .decode('utf-8') to the examples, or pick a better function that acts the same in both 2 and 3?

quarl · 2019-04-05T23:51:44Z

Good point. Let’s pick examples that work the same in py2/py3. Karl On Apr 5, 2019, at 18:49, Aaron Meurer <[email protected]<mailto:[email protected]>> wrote: The b64decode examples in the auto_eval doctest aren't very good, because it returns bytes in Python 3, producing different output (b'hello' vs. hello). Should I add .decode('utf-8') to the examples, or pick a better function that acts the same in both 2 and 3? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#10 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AADDZ4pNxfqpthw-gctPAUNWcopCGMzzks5vd9LqgaJpZM4cftCa>.

asmeurer · 2019-05-17T23:10:02Z

I've gone through all the places in https://greentreesnakes.readthedocs.io/en/latest/nodes.html where Python 2/3 differences are noted and added tests for those node types. I still need to find some good codebases that use Python 3 features to run some smoke tests on. Although I already know of at least one error in the parser, also present in master from the pyflyby sources themself #12.

This reverts commit 9bd9d73.

The file is only used in Python 2 and doesn't have prints. The extra line changes the docxref test.

It may be why tests are randomly failing on Travis again.

asmeurer · 2019-05-18T00:24:36Z

I guess this is ready for review. I'll try to think of some good Python 3 codebases I can test it against this weekend. Let me know if you have any suggestions. I'm not aware of any outstanding issues, except for the mandatory imports question, and the upstream fixes.

asmeurer · 2019-05-18T00:27:02Z

I guess I forgot to mention here that there is another relevant upstream issue, ipython/ipython#11714. In the newest IPython, the input transformers are called multiple times. This results in the pyflyby messages being printed multiple times. The workaround for this isn't exactly pretty. I don't know how to fix the issue upstream, so we'll have to wait on the IPython guys to do something about it.

scopatz · 2019-05-20T19:59:00Z

lib/python/pyflyby/_autoimp.py

@@ -24,6 +25,8 @@
 from   pyflyby._modules         import ModuleHandle
 from   pyflyby._parse           import PythonBlock, infer_compile_mode

+if PY2:
+    long = int


This seems less than ideal. Why not just change the lone long call to int?

I'm sure I meant to do this differently, but since it apparently works as an int rather than a long, I'll just remove it.

scopatz · 2019-05-20T20:02:52Z

lib/python/pyflyby/_flags.py

@@ -4,6 +4,7 @@

 from __future__ import absolute_import, division, with_statement

+from __future__ import print_function


This should be combined with the above import. I don't recall if it works to have multiple time machine statements because it is handled in a special way.

scopatz · 2019-05-20T20:06:09Z

lib/python/pyflyby/_interactive.py

+                        stack[3].function == 'run_cell_async',
+                        # stack[3].function == 'should_run_async',
+                        # stack[1].function == 'check_complete'
+                ]):


isn't this equivalent to just if stack[3].function == 'run_cell_async':

Yea, I wanted to leave the other ones in there as comments, in case this isn't actually correct (as far as I can tell from the tests, it should be). I don't fully understand where the three different places are called. I originally thought the inverse of this was correct, but apparently it isn't.

I think it is a little confusing to keep this as it is now. Can you please add a comment saying why the others are commented out and when to uncomment them. Also, you probably don't need the False in there.

I think I added the False so that I could comment all three out (which is stupid because any([]) is the same as any([False])).

scopatz · 2019-05-20T20:13:09Z

Thanks @asmeurer - just a couple of small requests

It apparently doesn't matter if it is an int or a long.

setup.py collect_imports is currently broken because of deshaw#12, but I have added it to .pyflyby manually. I have not yet re-run tidy-imports on the codebase.

The tidy-imports added a line to the top of the file.

scopatz · 2019-05-21T19:45:30Z

@quarl - I think this is good to go now. Should I merge it in?

quarl · 2019-05-21T19:46:43Z

Great! Yes please.

scopatz · 2019-05-21T20:17:20Z

🎉

asmeurer added 20 commits March 21, 2019 15:24

Run futurize -1 -w -n lib/python/pyflyby/

728c740

Run futurize -1 -w -n tests/

93d69a6

Selected fixes from futurize -2 -w -n lib/python/pyflyby/

b218b62

Most of the changes were unnecessary, such as converting iterators into lists, or adding a dependency on futurize.

Selected changes from futurize -2 -w -n tests/

0699e29

Most of the unchosen changes were the unnecessary addition of futurize as a dependency.

Add cmp to pyflyby._util

1463e73

It's needed for Python 2/3 compatibility, but isn't included with six.

Fix pyflakes errors in Python 3 in pyflyby

75a51c5

The exceptions are __init__.py, because __all__ is set to [], and in _py.py, launch_ipython_with_autoimporter is undefined, but I do not know what it should actually be (no such function is defined anywhere in pyflyby).

Fix pyflakes errors in tests/

d064534

With the exception of those in test_autoimp.py, which are undefined names intentionally. If it was desired to make pyflakes pass 100% on all test files those functions could be created using exec().

Merge branch 'tests-fix' into python3

d01ee26

Don't try to install epydoc in Python 2 on Travis

3b6adb5

Skip the docxref test in Python 3

b642ac8

Do not do automatic string formatting in ipython() in test_interactive

4bd09bc

This breaks pyflakes testing because there are variables that appear to be unused. When Python 2 support is dropped, these strings can become f-strings.

Fix docxref tests skipping logic

8381580

Run futurize -1 -w -n bin/*

d3e3f4e

Run futurize -2 -w -n bin/*

7065f9f

Fix code in util that deals with unbound methods

6f22176

The code can only run in Python 2, so revert the incorrect change from futurize. This fixes 'py console' in Python 2.

Revert an incorrect futurize change

b93f7ee

Fix the livepatch tests in Python 2

e84d088

This reverts some futurize changes. The tests are only relevant in Python 2, because they deal with oldstyle classes.

Fix test_py in Python 2.

27d0b0d

This reverts an incorrect futurize change, and makes it so it should work properly in both Python 2 and 3.

Fix the docxref tests to correctly skip in Python 3

2be2483

Merge branch 'master' into python3

44e6581

asmeurer marked this pull request as ready for review April 5, 2019 20:58

asmeurer added 6 commits April 5, 2019 16:00

Remove Python 3 from the allowed failures on Travis

a69e23b

Fix test_0testconfig for Python 3

7bafdbf

Define comparison methods on DottedIdentifier

d69e176

This allows the sorted() calls in autoimp to work in Python 3.

Fix several autoimp tests that were testing with invalid syntax in Py…

05e6629

…thon 3

Fix some autoimp tests for Python 3 (b64decode returns a bytes object)

11d79cd

Skip doctesting for _docxref.py in Python 3

4a31b5f

asmeurer added 7 commits May 17, 2019 16:57

Use consistent naming for the tests

37d403a

Add an autoimp test for an async comprehension

9b4cece

Add an autoimp test for async/await

0bbe8f4

Remove some accidentally repeated comments

0d1d203

Add an autoimp test for nested with statements

0ad7b1d

Add an autoimp test for yield from

98fcbaf

Make the async/await test test all the async/await node types

e4c9ca5

asmeurer added 4 commits May 17, 2019 17:28

Revert "Add __future__.print_function as a mandatory import"

5c7f65d

This reverts commit 9bd9d73.

Revert print_function addition to tests/xrefs.py

5bfd84a

The file is only used in Python 2 and doesn't have prints. The extra line changes the docxref test.

Test removing the rerun filter

7c2cb99

It may be why tests are randomly failing on Travis again.

Fix _py doctest

b00d688

scopatz reviewed May 20, 2019

View reviewed changes

asmeurer added 7 commits May 20, 2019 15:44

Merge two __future__ imports together

74b3771

Remove long call from Python 2 compatibility code

6499731

It apparently doesn't matter if it is an int or a long.

Add a clarifying comment

10fb5b2

Make __future__.print_function a mandatory import for pyflyby

02d2a7d

setup.py collect_imports is currently broken because of deshaw#12, but I have added it to .pyflyby manually. I have not yet re-run tidy-imports on the codebase.

Fix str/bytes issue

d0fc3a5

Run tidy-imports

7994507

Update the docxref test

5cf1cb1

The tidy-imports added a line to the top of the file.

scopatz merged commit e35ab41 into deshaw:master May 21, 2019

asmeurer added this to the FY 2019 - March 2020 milestone Jul 1, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Adding Python 3 support #10

Adding Python 3 support #10

asmeurer commented Apr 5, 2019 •

edited

Loading

asmeurer commented Apr 5, 2019

asmeurer commented Apr 5, 2019

quarl commented Apr 5, 2019 via email

asmeurer commented May 17, 2019

asmeurer commented May 18, 2019

asmeurer commented May 18, 2019

scopatz May 20, 2019

asmeurer May 20, 2019

scopatz May 20, 2019

scopatz May 20, 2019

asmeurer May 20, 2019

scopatz May 20, 2019

asmeurer May 20, 2019

scopatz commented May 20, 2019

scopatz commented May 21, 2019

quarl commented May 21, 2019

scopatz commented May 21, 2019

		@@ -4,6 +4,7 @@

		from __future__ import absolute_import, division, with_statement

		from __future__ import print_function

Adding Python 3 support #10

Adding Python 3 support #10

Conversation

asmeurer commented Apr 5, 2019 • edited Loading

asmeurer commented Apr 5, 2019

asmeurer commented Apr 5, 2019

quarl commented Apr 5, 2019 via email

asmeurer commented May 17, 2019

asmeurer commented May 18, 2019

asmeurer commented May 18, 2019

scopatz May 20, 2019

Choose a reason for hiding this comment

asmeurer May 20, 2019

Choose a reason for hiding this comment

scopatz May 20, 2019

Choose a reason for hiding this comment

scopatz May 20, 2019

Choose a reason for hiding this comment

asmeurer May 20, 2019

Choose a reason for hiding this comment

scopatz May 20, 2019

Choose a reason for hiding this comment

asmeurer May 20, 2019

Choose a reason for hiding this comment

scopatz commented May 20, 2019

scopatz commented May 21, 2019

quarl commented May 21, 2019

scopatz commented May 21, 2019

asmeurer commented Apr 5, 2019 •

edited

Loading