importlib.readers not valid until python 3.10 #9609

kdelee · 2022-02-04T21:07:43Z

This exists https://github.com/python/cpython/blob/3.10/Lib/importlib/readers.py and FileReader is in there
This is a 404 https://github.com/python/cpython/blob/3.9/Lib/importlib/readers.py

This change needs to get backported to the 7.0.z branch(s) too
closes #9608

I'm going to go ahead and open this PR, if you think we need a changelog for this LMK

bluetech · 2022-02-04T21:17:00Z

Thanks, this LGTM. Since I have no idea about the underlying importlib stuff, I'll leave it to someone else to actually approve, maybe @gaborbernat who added it (#9173), or @asottile who knows about this stuff.

I do notice that latest python says this about importlib.readers:

https://github.com/python/cpython/blob/bf95ff91f2c1fc5a57190491f9ccdc63458b089e/Lib/importlib/readers.py#L2-L5

Maybe we can be forward-compatible with Python 3.11 already by importing from importlib.resources.readers instead on python>=3.11.

bluetech · 2022-02-04T21:18:39Z

And if someone knows how to test this that would be great as well (though not necessary for this PR).

kdelee · 2022-02-04T21:27:18Z

Not surgical, but this test has my reproducer in it. If desired I can push this, or feel free to make edits to the PR

diff --git a/testing/test_compat.py b/testing/test_compat.py
index 88f0f33ef..e5bc5fa64 100644
--- a/testing/test_compat.py
+++ b/testing/test_compat.py
@@ -261,3 +261,16 @@ def test_assert_never_literal() -> None:
         pass
     else:
         assert_never(x)
+
+def test_pytest_get_resource_reader_works_with_pip(pytester: Pytester) -> None:
+    """Assert we dont regress on https://github.com/pytest-dev/pytest/issues/9608"""
+    pytester.makepyfile(
+        """
+        from pip._internal.req import parse_requirements
+
+        def test_import_works():
+            assert True
+    """
+    )
+    result = pytester.runpytest()
+    result.stdout.fnmatch_lines(["*1 passed*"])

nicoddemus · 2022-02-04T21:37:09Z

src/_pytest/assertion/rewrite.py

@@ -273,7 +273,7 @@ def get_data(self, pathname: Union[str, bytes]) -> bytes:
        with open(pathname, "rb") as f:
            return f.read()

-    if sys.version_info >= (3, 9):
+    if sys.version_info >= (3, 10):

        def get_resource_reader(self, name: str) -> importlib.abc.TraversableResources:  # type: ignore
            from types import SimpleNamespace


How about we move these two imports outside the function? This would at least caught the error in the correct Python versions.

if sys.version_info >= (3, 10): from types import SimpleNamespace from importlib.readers import FileReader def get_resource_reader(self, name: str) -> importlib.abc.TraversableResources: # type: ignore return FileReader(SimpleNamespace(path=self._rewritten_names[name]))

Hmm nevermind that, that conditional import does not work.

nicoddemus · 2022-02-04T21:54:16Z

Took the liberty of moving the import to the global level, and adding a CHANGELOG so this is "ready to go", but still would like to hear from someone who knows "importlib" stuff better.

changelog/9608.bugfix.rst

src/_pytest/assertion/rewrite.py

The-Compiler · 2022-02-05T13:37:53Z

I'm a bit confused by two things here:

Why didn't the test added in Support the importlib.resources files API in rewritten files #9173 by @gaborbernat catch this?
Why does the same test not fail now on Python 3.9, given that the code added there is now disabled there, but the test is unaltered?

asottile

the test in the original review passes in 3.9 because the implementation of importlib.resources doesn't attempt get_resource_reader and just utilizes the .submodule_search_locations of the module's spec

so the (3, 9) vs. (3, 10) difference is ~mostly harmless -- is there a place where you're seeing this break?

src/_pytest/assertion/rewrite.py

asottile · 2022-02-05T13:44:45Z

src/_pytest/assertion/rewrite.py

@@ -273,13 +277,10 @@ def get_data(self, pathname: Union[str, bytes]) -> bytes:
        with open(pathname, "rb") as f:
            return f.read()

-    if sys.version_info >= (3, 9):
+    if sys.version_info >= (3, 10):


I agree that this should be changed to (3, 10) since FileReader isn't needed there

kdelee · 2022-02-05T21:58:45Z

Not sure about the others reporting #9608, but this specifically occurs for us in a test file that imports from pip._internal.req import parse_requirements.

from pip._internal.req import parse_requirements

def test_broken():
    assert True

In the full traceback shown in #9608 we see it hitting this in https://github.com/pypa/pip/blob/44018de50cafba25445a225c1a1986d6312e1ef3/src/pip/_vendor/certifi/core.py#L50 which calls this path function in importlib.resources which ends up calling https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L161 which then checks if there is a attribute get_resource_reader on the "spec.loader" https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L74 -- which it looks like AssertionRewritingHook is a subclass of importlib.abc.Loader. This class are defines get_resource_loader on on the importlib.abc.Loader subclass and way down in importlib goes ahead and tries to call that function: https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L76.
This get_resource_loader defined on AssertionRewritingHook then tries to import code that doesn't exist in python 3.9 and everything breaks.

asottile · 2022-02-05T22:29:37Z

Not sure about the others reporting #9608, but this specifically occurs for us in a test file that imports from pip._internal.req import parse_requirements.
from pip._internal.req import parse_requirements

def test_broken():
    assert True
In the full traceback shown in #9608 we see it hitting this in https://github.com/pypa/pip/blob/44018de50cafba25445a225c1a1986d6312e1ef3/src/pip/_vendor/certifi/core.py#L50 which calls this path function in importlib.resources which ends up calling https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L161 which then checks if there is a attribute get_resource_reader on the "spec.loader" https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L74 -- which it looks like AssertionRewritingHook is a subclass of importlib.abc.Loader. This class are defines get_resource_loader on on the importlib.abc.Loader subclass and way down in importlib goes ahead and tries to call that function: https://github.com/python/cpython/blob/459e26f0987a12a19238baba422e13a8f7fcfca3/Lib/importlib/resources.py#L76. This get_resource_loader defined on AssertionRewritingHook then tries to import code that doesn't exist in python 3.9 and everything breaks.

ah yeah this was the whole 3.9.3 fiasco where the resources api was partially rewritten and reverted in a patch version but some of the apis use the readers already

I think if you take the test that was in the previous PR and use open_binary or any of the other resource apis that are not files() then it would demonstrate the failure -- that said, they wouldn't be supported pre-3.9 anyway due the missing reader helpers

so I guess to move forward, I would inline the conditional import and personally I wouldn't make it version-specific but that's up to you

there isn't really a way to test this "regression" because the feature itself doesn't exist in previous versions

kdelee · 2022-02-07T14:49:33Z

@nicoddemus @bluetech so what would ya'll like to do. It sounds like @asottile would like us to revert the move of the import outside the function.

bluetech · 2022-02-07T14:55:43Z

@kdelee Can you move the imports back into the function? Then I think we can merge.

kdelee · 2022-02-07T16:39:57Z

@bluetech done, lmk if anything else is needed

asottile

kdelee · 2022-02-07T17:34:23Z

😬 well not sure what those tests are blowing up about
Tried running them locally and I'm not seeing similar results

bluetech · 2022-02-07T17:39:14Z

The failures are not due to this PR, it suddenly started to happen, we're not sure why yet. We will rerun the tests once this is resolved.

This exists https://github.com/python/cpython/blob/3.10/Lib/importlib/readers.py and FileReader is in there This is a 404 https://github.com/python/cpython/blob/3.9/Lib/importlib/readers.py This change needs to get backported to the 7.0.z branch(s) too Fixes pytest-dev#9608

Fix pytest-dev#9608

Co-authored-by: Elijah DeLee <[email protected]>

Co-authored-by: Ran Benita <[email protected]>

@asottile

re: review from @asottile that this should only get imported in the function modify the else/if logic since inside the function we already know the python version is >= 3.10, and just have to know if it is 3.11 or greater

for more information, see https://pre-commit.ci

bluetech · 2022-02-08T13:58:52Z

(Should have squashed instead of merged - oops)

pombredanne · 2022-02-09T14:41:21Z

Thanks... 🙇 Do you mind to push a dot release and may be yank 7.0.0 on PyPI as this is making CI fails all over? That would be awesome!

nicoddemus · 2022-02-09T15:16:54Z

#9654 is up, if no other maintainer would like to include another fix, we can release it. (But doesn't make sense to yank 7.0.0 if we are releasing 7.0.1 right away).

pombredanne · 2022-02-09T15:23:01Z

You rock 🎸

matthiasmoke · 2022-05-27T13:35:53Z

src/_pytest/assertion/rewrite.py

@@ -273,13 +273,15 @@ def get_data(self, pathname: Union[str, bytes]) -> bytes:
        with open(pathname, "rb") as f:
            return f.read()

-    if sys.version_info >= (3, 9):
+    if sys.version_info >= (3, 10):

        def get_resource_reader(self, name: str) -> importlib.abc.TraversableResources:  # type: ignore


Hey, I'm currently writing a code analysis tool for Python projects. It pointed me to this code part, which I find quite interesting. Hence, I would have some questions about it:

Where exactly is the get_resource_reader() function called? I can't find any references to it.

Why is the function defined by a condition? Would it make sense to move the version check inside the function with an early return? (As there is already a version check within it) As the function is public, it would keep the interface of the AssertionRewritingHook independent to the Python version.

I think these questions may be adressed to @kdelee or @nicoddemus? I would be really happy to hear from you as this would give me some qualitative insights on the reports of my tool.

Kind regards,
Matthias

it's called by the python import machinery itself. this function isn't valid until 3.10 (it breaks 3.9)

I saw the Github issue on the FileReader import causing the break for 3.9

Before, the import statement in the function was without any version check. But having the check sys.version_info >= (3, 10) within the function with an early return shouldn't break anything as the import statement would never be reached for 3.9.
Or am I overlooking anything why the function would be invalid otherwise?

there's backports of parts of importlib which would reach into this and if the function is defined try and call it (and then error) -- by not defining the function, those backports won't even attempt to call it and therefore won't crash

Ahh ok I understand. Thanky you very much for your explanation, this helps me a lot!

bluetech added the needs backport applied to PRs, indicates that it should be ported to the current bug-fix branch label Feb 4, 2022

nicoddemus reviewed Feb 4, 2022

View reviewed changes

kdelee commented Feb 4, 2022

View reviewed changes

changelog/9608.bugfix.rst Outdated Show resolved Hide resolved

bluetech reviewed Feb 4, 2022

View reviewed changes

src/_pytest/assertion/rewrite.py Outdated Show resolved Hide resolved

kdelee mentioned this pull request Feb 4, 2022

Consume control capacity ansible/awx#11665

Merged

asottile requested changes Feb 5, 2022

View reviewed changes

asottile approved these changes Feb 7, 2022

View reviewed changes

kdelee and others added 6 commits February 8, 2022 00:44

Fix invalid importing of importlib.reader in Python 3.9

5f1a2f3

Fix pytest-dev#9608

Update changelog/9608.bugfix.rst

66dc79e

Co-authored-by: Elijah DeLee <[email protected]>

Update src/_pytest/assertion/rewrite.py

f86a87a

Co-authored-by: Ran Benita <[email protected]>

move function specific imports to function

d45a19c

re: review from @asottile that this should only get imported in the function modify the else/if logic since inside the function we already know the python version is >= 3.10, and just have to know if it is 3.11 or greater

[pre-commit.ci] auto fixes from pre-commit.com hooks

2073cce

for more information, see https://pre-commit.ci

bluetech force-pushed the patch-1 branch from 3235fdb to 2073cce Compare February 7, 2022 22:45

stephankn mentioned this pull request Feb 8, 2022

test fails on pytest with ModuleNotFoundError: No module named 'importlib.readers' osm-fr/osmose-backend#1400

Closed

bluetech merged commit 44290d1 into pytest-dev:main Feb 8, 2022

bluetech added the backport 7.0.x label Feb 8, 2022

github-actions bot mentioned this pull request Feb 8, 2022

[7.0.x] importlib.readers not valid until python 3.10 #9647

Merged

DanLipsitt mentioned this pull request Feb 10, 2022

Add schemathesis tests awtkns/fastapi-crudrouter#141

Open

matthiasmoke reviewed May 27, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

importlib.readers not valid until python 3.10 #9609

importlib.readers not valid until python 3.10 #9609

kdelee commented Feb 4, 2022

bluetech commented Feb 4, 2022

bluetech commented Feb 4, 2022

kdelee commented Feb 4, 2022 •

edited

Loading

nicoddemus Feb 4, 2022

nicoddemus Feb 4, 2022 •

edited

Loading

nicoddemus commented Feb 4, 2022

The-Compiler commented Feb 5, 2022

asottile left a comment

asottile Feb 5, 2022

kdelee commented Feb 5, 2022 •

edited

Loading

asottile commented Feb 5, 2022

kdelee commented Feb 7, 2022

bluetech commented Feb 7, 2022

kdelee commented Feb 7, 2022

asottile left a comment

kdelee commented Feb 7, 2022

bluetech commented Feb 7, 2022

bluetech commented Feb 8, 2022

pombredanne commented Feb 9, 2022

nicoddemus commented Feb 9, 2022 •

edited

Loading

pombredanne commented Feb 9, 2022

matthiasmoke May 27, 2022

asottile May 27, 2022

matthiasmoke May 27, 2022

asottile May 27, 2022

matthiasmoke May 29, 2022

importlib.readers not valid until python 3.10 #9609

importlib.readers not valid until python 3.10 #9609

Conversation

kdelee commented Feb 4, 2022

bluetech commented Feb 4, 2022

bluetech commented Feb 4, 2022

kdelee commented Feb 4, 2022 • edited Loading

nicoddemus Feb 4, 2022

Choose a reason for hiding this comment

nicoddemus Feb 4, 2022 • edited Loading

Choose a reason for hiding this comment

nicoddemus commented Feb 4, 2022

The-Compiler commented Feb 5, 2022

asottile left a comment

Choose a reason for hiding this comment

asottile Feb 5, 2022

Choose a reason for hiding this comment

kdelee commented Feb 5, 2022 • edited Loading

asottile commented Feb 5, 2022

kdelee commented Feb 7, 2022

bluetech commented Feb 7, 2022

kdelee commented Feb 7, 2022

asottile left a comment

Choose a reason for hiding this comment

kdelee commented Feb 7, 2022

bluetech commented Feb 7, 2022

bluetech commented Feb 8, 2022

pombredanne commented Feb 9, 2022

nicoddemus commented Feb 9, 2022 • edited Loading

pombredanne commented Feb 9, 2022

matthiasmoke May 27, 2022

Choose a reason for hiding this comment

asottile May 27, 2022

Choose a reason for hiding this comment

matthiasmoke May 27, 2022

Choose a reason for hiding this comment

asottile May 27, 2022

Choose a reason for hiding this comment

matthiasmoke May 29, 2022

Choose a reason for hiding this comment

kdelee commented Feb 4, 2022 •

edited

Loading

nicoddemus Feb 4, 2022 •

edited

Loading

kdelee commented Feb 5, 2022 •

edited

Loading

nicoddemus commented Feb 9, 2022 •

edited

Loading