__path__ missing in ModuleType for python 3

[sk-]

In python 2, ModuleType declares to have an optional __path__:

typeshed/stdlib/2/types.pyi

Line 131 in 3d14016

__path__: Optional[Iterable[str]]

However in Python 3, such field is missing:

typeshed/stdlib/3/_importlib_modulespec.pyi

Lines 35 to 42 in 3d14016

	class ModuleType:
	__name__: str
	__file__: str
	__dict__: Dict[str, Any]
	__loader__: Optional[_Loader]
	__package__: Optional[str]
	__spec__: Optional[ModuleSpec]
	def __init__(self, name: str, doc: Optional[str] = ...) -> None: ...

[hauntsaninja]

This wouldn't be technically true, e.g. sys.__path__... You may also be interested in python/mypy#9454

[sk-]

You are right, and I had find about it earlier, but forgot to post.

It may be harder to fix as it seems that the field is not always set and it can be missing.

>>> import importlib
>>> importlib.import_module('xml').__path__
['/usr/local/Cellar/python/3.7.7/Frameworks/Python.framework/Versions/3.7/lib/python3.7/xml']
>>> importlib.import_module('string').__path__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: module 'string' has no attribute '__path__'

Note that import_module returns a ModuleType.

The actual use case where I came up with this was when discovering plugins, where you need to use something like:

for module_info in pkgutil.iter_modules(package_path, prefix=f"{qualified_name}."):

and I ended up doing something like:

package_path = getattr(package, "__path__", None)
if not package_path:
   return

[hauntsaninja]

Yup, __path__ only exists on packages. I'm going to go ahead and close this issue then :-)

[sk-]

Could you clarify then why importlib.import_module returns a ModuleType and not a Union[PackageType, ModuleType] and why does python 2' ModuleType have a __path__ attribute. The examples I posted above also apply to Python 2.

At the very least I'd expect a comment on ModuleType explaining the difference between packages and modules, and how to deal with the type limitations.

[hauntsaninja]

I don't think Python really has a concept of a PackageType, whereas ModuleType is a thing that exists at runtime (and so is easy to discriminate). If you returned a Union here, where one of the members of the Union doesn't exist at runtime (but is a subclass of the other), it'd be basically impossible to discriminate and get type checking to work.

https://docs.python.org/3/library/types.html#types.ModuleType
https://docs.python.org/3/reference/import.html

>>> type(xml).__mro__
(<class 'module'>, <class 'object'>)
>>> isinstance(xml, types.ModuleType)
True

To be honest, I have the good fortune to able to forget how Python 2 works, but I suspect those stubs are incorrect.

~ λ python2
Python 2.7.16 (default, Jun  5 2020, 22:59:21) 
[GCC 4.2.1 Compatible Apple LLVM 11.0.3 (clang-1103.0.29.20) (-macos10.15-objc- on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import string
>>> string.__path__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__path__'

[sk-]

Ok, so ModuleType sets some fields like __path__ in specific code paths. I think this is the relevant code:

https://github.com/python/cpython/blob/622307142130d36a30644233441333247838af38/Python/import.c#L1170

Is this pattern pervasive (or at least somewhat common) in the stdlib? If so, we may need to extend the typing system to support something like type/interface optionals in Typescript where you can write:

interface X {
  always_set: int;
  sometimes_not_set?: int; // note the ?
}

Of course in Python the semantics would need to be different, because it raises AttributeError if the attribute is undefined, whereas in TS, it just returns undefined.