Skip to content

Commit 8e28720

Browse files
hippo91cdce8p
hippo91
and
cdce8p
authored
Bug pylint 4206 (#921)
* Takes into account the fact that inferring subscript when the node is a class may use the __class_getitem__ method of the current class instead of looking for __getitem__ in the metaclass. * OrderedDict in the collections module inherit from dict which is C coded and thus have no metaclass but starting from python3.9 it supports subscripting thanks to the __class_getitem__ method. * check_metaclass becomes a static class method because we need it in the class scope. The brain_typing module does not add a ABCMeta_typing class thus there is no need to test it. Moreover it doesn't add neither a __getitem__ to the metaclass * The brain_typing module does not add anymore _typing suffixed classes in the mro * The OrderedDict class inherits from C coded dict class and thus doesn't have a metaclass. * When trying to inherit from typing.Pattern the REPL says : TypeError: type 're.Pattern' is not an acceptable base type * The REPL says that Derived as ABCMeta for metaclass and the mro is Derived => Iterator => Iterable => object * Adds comments * Starting with Python39 some collections of the collections.abc module support subscripting thanks to __class_getitem__ method. However the wat it is implemented is not straigthforward and instead of complexifying the way __class_getitem__ is handled inside the getitem method of the ClassDef class, we prefer to hack a bit. * Thanks to __class_getitem__ method there is no need to hack the metaclass * SImplifies the inference system for typing objects before python3.9. Before python3.9 the objects of the typing module that are alias of the same objects in the collections.abc module have subscripting possibility thanks to the _GenericAlias metaclass. To mock the subscripting capability we add __class_getitem__ method on those objects. * check_metaclass_is_abc become global to be shared among different classes * Create a test class dedicated to the Collections brain * Rewrites and adds test * Corrects syntax error * Deque, defaultdict and OrderedDict are part of the _collections module which is a pure C lib. While letting those class mocks inside collections module is fair for astroid it leds to pylint acceptance tests fail. * Formatting according to black * Adds two entries * Extends the filter to determine what is subscriptable to include OrderedDict * Formatting according to black * Takes into account the fact that inferring subscript when the node is a class may use the __class_getitem__ method of the current class instead of looking for __getitem__ in the metaclass. * OrderedDict in the collections module inherit from dict which is C coded and thus have no metaclass but starting from python3.9 it supports subscripting thanks to the __class_getitem__ method. * check_metaclass becomes a static class method because we need it in the class scope. The brain_typing module does not add a ABCMeta_typing class thus there is no need to test it. Moreover it doesn't add neither a __getitem__ to the metaclass * The brain_typing module does not add anymore _typing suffixed classes in the mro * The OrderedDict class inherits from C coded dict class and thus doesn't have a metaclass. * When trying to inherit from typing.Pattern the REPL says : TypeError: type 're.Pattern' is not an acceptable base type * The REPL says that Derived as ABCMeta for metaclass and the mro is Derived => Iterator => Iterable => object * Adds comments * Starting with Python39 some collections of the collections.abc module support subscripting thanks to __class_getitem__ method. However the wat it is implemented is not straigthforward and instead of complexifying the way __class_getitem__ is handled inside the getitem method of the ClassDef class, we prefer to hack a bit. * Thanks to __class_getitem__ method there is no need to hack the metaclass * SImplifies the inference system for typing objects before python3.9. Before python3.9 the objects of the typing module that are alias of the same objects in the collections.abc module have subscripting possibility thanks to the _GenericAlias metaclass. To mock the subscripting capability we add __class_getitem__ method on those objects. * check_metaclass_is_abc become global to be shared among different classes * Create a test class dedicated to the Collections brain * Rewrites and adds test * Corrects syntax error * Deque, defaultdict and OrderedDict are part of the _collections module which is a pure C lib. While letting those class mocks inside collections module is fair for astroid it leds to pylint acceptance tests fail. * Formatting according to black * Adds two entries * Extends the filter to determine what is subscriptable to include OrderedDict * Formatting according to black * Takes into account AWhetter remarks * Deactivates access to __class_getitem__ method * OrderedDict appears in typing module with python3.7.2 * _alias function in the typing module appears with python3.7 * Formatting according to black * _alias function is used also for builtins type and not only for collections.abc ones * Adds tests for both builtins type that are subscriptable and typing builtin alias type that are also subscriptable * No need to handle builtin types in this brain. It is better suited inside brain_bulitin_inference * Adds brain to handle builtin types that are subscriptable starting with python39 * Formatting according to black * Uses partial function instead of closure in order pylint acceptance to be ok * Handling the __class_getitem__ method associated to EmptyNode for builtin types is made directly inside the getitem method * infer_typing_alias has to be an inference_tip to avoid interferences between typing module and others (collections or builtin) * Formatting * Removes useless code * Adds comment * Takes into account cdce8p remarks * Formatting * Style changes Co-authored-by: Marc Mueller <[email protected]>
1 parent 6cc2c66 commit 8e28720

File tree

5 files changed

+413
-101
lines changed

5 files changed

+413
-101
lines changed

‎ChangeLog

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,11 @@ What's New in astroid 2.5.3?
1111
============================
1212
Release Date: TBA
1313

14+
* Takes into account the fact that subscript inferring for a ClassDef may involve __class_getitem__ method
15+
16+
* Reworks the `collections` and `typing` brain so that `pylint`s acceptance tests are fine.
17+
18+
Closes PyCQA/pylint#4206
1419

1520
What's New in astroid 2.5.2?
1621
============================

‎astroid/brain/brain_collections.py

Lines changed: 47 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,7 @@ def __rmul__(self, other): pass"""
6868
if PY39:
6969
base_deque_class += """
7070
@classmethod
71-
def __class_getitem__(self, item): pass"""
71+
def __class_getitem__(self, item): return cls"""
7272
return base_deque_class
7373

7474

@@ -77,7 +77,53 @@ def _ordered_dict_mock():
7777
class OrderedDict(dict):
7878
def __reversed__(self): return self[::-1]
7979
def move_to_end(self, key, last=False): pass"""
80+
if PY39:
81+
base_ordered_dict_class += """
82+
@classmethod
83+
def __class_getitem__(cls, item): return cls"""
8084
return base_ordered_dict_class
8185

8286

8387
astroid.register_module_extender(astroid.MANAGER, "collections", _collections_transform)
88+
89+
90+
def _looks_like_subscriptable(node: astroid.nodes.ClassDef) -> bool:
91+
"""
92+
Returns True if the node corresponds to a ClassDef of the Collections.abc module that
93+
supports subscripting
94+
95+
:param node: ClassDef node
96+
"""
97+
if node.qname().startswith("_collections") or node.qname().startswith(
98+
"collections"
99+
):
100+
try:
101+
node.getattr("__class_getitem__")
102+
return True
103+
except astroid.AttributeInferenceError:
104+
pass
105+
return False
106+
107+
108+
CLASS_GET_ITEM_TEMPLATE = """
109+
@classmethod
110+
def __class_getitem__(cls, item):
111+
return cls
112+
"""
113+
114+
115+
def easy_class_getitem_inference(node, context=None):
116+
# Here __class_getitem__ exists but is quite a mess to infer thus
117+
# put an easy inference tip
118+
func_to_add = astroid.extract_node(CLASS_GET_ITEM_TEMPLATE)
119+
node.locals["__class_getitem__"] = [func_to_add]
120+
121+
122+
if PY39:
123+
# Starting with Python39 some objects of the collection module are subscriptable
124+
# thanks to the __class_getitem__ method but the way it is implemented in
125+
# _collection_abc makes it difficult to infer. (We would have to handle AssignName inference in the
126+
# getitem method of the ClassDef class) Instead we put here a mock of the __class_getitem__ method
127+
astroid.MANAGER.register_transform(
128+
astroid.nodes.ClassDef, easy_class_getitem_inference, _looks_like_subscriptable
129+
)

‎astroid/brain/brain_typing.py

Lines changed: 80 additions & 59 deletions
Original file line numberDiff line numberDiff line change
@@ -8,7 +8,7 @@
88
"""Astroid hooks for typing.py support."""
99
import sys
1010
import typing
11-
from functools import lru_cache
11+
from functools import partial
1212

1313
from astroid import (
1414
MANAGER,
@@ -19,6 +19,7 @@
1919
nodes,
2020
context,
2121
InferenceError,
22+
AttributeInferenceError,
2223
)
2324
import astroid
2425

@@ -116,37 +117,12 @@ def infer_typedDict( # pylint: disable=invalid-name
116117
node.root().locals["TypedDict"] = [class_def]
117118

118119

119-
GET_ITEM_TEMPLATE = """
120+
CLASS_GETITEM_TEMPLATE = """
120121
@classmethod
121-
def __getitem__(cls, value):
122+
def __class_getitem__(cls, item):
122123
return cls
123124
"""
124125

125-
ABC_METACLASS_TEMPLATE = """
126-
from abc import ABCMeta
127-
ABCMeta
128-
"""
129-
130-
131-
@lru_cache()
132-
def create_typing_metaclass():
133-
#  Needs to mock the __getitem__ class method so that
134-
#  MutableSet[T] is acceptable
135-
func_to_add = extract_node(GET_ITEM_TEMPLATE)
136-
137-
abc_meta = next(extract_node(ABC_METACLASS_TEMPLATE).infer())
138-
typing_meta = nodes.ClassDef(
139-
name="ABCMeta_typing",
140-
lineno=abc_meta.lineno,
141-
col_offset=abc_meta.col_offset,
142-
parent=abc_meta.parent,
143-
)
144-
typing_meta.postinit(
145-
bases=[extract_node(ABC_METACLASS_TEMPLATE)], body=[], decorators=None
146-
)
147-
typing_meta.locals["__getitem__"] = [func_to_add]
148-
return typing_meta
149-
150126

151127
def _looks_like_typing_alias(node: nodes.Call) -> bool:
152128
"""
@@ -161,10 +137,43 @@ def _looks_like_typing_alias(node: nodes.Call) -> bool:
161137
isinstance(node, nodes.Call)
162138
and isinstance(node.func, nodes.Name)
163139
and node.func.name == "_alias"
164-
and isinstance(node.args[0], nodes.Attribute)
140+
and (
141+
# _alias function works also for builtins object such as list and dict
142+
isinstance(node.args[0], nodes.Attribute)
143+
or isinstance(node.args[0], nodes.Name)
144+
and node.args[0].name != "type"
145+
)
165146
)
166147

167148

149+
def _forbid_class_getitem_access(node: nodes.ClassDef) -> None:
150+
"""
151+
Disable the access to __class_getitem__ method for the node in parameters
152+
"""
153+
154+
def full_raiser(origin_func, attr, *args, **kwargs):
155+
"""
156+
Raises an AttributeInferenceError in case of access to __class_getitem__ method.
157+
Otherwise just call origin_func.
158+
"""
159+
if attr == "__class_getitem__":
160+
raise AttributeInferenceError("__class_getitem__ access is not allowed")
161+
else:
162+
return origin_func(attr, *args, **kwargs)
163+
164+
if not isinstance(node, nodes.ClassDef):
165+
raise TypeError("The parameter type should be ClassDef")
166+
try:
167+
node.getattr("__class_getitem__")
168+
# If we are here, then we are sure to modify object that do have __class_getitem__ method (which origin is one the
169+
# protocol defined in collections module) whereas the typing module consider it should not
170+
# We do not want __class_getitem__ to be found in the classdef
171+
partial_raiser = partial(full_raiser, node.getattr)
172+
node.getattr = partial_raiser
173+
except AttributeInferenceError:
174+
pass
175+
176+
168177
def infer_typing_alias(
169178
node: nodes.Call, ctx: context.InferenceContext = None
170179
) -> typing.Optional[node_classes.NodeNG]:
@@ -174,38 +183,48 @@ def infer_typing_alias(
174183
:param node: call node
175184
:param context: inference context
176185
"""
177-
if not isinstance(node, nodes.Call):
178-
return None
179186
res = next(node.args[0].infer(context=ctx))
180187

181188
if res != astroid.Uninferable and isinstance(res, nodes.ClassDef):
182-
class_def = nodes.ClassDef(
183-
name=f"{res.name}_typing",
184-
lineno=0,
185-
col_offset=0,
186-
parent=res.parent,
187-
)
188-
class_def.postinit(
189-
bases=[res],
190-
body=res.body,
191-
decorators=res.decorators,
192-
metaclass=create_typing_metaclass(),
193-
)
194-
return class_def
195-
196-
if len(node.args) == 2 and isinstance(node.args[0], nodes.Attribute):
197-
class_def = nodes.ClassDef(
198-
name=node.args[0].attrname,
199-
lineno=0,
200-
col_offset=0,
201-
parent=node.parent,
202-
)
203-
class_def.postinit(
204-
bases=[], body=[], decorators=None, metaclass=create_typing_metaclass()
205-
)
206-
return class_def
207-
208-
return None
189+
if not PY39:
190+
# Here the node is a typing object which is an alias toward
191+
# the corresponding object of collection.abc module.
192+
# Before python3.9 there is no subscript allowed for any of the collections.abc objects.
193+
# The subscript ability is given through the typing._GenericAlias class
194+
# which is the metaclass of the typing object but not the metaclass of the inferred
195+
# collections.abc object.
196+
# Thus we fake subscript ability of the collections.abc object
197+
# by mocking the existence of a __class_getitem__ method.
198+
# We can not add `__getitem__` method in the metaclass of the object because
199+
# the metaclass is shared by subscriptable and not subscriptable object
200+
maybe_type_var = node.args[1]
201+
if not (
202+
isinstance(maybe_type_var, node_classes.Tuple)
203+
and not maybe_type_var.elts
204+
):
205+
# The typing object is subscriptable if the second argument of the _alias function
206+
# is a TypeVar or a tuple of TypeVar. We could check the type of the second argument but
207+
# it appears that in the typing module the second argument is only TypeVar or a tuple of TypeVar or empty tuple.
208+
# This last value means the type is not Generic and thus cannot be subscriptable
209+
func_to_add = astroid.extract_node(CLASS_GETITEM_TEMPLATE)
210+
res.locals["__class_getitem__"] = [func_to_add]
211+
else:
212+
# If we are here, then we are sure to modify object that do have __class_getitem__ method (which origin is one the
213+
# protocol defined in collections module) whereas the typing module consider it should not
214+
# We do not want __class_getitem__ to be found in the classdef
215+
_forbid_class_getitem_access(res)
216+
else:
217+
# Within python3.9 discrepencies exist between some collections.abc containers that are subscriptable whereas
218+
# corresponding containers in the typing module are not! This is the case at least for ByteString.
219+
# It is far more to complex and dangerous to try to remove __class_getitem__ method from all the ancestors of the
220+
# current class. Instead we raise an AttributeInferenceError if we try to access it.
221+
maybe_type_var = node.args[1]
222+
if isinstance(maybe_type_var, nodes.Const) and maybe_type_var.value == 0:
223+
# Starting with Python39 the _alias function is in fact instantiation of _SpecialGenericAlias class.
224+
# Thus the type is not Generic if the second argument of the call is equal to zero
225+
_forbid_class_getitem_access(res)
226+
return iter([res])
227+
return iter([astroid.Uninferable])
209228

210229

211230
MANAGER.register_transform(
@@ -223,4 +242,6 @@ def infer_typing_alias(
223242
)
224243

225244
if PY37:
226-
MANAGER.register_transform(nodes.Call, infer_typing_alias, _looks_like_typing_alias)
245+
MANAGER.register_transform(
246+
nodes.Call, inference_tip(infer_typing_alias), _looks_like_typing_alias
247+
)

‎astroid/scoped_nodes.py

Lines changed: 31 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,8 @@
5454
from astroid import util
5555

5656

57+
PY39 = sys.version_info[:2] >= (3, 9)
58+
5759
BUILTINS = builtins.__name__
5860
ITER_METHODS = ("__iter__", "__getitem__")
5961
EXCEPTION_BASE_CLASSES = frozenset({"Exception", "BaseException"})
@@ -2617,7 +2619,22 @@ def getitem(self, index, context=None):
26172619
try:
26182620
methods = dunder_lookup.lookup(self, "__getitem__")
26192621
except exceptions.AttributeInferenceError as exc:
2620-
raise exceptions.AstroidTypeError(node=self, context=context) from exc
2622+
if isinstance(self, ClassDef):
2623+
# subscripting a class definition may be
2624+
# achieved thanks to __class_getitem__ method
2625+
# which is a classmethod defined in the class
2626+
# that supports subscript and not in the metaclass
2627+
try:
2628+
methods = self.getattr("__class_getitem__")
2629+
# Here it is assumed that the __class_getitem__ node is
2630+
# a FunctionDef. One possible improvement would be to deal
2631+
# with more generic inference.
2632+
except exceptions.AttributeInferenceError:
2633+
raise exceptions.AstroidTypeError(
2634+
node=self, context=context
2635+
) from exc
2636+
else:
2637+
raise exceptions.AstroidTypeError(node=self, context=context) from exc
26212638

26222639
method = methods[0]
26232640

@@ -2627,6 +2644,19 @@ def getitem(self, index, context=None):
26272644

26282645
try:
26292646
return next(method.infer_call_result(self, new_context))
2647+
except AttributeError:
2648+
# Starting with python3.9, builtin types list, dict etc...
2649+
# are subscriptable thanks to __class_getitem___ classmethod.
2650+
# However in such case the method is bound to an EmptyNode and
2651+
# EmptyNode doesn't have infer_call_result method yielding to
2652+
# AttributeError
2653+
if (
2654+
isinstance(method, node_classes.EmptyNode)
2655+
and self.name in ("list", "dict", "set", "tuple", "frozenset")
2656+
and PY39
2657+
):
2658+
return self
2659+
raise
26302660
except exceptions.InferenceError:
26312661
return util.Uninferable
26322662

0 commit comments

Comments
 (0)