Skip to content

Add mixin-class-rgx option #5203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 8 commits into from
Oct 25, 2021
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CONTRIBUTORS.txt
Original file line number Diff line number Diff line change
Expand Up @@ -390,6 +390,8 @@ contributors:

* Ram Rachum (cool-RR)

* D. Alphus (Alphadelta14): contributor

* Pieter Engelbrecht

* Ethan Leba: contributor
Expand Down
2 changes: 2 additions & 0 deletions ChangeLog
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,8 @@ Release date: TBA

Closes #4981

* Support configuring mixin class pattern via ``mixin-class-rgx``

* Added new checker ``use-implicit-booleaness-not-comparison``: Emitted when
collection literal comparison is being used to check for emptiness.

Expand Down
5 changes: 3 additions & 2 deletions doc/faq.rst
Original file line number Diff line number Diff line change
Expand Up @@ -238,8 +238,9 @@ methods is doing nothing but raising NotImplementedError.
-------------------------------------------------------------------------------

To do so you have to set the ignore-mixin-members option to
"yes" (this is the default value) and to name your mixin class with
a name which ends with "mixin" (whatever case).
"yes" (this is the default value) and name your mixin class with
a name which ends with "Mixin" or "mixin" (default) or change the
default value by changing the mixin-class-rgx option.


6. Troubleshooting
Expand Down
2 changes: 2 additions & 0 deletions doc/whatsnew/2.12.rst
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,8 @@ Other Changes

Closes #4426

* Support configuring mixin class pattern via ``mixin-class-rgx``

* Normalize the input to the ``ignore-paths`` option to allow both Posix and
Windows paths

Expand Down
10 changes: 6 additions & 4 deletions pylint/checkers/async.py
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@ def open(self):
self._ignore_mixin_members = utils.get_global_option(
self, "ignore-mixin-members"
)
self._mixin_class_rgx = utils.get_global_option(self, "mixin-class-rgx")
self._async_generators = ["contextlib.asynccontextmanager"]

@checker_utils.check_messages("yield-inside-async-function")
Expand Down Expand Up @@ -81,10 +82,11 @@ def visit_asyncwith(self, node: nodes.AsyncWith) -> None:
# just skip it.
if not checker_utils.has_known_bases(inferred):
continue
# Just ignore mixin classes.
if self._ignore_mixin_members:
if inferred.name[-5:].lower() == "mixin":
continue
# Ignore mixin classes if they match the rgx option.
if self._ignore_mixin_members and self._mixin_class_rgx.match(
inferred.name
):
continue
Comment on lines +85 to +89
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Ignore mixin classes if they match the rgx option.
if self._ignore_mixin_members and self._mixin_class_rgx.match(
inferred.name
):
continue
# Ignore mixin classes if they match the rgx option.
if self._mixin_class_rgx.match(
inferred.name
):
continue

or

Suggested change
# Ignore mixin classes if they match the rgx option.
if self._ignore_mixin_members and self._mixin_class_rgx.match(
inferred.name
):
continue

@Pierre-Sassoulas I meant either removing all checks for mixim's (so both mixin_members and the regex) or only removing the members check

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, I understand now. I'll try to see why it was added in the first place, there must be a reason.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's like that since the beginning of time and I don't see any explanation. 24e9d35#diff-76ab71e7eb3954c60b39e11f3dfe5cc45ee2b139c7738dfbca1566b8d50cf0b8R71

But now that I think of it, It might be to avoid false positive when the mixin is actually an async manager even if the current class is not. It's impossible to know that without analyzing the class that are supposed to be mixed. So in fact I think we should keep it. What do you think ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That is probably the reason why "they" did this initially. However, I think that might warrant a change of the ignore_mixin_members option name to something like ignore_mixin_class_checks. That would be something for a different PR, but this name really does not fit what it is doing right now.

Is there a "old-name" system for option names?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should use the new regex pattern only in order to check if a class is supposed to be a mixin and drop the other check.

else:
continue
self.add_message(
Expand Down
5 changes: 4 additions & 1 deletion pylint/checkers/classes.py
Original file line number Diff line number Diff line change
Expand Up @@ -808,6 +808,9 @@ def __init__(self, linter=None):
self._first_attrs = []
self._meth_could_be_func = None

def open(self) -> None:
self._mixin_class_rgx = get_global_option(self, "mixin-class-rgx")

@astroid.decorators.cachedproperty
def _dummy_rgx(self):
return get_global_option(self, "dummy-variables-rgx", default=None)
Expand Down Expand Up @@ -1029,7 +1032,7 @@ def _check_unused_private_attributes(self, node: nodes.ClassDef) -> None:

def _check_attribute_defined_outside_init(self, cnode: nodes.ClassDef) -> None:
# check access to existent members on non metaclass classes
if self._ignore_mixin and cnode.name[-5:].lower() == "mixin":
if self._ignore_mixin and self._mixin_class_rgx.match(cnode.name):
# We are in a mixin class. No need to try to figure out if
# something is missing, since it is most likely that it will
# miss.
Expand Down
29 changes: 24 additions & 5 deletions pylint/checkers/typecheck.py
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,14 @@ def _missing_member_hint(owner, attrname, distance_threshold, max_choices):
}


def _emit_no_member(node, owner, owner_name, ignored_mixins=True, ignored_none=True):
def _emit_no_member(
node,
owner,
owner_name,
mixin_class_rgx: Pattern[str],
ignored_mixins=True,
ignored_none=True,
):
"""Try to see if no-member should be emitted for the given owner.

The following cases are ignored:
Expand All @@ -462,7 +469,7 @@ def _emit_no_member(node, owner, owner_name, ignored_mixins=True, ignored_none=T
return False
if is_super(owner) or getattr(owner, "type", None) == "metaclass":
return False
if owner_name and ignored_mixins and owner_name[-5:].lower() == "mixin":
if owner_name and ignored_mixins and mixin_class_rgx.match(owner_name):
return False
if isinstance(owner, nodes.FunctionDef) and (
owner.decorators or owner.is_abstract()
Expand Down Expand Up @@ -780,15 +787,25 @@ class TypeChecker(BaseChecker):
"no-member and other checks for the rest of the inferred objects.",
},
),
(
"mixin-class-rgx",
{
"default": ".*[Mm]ixin",
"type": "regexp",
"metavar": "<regexp>",
"help": "Regex pattern to define which classes are considered mixins "
"ignore-mixin-members is set to 'yes'",
},
),
(
"ignore-mixin-members",
{
"default": True,
"type": "yn",
"metavar": "<y_or_n>",
"help": 'Tells whether missing members accessed in mixin \
class should be ignored. A mixin class is detected if its name ends with \
"mixin" (case insensitive).',
"help": "Tells whether missing members accessed in mixin "
"class should be ignored. A class is considered mixin if its name matches "
"the mixin-class-rgx option.",
},
),
(
Expand Down Expand Up @@ -898,6 +915,7 @@ class should be ignored. A mixin class is detected if its name ends with \
def open(self) -> None:
py_version = get_global_option(self, "py-version")
self._py310_plus = py_version >= (3, 10)
self._mixin_class_rgx = get_global_option(self, "mixin-class-rgx")

@astroid.decorators.cachedproperty
def _suggestion_mode(self):
Expand Down Expand Up @@ -1040,6 +1058,7 @@ def visit_attribute(self, node: nodes.Attribute) -> None:
node,
owner,
name,
self._mixin_class_rgx,
ignored_mixins=self.config.ignore_mixin_members,
ignored_none=self.config.ignore_none,
):
Expand Down
5 changes: 4 additions & 1 deletion pylint/utils/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,10 @@
GLOBAL_OPTION_INT = Literal["max-line-length", "docstring-min-length"]
GLOBAL_OPTION_LIST = Literal["ignored-modules"]
GLOBAL_OPTION_PATTERN = Literal[
"no-docstring-rgx", "dummy-variables-rgx", "ignored-argument-names"
"no-docstring-rgx",
"dummy-variables-rgx",
"ignored-argument-names",
"mixin-class-rgx",
]
GLOBAL_OPTION_PATTERN_LIST = Literal["ignore-paths"]
GLOBAL_OPTION_TUPLE_INT = Literal["py-version"]
Expand Down
7 changes: 5 additions & 2 deletions pylintrc
Original file line number Diff line number Diff line change
Expand Up @@ -270,10 +270,13 @@ property-classes=abc.abstractproperty

[TYPECHECK]

# Tells whether missing members accessed in mixin class should be ignored. A
# mixin class is detected if its name ends with "mixin" (case insensitive).
# Tells whether missing members accessed in mixin class should be ignored.
# A class is considered mixin if its name matches the mixin-class-rgx option.
ignore-mixin-members=yes

# Regex pattern to define which classes are considered mixins.
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Regex pattern to define which classes are considered mixins.
# Regex pattern to define which classes are considered mixins and should be
exempted from certain checks.

@Pierre-Sassoulas Agree, but then let's add this. Perhaps we should also keep a list of exempted checks somewhere.. I don't know of a good place though.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe it should be a configurable option ? Also it seems we're skipping the check after doing all the calculations. We might want to skip the check as soon as we detect a mixin.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What about removing ignore-members and adding a new option ignore-checks-for-mixin which would be a comma separated list of checks that exclude Mixin? We could add the supported checks to the description of that option.
I think I have already thought of good way to deprecate an option.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, don't you prefer ignored-checks-for-mixin though ?

We already "deprecated" an option with blacklist/whitelist at some point #3961. I think the plan is to handle both until the end of time so users have no need to upgrade their configuration. There's no warning either for the same reason. Having to upgrade the configuration is bothersome with so many pylintrc in the wild already. I guess you had a different idea on how to do that ? I guess if the change isn't political it will attracts trolls less.

I think that if we don't want to maintain all of the legacy options we might have to put a configuration upgrader into place (so the migration to the new configuration file is automated). It could work well with the need of using a single possible configuration file (instead of pylintrc, .pylintrc, setup.cfg, pyproject.toml, ~/.pylintrc) and it will probably be only pyproject.toml in the long term.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Right, don't you prefer ignored-checks-for-mixin though ?

👍

We already "deprecated" an option with blacklist/whitelist at some point #3961. I think the plan is to handle both until the end of time so users have no need to upgrade their configuration. There's no warning either for the same reason. Having to upgrade the configuration is bothersome with so many pylintrc in the wild already. I guess you had a different idea on how to do that ? I guess if the change isn't political it will attracts trolls less.

I think that if we don't want to maintain all of the legacy options we might have to put a configuration upgrader into place (so the migration to the new configuration file is automated). It could work well with the need of using a single possible configuration file (instead of pylintrc, .pylintrc, setup.cfg, pyproject.toml, ~/.pylintrc) and it will probably be only pyproject.toml in the long term.

I think we should only do a configuration upgrader after have moved to argparse.

I intend to support both indeed, but with a deprecation warning when we notice ignore-members is being used.

Copy link
Member

@Pierre-Sassoulas Pierre-Sassoulas Oct 25, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sure, an upgrader wouldn't make sense as long as we still have optparse and aren't clear about what we want to deprecate. I'm not even sure if I want to have only pyporject.toml in 3.0. Just giving you the long term picture here :)

I intend to support both indeed, but with a deprecation warning when we notice ignore-members is being used

Sounds good. If the option is not being used by pylint but exists it's logical that we warn. Maybe we can suggest the option that replaced it in the warning so the user can copy paste the next conf ? It's not an upgrader, but it's close so we could create the base class for the upgrader and use it to get the recommendation ? :)

Something like:

class ConfigurationUpgrader:
    def next_value(option_name:str, value: Any) -> [str, any]:
        next_for_option_name = getattr(f"next_value_for_{option_name.replace('-', '_')}", None)
        if next_for_option_name:
            return next_for_option_name(value)
        return option_name, value

   def next_value_for_ignore_members(value: Any);
         return 'mixin-class-rgx", [".*Mmixin"]

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I toyed around with this, but it turned out to be a little bit more difficult than I initially thought.

They way we're loading options makes it really difficult to deprecate them and emit warnings for when users still use them.
They need to have an optdict in one of the checkers, but then we also load the option every time ourselves. Catching the option and changing it directly to the new name is also difficult as we would need to create combinations of every validator type with every other validator type (ie., to be able to translate a "yes-no" option to a "list of strings" option.
I think we should leave this as is, open an issue to track this and wait for the move to argparse. I hope that module provides better ways to do this.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for trying ! Do you think we should change the design of the options so that we're passing an object (argparse.Namespace ?) to the checker at initialization ?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think ideally we wouldn't work with any OptionsProviders at all and just load all possible options from a large dict of optdicts (or Arguments as suggested in #5152) at start up once and subsequently load the user defined options/overwrites.

I found that the load_defaults function was called 2/3 times for one single test and the whole concept of OptionsProviders creates numerous problems when different checker classes need to access the same option. (See #5195).
As indicated in #4814 option processing take a considerable time for our start-up, which I can imagine is because of stuff like calling load_defaults twice.
The get_global_option function is basically trying to circumvent this design of OptionProviders, as it turns out it is often much more logical to make options "global" for all checkers.

I don't know what the initial objective of OptionProviders was and I think we will likely run into unexpected problems when we would move to such a design, but from my experience with our config module this might in the end be much more maintainable.

mixin-class-rgx=.*MixIn

# List of module names for which member attributes should not be checked
# (useful for modules/projects where namespaces are manipulated during runtime
# and thus existing member attributes cannot be deduced by static analysis)
Expand Down
60 changes: 60 additions & 0 deletions tests/functional/m/mixin_class_rgx.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
"""Tests for the mixin-class-rgx option"""
# pylint: disable=too-few-public-methods


# Tests for not-async-context-manager


class AsyncManagerMixedin:
"""Class that does not match the option pattern"""

def __aenter__(self):
pass


class AsyncManagerMixin:
"""Class that does match the option pattern"""

def __aenter__(self):
pass


async def check_not_async_context_manager():
"""Function calling the classes for not-async-context-manager"""
async with AsyncManagerMixedin: # [not-async-context-manager]
pass
async with AsyncManagerMixin():
pass


# Tests for attribute-defined-outside-init


class OutsideInitMixedin:
"""Class that does not match the option pattern"""

def set_attribute(self):
"""Set an attribute outside of __init__"""
self.attr = 1 # [attribute-defined-outside-init]


class OutsideInitMixin:
"""Class that does match the option pattern"""

def set_attribute(self):
"""Set an attribute outside of __init__"""
self.attr = 1


# Tests for no-member


class NoMemberMixedin:
"""Class that does not match the option pattern"""

MY_CLASS = OutsideInitMixedin().method() # [no-member]

class NoMemberMixin:
"""Class that does match the option pattern"""

MY_OTHER_CLASS = NoMemberMixin().method()
3 changes: 3 additions & 0 deletions tests/functional/m/mixin_class_rgx.rc
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[TYPECHECK]
ignore-mixin-members=yes
mixin-class-rgx=.*[Mm]ixin
3 changes: 3 additions & 0 deletions tests/functional/m/mixin_class_rgx.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
not-async-context-manager:24:4:check_not_async_context_manager:Async context manager 'AsyncManagerMixedin' doesn't implement __aenter__ and __aexit__.:HIGH
attribute-defined-outside-init:38:8:OutsideInitMixedin.set_attribute:Attribute 'attr' defined outside __init__:HIGH
no-member:55:11::Instance of 'OutsideInitMixedin' has no 'method' member:INFERENCE