Skip to content

Change cache/lru_cache typing to retain function signatures #12952

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
wants to merge 1 commit into from
Closed

Change cache/lru_cache typing to retain function signatures #12952

wants to merge 1 commit into from

Conversation

alwaysmpe
Copy link

@alwaysmpe alwaysmpe commented Nov 4, 2024 •
edited
Loading

My (second - #12945 my comments there aren't really relevant) attempt at fixing the cache & lru_cache typing problem. This is based on a local wrapper I wrote using newer generics syntax, see my comment for the newer generics approach. I'm not sure on type variance and have a few ignores. Mypy and pyright disagree in places too. I defer to maintainers as to what is right, my understanding of type variance is limited.

Note: the current implementation is quite clunky. I have attempted to meet the following constraints:

  1. Retain the Hashable argument requirement (requires a Hashable descriptor class)
  2. Ensure the return type has a cache_info, cache_clear and cache_parameters function
  3. Compatible with runtime type behaviour - at runtime the return type is _lru_cache_wrapper and so this must be part of the cached return type.
  4. Minimal ignores - _lru_cache_wrapper at runtime isn't subclassable so one of the tests expect it to be annotated with final. Making the descriptor protocols subclasses of this would simplify typing slightly but require ignores.
  5. Correct function argument type annotations of the returned callable object

It might be possible to simplify the code as is or some of these constraints may be considered not necessary. My takeaway from other conversations is that 2 is a hard requirement of maintainers, 5 is what I want. The other requirements may be negotiable.

But using this typeshed fork locally the below code example correctly type checks in pyright without any errors (and all type ignore errors are correctly ignoring an error).

# pyright: reportUnnecessaryTypeIgnoreComment=true
from functools import cache

from typing import (
    assert_type,
    TYPE_CHECKING,
    overload,
    override,
)

class CFnCls:
    @cache
    def fn(self, arg: int) -> int:
        print("method fn called")
        return arg

    @classmethod
    @cache
    def cls_fn(cls, arg: int) -> int:
        print("class fn called")
        return arg

    @staticmethod
    @cache
    def st_fn(arg: int) -> int:
        print("static fn called")
        return arg

cfn_inst = CFnCls()
cfn_inst.fn(1)
cfn_inst.fn.__wrapped__(cfn_inst, 1)
CFnCls.fn.__wrapped__(CFnCls(), 1)
CFnCls.st_fn(1)
CFnCls.st_fn.__wrapped__(1)

CFnCls.cls_fn(1)
CFnCls.cls_fn.__wrapped__(CFnCls, 1)
cfn_inst.fn(1)
CFnCls.st_fn(1)
CFnCls.cls_fn(1)
assert_type(cfn_inst.fn(1), int)
assert_type(CFnCls.st_fn(1), int)
assert_type(CFnCls.cls_fn(1), int)
CFnCls().fn.cache_clear()
CFnCls.fn.cache_clear()
CFnCls.st_fn.cache_clear()
CFnCls.cls_fn.cache_clear()
if TYPE_CHECKING:
    # type errors - correct
    CFnCls().fn(1, 1) # type: ignore reportCallIssue
    CFnCls().fn.__wrapped__(CFnCls(), 1, 1) # type: ignore reportCallIssue
    CFnCls.fn(arg=1) # type: ignore reportCallIssue
    CFnCls.st_fn(1, 1) # type: ignore reportCallIssue
    CFnCls.cls_fn(1, 1) # type: ignore reportCallIssue

@cache
def fn(arg: int) -> int:
    return arg

@cache
def df_fn(arg: int, darg: str = "default"):
    print("default fn called")
    return darg
df_fn(1)

fn(1)
assert_type(fn(1), int)
if TYPE_CHECKING:
    # type error - correct
    fn(1, 2) # type: ignore reportCallIssue
fn.cache_clear()

@overload
@cache
def fn_overload(arg: int) -> int:
    ...
@overload
@cache
def fn_overload(arg: str) -> str:
    ...
@cache
def fn_overload(arg: int | str) -> int | str:
    return arg

fn_overload(1)
fn_overload("1")
# behaves same as previous cache, merges overloads
assert_type(fn_overload(1), int | str)
assert_type(fn_overload("1"), int | str)
fn_overload.cache_clear()
if TYPE_CHECKING:
    # type error - correct
    fn_overload(frozenset({1,2})) # type: ignore reportCallIssue

class Unhashable:
    @override
    def __eq__(self, value: object) -> bool:
        return False

@cache
def no_cache(arg: Unhashable, arg2: int) -> None:
    pass

if TYPE_CHECKING:
    no_cache(Unhashable(), 2) # type: ignore reportCallIssue

To do this I've defined overloads for cache/lru_cache that are constrained by the signature of the function they're called with. This assumes self as a first parameter is an object method, cls is a classmethod and everything else is either a static method or a function.

Then direct each overload to a descriptor protocol that mirrors the binding behaviour of methods/functions/classmethods.

The returned type is then a union with a callable that takes arbitrary arguments that are hashable. This means that the previous hashable requirement is retained (type checkers don't know which type was returned so calls must satisfy both, the union takes any arguments as long as they're hashable while the descriptor protocol mirrors the cached function's behaviour).

@alwaysmpe
Change cache/lru_cache typing to retain function signatures
2874459 
Define overloads for `cache`/`lru_cache` that are constrained by the signature of the function they're called with. This assumes `self` as a first parameter is a function method, `cls` is a classmethod and everything else are either static methods or functions.

Then direct each overload to a descriptor protocol that mirrors the binding behaviour of methods/functions/classmethods.

The returned type is then a union with a callable that takes arbitrary arguments that are hashable. This means that the previous hashable requirement is retained (type checkers don't know which type was returned so calls must satisfy both, the union takes any arguments as long as they're hashable while the descriptor protocol mirrors the cached function's behaviour).
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Nov 4, 2024

Diff from mypy_primer, showing the effect of this PR on open source code:

dacite (https://github.com/konradhalas/dacite)
+ dacite/cache.py:11: error: Unused "type: ignore" comment  [unused-ignore]

pylint (https://github.com/pycqa/pylint)
+ pylint/checkers/utils.py:2312: error: "_lru_cache_wrapper" expects 2 type arguments, but 1 given  [type-arg]
+ pylint/checkers/utils.py:2312: error: Missing type parameters for generic type "_lru_cache_wrapper"  [type-arg]
+ pylint/checkers/utils.py:2313: error: List item 0 has incompatible type "_FunctionDescriptor[[Any], bool] | _FunctionHasHashable[[Any], bool] | _lru_cache_wrapper[[Any], bool]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2314: error: List item 1 has incompatible type "_FunctionDescriptor[[Any, Any], bool] | _FunctionHasHashable[[Any, Any], bool] | _lru_cache_wrapper[[Any, Any], bool]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2315: error: List item 2 has incompatible type "_FunctionDescriptor[[Any, Any | None], list[Any]] | _FunctionHasHashable[[Any, Any | None], list[Any]] | _lru_cache_wrapper[[Any, Any | None], list[Any]]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2316: error: List item 3 has incompatible type "_FunctionDescriptor[[Any], bool] | _FunctionHasHashable[[Any], bool] | _lru_cache_wrapper[[Any], bool]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2317: error: List item 4 has incompatible type "_FunctionDescriptor[[Any, str | None], Any | None] | _FunctionHasHashable[[Any, str | None], Any | None] | _lru_cache_wrapper[[Any, str | None], Any | None]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2318: error: List item 5 has incompatible type "_FunctionDescriptor[[Any, Any | None], dict[str, Any]] | _FunctionHasHashable[[Any, Any | None], dict[str, Any]] | _lru_cache_wrapper[[Any, Any | None], dict[str, Any]]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]
+ pylint/checkers/utils.py:2319: error: List item 6 has incompatible type "_FunctionDescriptor[[Any, Any | None, DefaultNamedArg(bool, 'compare_constants'), DefaultNamedArg(bool, 'compare_constructors')], Any | None] | _FunctionHasHashable[[Any, Any | None, DefaultNamedArg(bool, 'compare_constants'), DefaultNamedArg(bool, 'compare_constructors')], Any | None] | _lru_cache_wrapper[[Any, Any | None, DefaultNamedArg(bool, 'compare_constants'), DefaultNamedArg(bool, 'compare_constructors')], Any | None]"; expected "_lru_cache_wrapper[Any, Any]"  [list-item]

psycopg (https://github.com/psycopg/psycopg)
+ psycopg/psycopg/rows.py:143: error: Argument 2 to "__call__" of "_FunctionDescriptor" has incompatible type "*Generator[bytes | None, None, None]"; expected "bytes"  [arg-type]
+ psycopg/psycopg/types/enum.py:156: error: Incompatible types in assignment (expression has type "type[_BaseEnumLoader[E@register_enum]] | type[_BaseEnumLoader[E@_make_binary_loader]]", variable has type "type[_BaseEnumLoader[E@register_enum]] | type[_BaseEnumLoader[E@_make_loader]]")  [assignment]
+ psycopg/psycopg/types/enum.py:164: error: Incompatible types in assignment (expression has type "type[_BaseEnumDumper[E@register_enum]] | type[_BaseEnumDumper[E@_make_binary_dumper]]", variable has type "type[_BaseEnumDumper[E@register_enum]] | type[_BaseEnumDumper[E@_make_dumper]]")  [assignment]

prefect (https://github.com/PrefectHQ/prefect)
- src/prefect/settings/legacy.py:105: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "type[PrefectBaseSettings]"; expected "Hashable"  [arg-type]
- src/prefect/settings/legacy.py:105: note: Following member(s) of "PrefectBaseSettings" have conflicts:
- src/prefect/settings/legacy.py:105: note:     Expected:
- src/prefect/settings/legacy.py:105: note:         def __hash__() -> int
- src/prefect/settings/legacy.py:105: note:     Got:
- src/prefect/settings/legacy.py:105: note:         def __hash__(self: object) -> int
- src/prefect/settings/legacy.py:105: note:     Expected:
- src/prefect/settings/legacy.py:105: note:         def __hash__() -> int
- src/prefect/settings/legacy.py:105: note:     Got:
- src/prefect/settings/legacy.py:105: note:         def __hash__(self: object) -> int
- src/prefect/settings/legacy.py:136: error: Argument 1 to "__call__" of "_lru_cache_wrapper" has incompatible type "type[PrefectBaseSettings]"; expected "Hashable"  [arg-type]
+ src/prefect/settings/legacy.py:136: error: Argument 1 to "__call__" of "_FunctionHasHashable" has incompatible type "type[PrefectBaseSettings]"; expected "Hashable"  [arg-type]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/tests/test_temporal.py:2162: error: Argument "exclude" to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[str, str]"; expected "tuple[str]"  [arg-type]
+ ibis/backends/tests/test_temporal.py:2180: error: Argument "exclude" to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[str, str, str, str, str, str]"; expected "tuple[str]"  [arg-type]

twine (https://github.com/pypa/twine)
+ twine/settings.py:135: error: Incompatible return value type (got "_BoundDescriptor[[Resolver], [], str | None]", expected "str | None")  [return-value]
+ twine/settings.py:140: error: Incompatible return value type (got "_BoundDescriptor[[Resolver], [], str | None]", expected "str | None")  [return-value]

jinja (https://github.com/pallets/jinja)
+ src/jinja2/environment.py:1189: error: Unused "type: ignore" comment  [unused-ignore]
+ src/jinja2/environment.py:1204: error: Unused "type: ignore" comment  [unused-ignore]

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/proxy/mode_specs.py:167: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/proxy/mode_specs.py:167: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/connection.py:193: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/connection.py:193: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/utils/asyncio_utils.py:62: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "Any | str"; expected "tuple[Any, ...] | None"  [arg-type]
+ mitmproxy/tools/console/common.py:811: error: Argument "error_message" to "__call__" of "_FunctionDescriptor" has incompatible type "str | None"; expected "str"  [arg-type]
+ mitmproxy/tools/console/common.py:848: error: Incompatible types in assignment (expression has type "_FunctionDescriptor[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(str | None, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any] | _FunctionHasHashable[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(str | None, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any] | _lru_cache_wrapper[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(str | None, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any]", variable has type "_FunctionDescriptor[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(bool, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any] | _FunctionHasHashable[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(bool, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any] | _lru_cache_wrapper[[NamedArg(RenderMode, 'render_mode'), NamedArg(bool, 'focused'), NamedArg(str, 'marked'), NamedArg(bool, 'is_replay'), NamedArg(str, 'request_method'), NamedArg(str, 'request_scheme'), NamedArg(str, 'request_host'), NamedArg(str, 'request_path'), NamedArg(str, 'request_url'), NamedArg(str, 'request_http_version'), NamedArg(float, 'request_timestamp'), NamedArg(bool, 'request_is_push_promise'), NamedArg(bool, 'intercepted'), NamedArg(int | None, 'response_code'), NamedArg(str | None, 'response_reason'), NamedArg(int | None, 'response_content_length'), NamedArg(str | None, 'response_content_type'), NamedArg(float | None, 'duration'), NamedArg(str | None, 'error_message')], Any]")  [assignment]
+ mitmproxy/tools/console/common.py:853: error: Argument "is_replay" to "__call__" of "_FunctionDescriptor" has incompatible type "str | None"; expected "bool"  [arg-type]
+ mitmproxy/test/tflow.py:134: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/test/tflow.py:134: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/test/tflow.py:235: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/test/tflow.py:235: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/proxy/server.py:582: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/proxy/server.py:582: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/proxy/mode_servers.py:115: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/proxy/mode_servers.py:115: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/clientplayback.py:98: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[UpstreamMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/clientplayback.py:98: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[UpstreamMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/clientplayback.py:98: error: Need type annotation for "mode"  [var-annotated]
+ mitmproxy/addons/proxyserver.py:109: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:109: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:255: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:255: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:299: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/addons/proxyserver.py:299: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ProxyMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/master.py:135: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[ReverseMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/master.py:135: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[ReverseMode]"; expected "type[Never]"  [arg-type]
+ mitmproxy/master.py:135: error: Need type annotation for "mode"  [var-annotated]

rich (https://github.com/Textualize/rich)
+ rich/progress_bar.py:145: error: Argument 3 to "__call__" of "_BoundDescriptor" has incompatible type "str | None"; expected "str"  [arg-type]

schemathesis (https://github.com/schemathesis/schemathesis)
+ src/schemathesis/stateful/state_machine.py: note: In class "APIStateMachine":
+ src/schemathesis/stateful/state_machine.py:68: error: Signature of "_to_test_case" incompatible with supertype "RuleBasedStateMachine"  [override]
+ src/schemathesis/stateful/state_machine.py:68: note:      Superclass:
+ src/schemathesis/stateful/state_machine.py:68: note:          Union[_ClassmethodDescriptor[RuleBasedStateMachine, [], Any], _ClassmethodHasHashableDescriptor[RuleBasedStateMachine, [], Any], _lru_cache_wrapper[[Type[RuleBasedStateMachine]], Any]]
+ src/schemathesis/stateful/state_machine.py:68: note:      Subclass:
+ src/schemathesis/stateful/state_machine.py:68: note:          Union[_ClassmethodDescriptor[APIStateMachine, [], type], _ClassmethodHasHashableDescriptor[APIStateMachine, [], type], _lru_cache_wrapper[[Type[APIStateMachine]], type]]
+ src/schemathesis/specs/openapi/stateful/__init__.py: note: In function "create_state_machine":
+ src/schemathesis/specs/openapi/stateful/__init__.py:79: error: Argument 2 to "__call__" of "_FunctionDescriptor" has incompatible type "Tuple[Any, ...]"; expected "Iterator[str]"  [arg-type]
+ src/schemathesis/specs/openapi/stateful/__init__.py:79: note: "tuple" is missing following "Iterator" protocol member:
+ src/schemathesis/specs/openapi/stateful/__init__.py:79: note:     __next__
+ src/schemathesis/specs/openapi/stateful/__init__.py: note: At top level:

pip (https://github.com/pypa/pip)
+ src/pip/_internal/index/package_finder.py:902: error: Argument 1 to "__call__" of "_BoundDescriptor" has incompatible type "str | None"; expected "str"  [arg-type]
+ src/pip/_internal/resolution/resolvelib/provider.py:242: error: Signature of "is_satisfied_by" incompatible with supertype "AbstractProvider"  [override]
+ src/pip/_internal/resolution/resolvelib/provider.py:242: note:      Superclass:
+ src/pip/_internal/resolution/resolvelib/provider.py:242: note:          def is_satisfied_by(self, requirement: Requirement, candidate: Candidate) -> bool
+ src/pip/_internal/resolution/resolvelib/provider.py:242: note:      Subclass:
+ src/pip/_internal/resolution/resolvelib/provider.py:242: note:          _MethodDescriptor[PipProvider, [Requirement, Candidate], bool] | _MethodHasHashableDescriptor[PipProvider, [Requirement, Candidate], bool] | _lru_cache_wrapper[[PipProvider, Requirement, Candidate], bool]

manticore (https://github.com/trailofbits/manticore)
+ manticore/core/smtlib/solver.py:505: error: Signature of "can_be_true" incompatible with supertype "Solver"  [override]
+ manticore/core/smtlib/solver.py:505: note:      Superclass:
+ manticore/core/smtlib/solver.py:505: note:          def can_be_true(self, constraints: Any, expression: Any = ...) -> bool
+ manticore/core/smtlib/solver.py:505: note:      Subclass:
+ manticore/core/smtlib/solver.py:505: note:          _MethodDescriptor[SMTLIBSolver, [ConstraintSet, bool | Bool], bool] | _MethodHasHashableDescriptor[SMTLIBSolver, [ConstraintSet, bool | Bool], bool] | _lru_cache_wrapper[[SMTLIBSolver, ConstraintSet, bool | Bool], bool]
+ manticore/core/smtlib/solver.py:605: error: Signature of "get_all_values" incompatible with supertype "Solver"  [override]
+ manticore/core/smtlib/solver.py:605: note:      Superclass:
+ manticore/core/smtlib/solver.py:605: note:          def get_all_values(self, constraints: Any, x: Any, maxcnt: Any = ..., silent: Any = ...) -> Any
+ manticore/core/smtlib/solver.py:605: note:      Subclass:
+ manticore/core/smtlib/solver.py:605: note:          _MethodDescriptor[SMTLIBSolver, [ConstraintSet, Any, int | None, bool], Any] | _MethodHasHashableDescriptor[SMTLIBSolver, [ConstraintSet, Any, int | None, bool], Any] | _lru_cache_wrapper[[SMTLIBSolver, ConstraintSet, Any, int | None, bool], Any]

core (https://github.com/home-assistant/core)
+ homeassistant/util/unit_conversion.py:367: error: Signature of "converter_factory" incompatible with supertype "BaseUnitConverter"  [override]
+ homeassistant/util/unit_conversion.py:367: note:      Superclass:
+ homeassistant/util/unit_conversion.py:367: note:          _ClassmethodDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float], float]] | _ClassmethodHasHashableDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float], float]] | _lru_cache_wrapper[[type[BaseUnitConverter], str | None, str | None], Callable[[float], float]]
+ homeassistant/util/unit_conversion.py:367: note:      Subclass:
+ homeassistant/util/unit_conversion.py:367: note:          _ClassmethodDescriptor[SpeedConverter, [str | None, str | None], Callable[[float], float]] | _ClassmethodHasHashableDescriptor[SpeedConverter, [str | None, str | None], Callable[[float], float]] | _lru_cache_wrapper[[type[SpeedConverter], str | None, str | None], Callable[[float], float]]
+ homeassistant/util/unit_conversion.py:381: error: Signature of "converter_factory_allow_none" incompatible with supertype "BaseUnitConverter"  [override]
+ homeassistant/util/unit_conversion.py:381: note:      Superclass:
+ homeassistant/util/unit_conversion.py:381: note:          _ClassmethodDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float | None], float | None]] | _ClassmethodHasHashableDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float | None], float | None]] | _lru_cache_wrapper[[type[BaseUnitConverter], str | None, str | None], Callable[[float | None], float | None]]
+ homeassistant/util/unit_conversion.py:381: note:      Subclass:
+ homeassistant/util/unit_conversion.py:381: note:          _ClassmethodDescriptor[SpeedConverter, [str | None, str | None], Callable[[float | None], float | None]] | _ClassmethodHasHashableDescriptor[SpeedConverter, [str | None, str | None], Callable[[float | None], float | None]] | _lru_cache_wrapper[[type[SpeedConverter], str | None, str | None], Callable[[float | None], float | None]]
+ homeassistant/util/unit_conversion.py:447: error: Signature of "converter_factory" incompatible with supertype "BaseUnitConverter"  [override]
+ homeassistant/util/unit_conversion.py:447: note:      Superclass:
+ homeassistant/util/unit_conversion.py:447: note:          _ClassmethodDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float], float]] | _ClassmethodHasHashableDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float], float]] | _lru_cache_wrapper[[type[BaseUnitConverter], str | None, str | None], Callable[[float], float]]
+ homeassistant/util/unit_conversion.py:447: note:      Subclass:
+ homeassistant/util/unit_conversion.py:447: note:          _ClassmethodDescriptor[TemperatureConverter, [str | None, str | None], Callable[[float], float]] | _ClassmethodHasHashableDescriptor[TemperatureConverter, [str | None, str | None], Callable[[float], float]] | _lru_cache_wrapper[[type[TemperatureConverter], str | None, str | None], Callable[[float], float]]
+ homeassistant/util/unit_conversion.py:461: error: Signature of "converter_factory_allow_none" incompatible with supertype "BaseUnitConverter"  [override]
+ homeassistant/util/unit_conversion.py:461: note:      Superclass:
+ homeassistant/util/unit_conversion.py:461: note:          _ClassmethodDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float | None], float | None]] | _ClassmethodHasHashableDescriptor[BaseUnitConverter, [str | None, str | None], Callable[[float | None], float | None]] | _lru_cache_wrapper[[type[BaseUnitConverter], str | None, str | None], Callable[[float | None], float | None]]
+ homeassistant/util/unit_conversion.py:461: note:      Subclass:
+ homeassistant/util/unit_conversion.py:461: note:          _ClassmethodDescriptor[TemperatureConverter, [str | None, str | None], Callable[[float | None], float | None]] | _ClassmethodHasHashableDescriptor[TemperatureConverter, [str | None, str | None], Callable[[float | None], float | None]] | _lru_cache_wrapper[[type[TemperatureConverter], str | None, str | None], Callable[[float | None], float | None]]
+ homeassistant/components/xiaomi_aqara/config_flow.py:216: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "str | None"; expected "str"  [arg-type]
+ homeassistant/components/mqtt_room/sensor.py:199: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "Any | None"; expected "str"  [arg-type]
+ homeassistant/components/esphome/update.py:63: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/update.py:63: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/update.py:63: error: "Never" has no attribute "get_entry_data"  [attr-defined]
+ homeassistant/components/esphome/update.py:64: error: Statement is unreachable  [unreachable]
+ homeassistant/components/esphome/light.py:178: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:187: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:198: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:207: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:208: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:213: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:225: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:242: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:247: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:261: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/light.py:342: error: Argument 1 to "__call__" of "_FunctionDescriptor" has incompatible type "tuple[int, ...]"; expected "list[int]"  [arg-type]
+ homeassistant/components/esphome/__init__.py:61: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/__init__.py:61: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/__init__.py:61: error: Need type annotation for "domain_data"  [var-annotated]
+ homeassistant/components/esphome/__init__.py:62: error: Statement is unreachable  [unreachable]
+ homeassistant/components/esphome/__init__.py:89: error: Argument 2 to "__get__" of "_ClassmethodDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/__init__.py:89: error: Argument 2 to "__get__" of "_ClassmethodHasHashableDescriptor" has incompatible type "type[DomainData]"; expected "type[Never]"  [arg-type]
+ homeassistant/components/esphome/__init__.py:89: error: "Never" has no attribute "get_or_create_store"  [attr-defined]

@alwaysmpe
Copy link
Author

alwaysmpe commented Nov 4, 2024 •
edited
Loading

My summary of the mypy primer errors:

  1. pylint errors result from differences in return type and would require a change to pylint, unlikely to be a problem.
  2. some errors (eg those in psycopg) may be because my type invariance isn't specified correctly, or they're calling incorrectly.
  3. Where a parent class method definition didn't have type annotations and a cached overriding method did (eg manticore can_be_true), this now results in a correct error, the more constrained method breaks liskov substitution.
  4. Where a method is cached in both the parent and subclass this is now causing a type error (eg core) due to type invariance. I'm not sure if this is correct and desirable. I defer to others.
  5. I'm not sure about some of mitmproxy errors.

There is an error in twine caused because they're using lru_cache with property instead of cached_property, so apparently this isn't currently compatible with that use case. They're not specifying any arguments to lru_cache so switching to cached_property would solve their use case, but not others.

Testing something similar locally, I get Any as the return type which isn't ideal. Not sure why there is a difference. This could be fixed if a similar change was made to property so that it correctly mapped the descriptor protocol. I might make that change. Without this change I get the correct type, but I suspect pyright has property as a special case.

However, I think this just highlights a bigger issue - the descriptor protocol isn't easily implemented or wrapped. The current conversation to fix this was about changing how ParamSpec binds self, but I think that's trying to solve the wrong problem, I think we'd benefit from a descriptor/decorator ABC. For example, in the functools file there's this:

typeshed/stdlib/functools.pyi

Lines 137 to 138 in 5243d30

# With protocols, this could change into a generic protocol that defines __get__ and returns _T
_Descriptor: TypeAlias = Any

descriptors and decorators are a pain.

@alwaysmpe
Copy link
Author

alwaysmpe commented Nov 4, 2024 •
edited
Loading

I've asked a question at microsoft/pyright#9384 as I think it might be possible to simplify this, but possibly some work elsewhere needed.

@alwaysmpe
Copy link
Author

This works but I also don't think this is the right way to do it.

@alwaysmpe alwaysmpe closed this Nov 5, 2024
@brianschubert brianschubert mentioned this pull request Nov 18, 2024
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@alwaysmpe