builtins.filter compat with typing.TypeGuard

[Jasha10]

Based on microsoft/pyright#2768.

This change enables the following use-case:

def is_not_none(x: Optional[int]) -> TypeGuard[int]:
    return x is not None

list_optional: list[Optional[int]] = [0, None, 1, None, 2]
generate_ints: Iterable[int] = filter(is_not_none, list_optional)

[Jasha10]

It has been brought to my attention that this PR is related to issue #5661 and to PR #6140.

[Jasha10]

It appears that the diff implemented in this PR is identical to that from #6140.

[JelleZijlstra]

Interesting, maybe the latest mypy release fixed things. cc @sobolevn.

[sobolevn]

Yes, this was my PR to mypy: python/mypy#11314 after our initial attempt: #6140

[sobolevn]

Can you please attach a simple manual test to show that it is working?

[hauntsaninja]

Yup, #6140 has the correct overload order. I think that wasn't tested against mypy 0.930, so worth trying again.

[sobolevn]

Yup, #6140 has the correct overload order. I think that wasn't tested against mypy 0.930, so worth trying again.

Let's leave this to @Jasha10, this is their first contribution 🙂

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/csv/__init__.py:72: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[Any], TypeGuard[bool]]"
+ ibis/backends/csv/__init__.py:78: error: "bool" has no attribute "name"
+ ibis/backends/csv/__init__.py:82: error: "bool" has no attribute "schema"
+ ibis/backends/csv/__init__.py:91: error: "bool" has no attribute "schema"
+ ibis/backends/csv/__init__.py:92: error: Dict entry 0 has incompatible type "bool": "Any"; expected "Node": "Any"

[AlexWaygood]

Diff from mypy_primer, showing the effect of this PR on open source code:

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/csv/__init__.py:72: error: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[Any], TypeGuard[bool]]"
+ ibis/backends/csv/__init__.py:78: error: "bool" has no attribute "name"
+ ibis/backends/csv/__init__.py:82: error: "bool" has no attribute "schema"
+ ibis/backends/csv/__init__.py:91: error: "bool" has no attribute "schema"
+ ibis/backends/csv/__init__.py:92: error: Dict entry 0 has incompatible type "bool": "Any"; expected "Node": "Any"

Huh, that's weird. I'm guessing mypy is failing to fall back to the third overload due to some strange bool/TypeGuard subtyping behaviour.

[sobolevn]

This one is the most strange: Argument 1 to "filter" has incompatible type "Callable[[Any], bool]"; expected "Callable[[Any], TypeGuard[bool]]"

It should just continue with other overload (probably this one: def __init__(self, __function: Callable[[_T], Any], __iterable: Iterable[_T]) -> None: ...). Why does it error?

I will take a look into mypy itself 👍

[sobolevn]

I am unable to reproduce this locally.
All my regular examples work as intended. But, I am using master branch, not 0.930

[JelleZijlstra]

Going to merge this since the change is correct and mostly works with mypy. If it does expose a mypy bug, that can be tracked over on the mypy repo.