Specialize type variables of generic functions that are arguments

[rwbarton]

This kind of situation appears in pytokenize.py:

tokenprog, pseudoprog, single3prog, double3prog = map(
    re.compile, (Token, PseudoToken, Single3, Double3))

The inferred types of tokenprog, ... are currently Pattern[AnyStr], which is very wrong.
(Not enough of this file is annotated for that to cause other mypy errors yet, though.)

[JukkaL]

This doesn't seem to fix this for more general argument types. For example, what if the formal argument type is List[Callable[...]]? We could look into the types recursively, but not sure if that's the right thing to do (and it might be inefficient). I thought about this for a bit and this is far from clear to me.

[rwbarton]

A list-of-generic-function isn't a subtype of list-of-specific-function anyways, because List is invariant in its argument. But your concern is valid for covariant type constructors.

I'm going to think some more about why exactly this change is needed and what goes wrong without it...

[JukkaL]

The example I used involved a [...] expression in the argument so the list-of-generic-function could be specialized based on context (but isn't specialized correctly), but you are correct and it's a different case due to invariance.

[rwbarton]

I think you're talking about a case like

def f(xs: List[Callable[[T], S]], y: T) -> List[S]: pass
def id(x: U) -> U: pass

f([id], 1)

In principle we should be able to type check this as follows:

• The first pass of type checking finds that T = int.
• f provides the context List[Callable[[int], S]] to the expression [id], which is itself treated as a call to a generic function "[_]" that produces a list.
• [_] provides the context Callable[[int], S] to the expression id. Now when id is type checked, the result is a generic callable type, so the code in this change unifies that type with the context, resulting in Callable[[int], int].
• Now the first argument of f has type List[Callable[[int], int]] so we can infer that S = int.

In practice the third step doesn't work yet because we don't substitute variables in infer_function_type_arguments_using_context when the replacement would involve a type variable (here S).

[rwbarton]

I'm going to abandon this attempt for now since this is turning into a larger issue than expected and I'd rather work on #1261 first.

However one observation I've made is that in situations like #1506 and #1507 it usually works correctly, or almost correctly, to replace the generic function by a lambda (re.compile -> lambda s: re.compile(s)). Perhaps that suggests a better strategy for dealing with these cases.