Do not overuse current_node_deferred flag

[ilevkivskyi]

The only valid use case for this flag is to prevent immediate use of types that contain bogus Anys. All other uses are bugs, since this means that an Any type appearing somewhere can trigger an undesired behavior.

This is a first PR in a short series of PRs to get rid of current_node_deferred completely in favor of a new TypeOfAny, and ultimately deferring top-levels.

[ilevkivskyi]

Hm, the fallout is bigger than I expected. And it looks like I was wrong at least in one aspect: --warn-no-return requires some kind of a "global" flag, as some expressions can turn from Any (not ready) to NoReturn. I will go through the errors tomorrow.

[ilevkivskyi]

Actually I think I figured out another thing: the overload ambiguity check already has some special handling for deferred Anys (that have TypeOfAny.special_form), I adjusted it to avoid a false positive in a test, but now more Anys are generated. This may mean that it would be impractical (if not impossible) to split my change into multiple PRs. I will try one thing now to check.

[github-actions]

Diff from mypy_primer, showing the effect of this PR on open source code:

static-frame (https://github.com/static-frame/static-frame)
+ static_frame/core/util.py:1494: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/util.py:1663: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/util.py:1664: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
- static_frame/core/util.py:2010: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[Any]]", variable has type "ndarray[tuple[int], dtype[Any]]")  [assignment]
- static_frame/core/util.py:2012: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[Any]]", variable has type "ndarray[tuple[int], dtype[Any]]")  [assignment]
- static_frame/core/util.py:2015: error: Incompatible types in assignment (expression has type "ndarray[tuple[int, ...], dtype[Any]]", variable has type "ndarray[tuple[int], dtype[Any]]")  [assignment]
+ static_frame/core/util.py:2350: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/util.py:2352: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/util.py:2362: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/util.py:2386: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/node_fill_value.py:191: error: Returning Any from function declared to return "Frame[Any, Any, *tuple[Any, ...]] | Series[Any, Any]"  [no-any-return]
+ static_frame/core/container_util.py:1702: error: Returning Any from function declared to return "Hashable | int | builtins.bool | numpy.bool[builtins.bool] | integer[Any] | <10 more items> | None"  [no-any-return]
+ static_frame/core/index_base.py:502: error: Returning Any from function declared to return "ndarray[Any, dtype[numpy.bool[builtins.bool]]]"  [no-any-return]
+ static_frame/core/index_base.py:521: error: Returning Any from function declared to return "ndarray[Any, dtype[numpy.bool[builtins.bool]]]"  [no-any-return]
+ static_frame/core/index.py:787: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/index.py:1386: error: Returning Any from function declared to return "TLabel | ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/index_hierarchy.py:1369: error: Returning Any from function declared to return "int"  [no-any-return]
+ static_frame/core/index_hierarchy.py:1923: error: Returning Any from function declared to return "int | integer[Any] | ndarray[Any, Any] | list[int] | slice[Any, Any, Any] | None"  [no-any-return]
+ static_frame/core/index_hierarchy.py:2329: error: Returning Any from function declared to return "ndarray[Any, Any]"  [no-any-return]
+ static_frame/core/series.py:3143: error: Returning Any from function declared to return "TLabel | ndarray[Any, Any]"  [no-any-return]
- static_frame/core/pivot.py:318: error: Invalid index type "list[int | integer[Any] | ndarray[Any, Any] | list[int] | slice[Any, Any, Any] | None]" for "ndarray[tuple[int], Any]"; expected type "SupportsIndex | slice[Any, Any, Any] | EllipsisType | _SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]]] | builtins.bool | int | _NestedSequence[builtins.bool | int] | None | tuple[SupportsIndex | slice[Any, Any, Any] | EllipsisType | _SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]]] | builtins.bool | int | _NestedSequence[builtins.bool | int] | None, ...]"  [index]
+ static_frame/core/pivot.py:318: error: Invalid index type "list[int | integer[Any] | ndarray[Any, Any] | list[int] | slice[Any, Any, Any] | None]" for "ndarray[Any, Any]"; expected type "SupportsIndex | slice[Any, Any, Any] | EllipsisType | _SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]]] | builtins.bool | int | _NestedSequence[builtins.bool | int] | None | tuple[SupportsIndex | slice[Any, Any, Any] | EllipsisType | _SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]] | _NestedSequence[_SupportsArray[dtype[numpy.bool[builtins.bool]] | dtype[integer[Any]]]] | builtins.bool | int | _NestedSequence[builtins.bool | int] | None, ...]"  [index]

pandas (https://github.com/pandas-dev/pandas)
+ pandas/core/nanops.py:656: error: Unused "type: ignore" comment  [unused-ignore]
- pandas/core/common.py:295: error: Incompatible return value type (got "ndarray[Any, Any] | Iterable[Any]", expected "ndarray[Any, Any]")  [return-value]
+ pandas/core/internals/managers.py:563: error: Unused "type: ignore" comment  [unused-ignore]
- pandas/core/internals/blocks.py:2097: error: Argument 1 to "BlockPlacement" has incompatible type "signedinteger[_32Bit | _64Bit]"; expected "int | slice[Any, Any, Any] | ndarray[Any, Any]"  [arg-type]

mitmproxy (https://github.com/mitmproxy/mitmproxy)
+ mitmproxy/http.py:1076: error: Unused "type: ignore" comment  [unused-ignore]

werkzeug (https://github.com/pallets/werkzeug)
+ src/werkzeug/routing/rules.py:280: error: Unsupported target for indexed assignment ("None")  [index]

steam.py (https://github.com/Gobot1234/steam.py)
+ steam/trade.py:645: error: Missing keys ("instanceid", "classid", "market_hash_name", "assetid", "amount", "appid", "contextid", "missing", "rollback_new_assetid", "rollback_new_contextid") for TypedDict "TradeOfferReceiptItem"  [typeddict-item]
+ steam/trade.py:650: error: Missing keys ("instanceid", "classid", "market_hash_name", "assetid", "amount", "appid", "contextid", "missing", "rollback_new_assetid", "rollback_new_contextid") for TypedDict "TradeOfferReceiptItem"  [typeddict-item]

spark (https://github.com/apache/spark)
- python/pyspark/pandas/indexing.py:568: error: Argument 1 to "rename" of "DataFrame" has incompatible type "Any | tuple[Any, ...]"; expected "dict[Any, Any] | Callable[[Any], Any]"  [arg-type]

xarray (https://github.com/pydata/xarray)
- xarray/plot/utils.py:1747: error: Incompatible types in assignment (expression has type "str | list[str] | list[list[str]] | list[list[list[Any]]]", variable has type "list[Any]")  [assignment]

porcupine (https://github.com/Akuli/porcupine)
- porcupine/plugins/highlight/tree_sitter_highlighter.py:84: error: Incompatible return value type (got "Any | None", expected "str")  [return-value]

pydantic (https://github.com/pydantic/pydantic)
+ pydantic/fields.py:560: error: No overload variant of "Field" matches argument types "Any", "Any", "bool", "Any"  [call-overload]

sympy (https://github.com/sympy/sympy)
+ sympy/core/evalf.py:1329: error: Unused "type: ignore" comment  [unused-ignore]

freqtrade (https://github.com/freqtrade/freqtrade)
- freqtrade/exchange/binance.py:75: error: Incompatible return value type (got "Any | None", expected "str")  [return-value]
- freqtrade/data/entryexitanalysis.py:334: error: Argument 1 to "Path" has incompatible type "Any | None"; expected "str | PathLike[str]"  [arg-type]

hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen)
- src/hydra_zen/structured_configs/_implementations.py:1196: error: Incompatible types in assignment (expression has type "None", variable has type "list[Just[Callable[[Any], Any]]]")  [assignment]
+ src/hydra_zen/structured_configs/_implementations.py:1196: error: Incompatible types in assignment (expression has type "None", variable has type "list[Any]")  [assignment]
- src/hydra_zen/structured_configs/_implementations.py:1198: error: Incompatible types in assignment (expression has type "Just[Callable[[Any], Any]]", variable has type "list[Just[Callable[[Any], Any]]]")  [assignment]

pandas-stubs (https://github.com/pandas-dev/pandas-stubs)
+ tests/test_scalars.py:1380: error: Expression is of type "Any", not "ndarray[tuple[int, ...], dtype[numpy.bool[builtins.bool]]]"  [assert-type]
+ tests/test_scalars.py:1383: error: Expression is of type "Any", not "ndarray[tuple[int, ...], dtype[numpy.bool[builtins.bool]]]"  [assert-type]

setuptools (https://github.com/pypa/setuptools)
+ pkg_resources/__init__.py:719: error: Need type annotation for "candidate"  [var-annotated]

discord.py (https://github.com/Rapptz/discord.py)
+ discord/app_commands/tree.py:1028: error: Argument "callback" to "ContextMenu" has incompatible type "Callable[[Interaction[Any], Never], CoroutineType[Any, Any, Any]] | Callable[..., CoroutineType[Any, Any, Any]]"; expected "Callable[[Interaction[Any], Member], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], User], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], Message], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], Member | User], Coroutine[Any, Any, Any]]"  [arg-type]
+ discord/app_commands/commands.py:2128: error: Argument "callback" to "ContextMenu" has incompatible type "Callable[[Interaction[Any], Never], CoroutineType[Any, Any, Any]] | Callable[..., CoroutineType[Any, Any, Any]]"; expected "Callable[[Interaction[Any], Member], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], User], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], Message], Coroutine[Any, Any, Any]] | Callable[[Interaction[Any], Member | User], Coroutine[Any, Any, Any]]"  [arg-type]
- discord/ui/button.py:317: error: "Callable[[V, Interaction[Any], Button[V]], Coroutine[Any, Any, Any]]" has no attribute "__discord_ui_model_type__"  [attr-defined]
+ discord/ui/button.py:317: error: Item "function" of "Callable[[V, Interaction[Any], Button[V]], CoroutineType[Any, Any, Any]] | Callable[..., CoroutineType[Any, Any, Any]]" has no attribute "__discord_ui_model_type__"  [union-attr]
- discord/ui/button.py:318: error: "Callable[[V, Interaction[Any], Button[V]], Coroutine[Any, Any, Any]]" has no attribute "__discord_ui_model_kwargs__"  [attr-defined]
+ discord/ui/button.py:318: error: Item "function" of "Callable[[V, Interaction[Any], Button[V]], CoroutineType[Any, Any, Any]] | Callable[..., CoroutineType[Any, Any, Any]]" has no attribute "__discord_ui_model_kwargs__"  [union-attr]

ibis (https://github.com/ibis-project/ibis)
+ ibis/backends/mysql/datatypes.py:60: error: Need type annotation for "typ"  [var-annotated]
+ ibis/backends/snowflake/__init__.py:587: note:     Expected:
+ ibis/backends/snowflake/__init__.py:587: note:         def __iter__(self) -> Iterator[tuple[str, DataType]]
+ ibis/backends/snowflake/__init__.py:587: note:     Got:
+ ibis/backends/snowflake/__init__.py:587: note:         def __iter__(self) -> Iterator[str]
+ ibis/backends/snowflake/__init__.py:588: error: Argument 1 to "Schema" has incompatible type "dict[Any, Any]"; expected "FrozenOrderedDict[str, DataType]"  [arg-type]
+ ibis/backends/snowflake/__init__.py:604: error: Argument 1 to "Schema" has incompatible type "dict[Any, Any]"; expected "FrozenOrderedDict[str, DataType]"  [arg-type]
+ ibis/backends/duckdb/__init__.py:312: error: Argument 1 to "Schema" has incompatible type "dict[Any, Any]"; expected "FrozenOrderedDict[str, DataType]"  [arg-type]

jax (https://github.com/google/jax)
+ jax/_src/state/primitives.py:561: error: Argument 1 to "insert" of "list" has incompatible type "int | None"; expected "SupportsIndex"  [arg-type]

core (https://github.com/home-assistant/core)
+ homeassistant/components/homeassistant/exposed_entities.py:260: error: Returning Any from function declared to return "bool"  [no-any-return]
+ homeassistant/components/isy994/sensor.py:166: error: Returning Any from function declared to return "dict[Any, Any] | str | None"  [no-any-return]

spack (https://github.com/spack/spack)
+ lib/spack/spack/spec_parser.py:403: error: Need type annotation for "virtuals"  [var-annotated]

AutoSplit (https://github.com/Toufool/AutoSplit)
+ src/user_profile.py:144:35: error: Missing keys ("split_hotkey", "reset_hotkey", "undo_split_hotkey", "skip_split_hotkey", "pause_hotkey", "screenshot_hotkey", "toggle_auto_reset_image_hotkey", "fps_limit", "live_capture_region", "capture_method", "capture_device_id", "capture_device_name", "default_comparison_method", "default_similarity_threshold", "default_delay_time", "default_pause_time", "loop_splits", "start_also_resets", "enable_auto_reset", "split_image_directory", "screenshot_directory", "open_screenshot", "screenshot_on", "captured_window_title", "capture_region") for TypedDict "UserProfileDict"  [typeddict-item]

[ilevkivskyi]

Unfortunately this looks harder than I thought, this opened a whole can of worms that would take time to untangle. So I will postpone this for now.