Redundant type specification for attributes

[dvarrazzo]

It has been raised to my attention (psycopg/psycopg#212) that Python typing tools are going in the direction of applying different, stricter rules to class attributes compared to the inference rules used for normal variables. From https://github.com/python/typing/blob/master/docs/source/libraries.rst#type-completeness:

# Class with partially unknown type
class BaseClass:
    # Missing type annotation
    height = 2.0

As far as I can see, humans can see, computers can see, the type of the height attribute is specified as a float. Should the class author wanted to allow broader types, they could specify its type: float | None or float | complex. But the type of the expression being transferred, by default, to the attribute, is not only a reasonable thing: doing otherwise is inconsistent with how types are inferred in local variables.

As far as I can see (#913) there is no consensus for this change.

As a library author I don't want type annotation to become more of a chore than already is.

class BaseCursor(Generic[ConnectionType, Row]):
    ...
    def __init__(self, connection: ConnectionType):                             
        self._conn = connection
        self.format = Format.TEXT  # It is *explicit* what is this type.

I would appreciate if this choice will be reconsidered.

[erictraut]

To clarify, this isn't specific to class attributes. It applies to all symbols that comprise the public interface of a library that includes a "py.typed" marker and therefore claims to be "typed".

Can we agree that it's in the interests of the consumers of a typed library to have consistent behaviors among all type checkers and language servers? If we can agree on that point, then the next step is to discuss how best to achieve this objective. I authored the initial draft of the guidance that you cited above, and it represents my best attempt to achieve this objective. If you have other proposals, let's discuss them.

As I mentioned in the guidance document and in other threads on this topic, a variable assignment is not the same as a type declaration, at least from the perspective of a type checker. A type declaration is a clear statement of intent that has well-defined meaning in the Python typing specifications (PEP 484, 526, etc.). When a variable has no type annotation, the intent (from a typing perspective) is left to interpretation. Type checkers will generally attempt to infer the type of the variable based on its assignment(s), but the mechanisms by which this are done are not standardized, and they vary by type checker. In a simple case like x = 2.0, most (all?) of the major type checkers will infer a type of float, but as soon as you move away from a trivial example, type inference rules differ between type checkers.

We could potentially carve out some specific rules for how certain trivial inference cases are handled. We could document those as part of the typing standard, and all of the type checkers could adopt those behaviors. That approach would require us to also clearly specify what cases are not standardized.

Alternatively, we could collectively agree that consistent type interpretations for typed libraries across the Python ecosystem is not a goal, and we can leave the interpretation of types up to each tool. That would seem to run counter to the significant efforts over the past several years to improve static type analysis within the Python community. Static type analysis provides many benefits beyond finding bugs at edit time. It also powers time-saving features like completion suggestions, function signature help, and in-place docstring viewing.

@dvarrazzo, it sounds like your major concern here is that it's a burden for library maintainers to include type annotations for the symbols that comprise the library's public interface. Is my understanding correct, or do you have other objections or concerns? If my understanding of your concern is correct, perhaps there are ways we can mitigate that burden through tooling, community contributions, etc. For example, I've invested in a "--verifytypes" mode in pyright that will help identify which symbols in a "py.typed" library are missing type annotations. Many library authors have started using this tool over the past year to make their libraries "type complete" and keep them so.

For full transparency, I'm the primary author of pyright, Microsoft's open-source Python type checker. Pyright forms the core of Pylance, the default Python language server for VS Code which is used by millions of Python developers. My colleagues and I care deeply about making the Python development experience as enjoyable and productive as possible. I think it's safe to say that everyone in the Python typing community share these goals. Let's continue to work together to figure out the best way to do this.

[erictraut]

@dvarrazzo, thanks for articulating your concerns. Let me provide some additional context and history.

Type annotations within Python were initially used primarily for static type checking. Today, these annotations power other features that benefit Python developers. This includes completion suggestions, call signature help, and in-editor documentation. For every Python developer who runs a static type checker on their code, you will find another hundred Python developers who do not use a static type checker but take advantage of these other features I mentioned. This means the investments that you and other library authors have made in type annotations are having a big impact. We appreciate the work you’ve done!

Pyright was designed to be not only a standards-compliant static type checker for large Python code bases but also the foundation for a performant Python language server. Pylance, which is the default Python language server for VS Code, is built on top of Pyright and is actively used by millions of developers every day. The design considerations for Pyright are therefore a little different from those of a command-line static type checker like mypy. Users want to see completion suggestions appear quickly (within tens of milliseconds ideally), and they want to see docstrings appear inline for the functions and methods they are calling.

I wrote the guidance document to help library authors and maintainers annotate and document their code in a way that works great with not only static type checkers but also language servers. After posting the document to the pyright site, one of the maintainers of the python/typing site asked whether I was OK with him including the document here. The pyright-specific references were removed along the way.

I’ve heard from many other library authors and maintainers, and I’ve used their feedback to develop and refine the "--verifytypes" feature in pyright. For example, the warnings about missing docstrings came from a suggestion from a library author who said “we know that some of our classes, functions, and methods are missing docstrings; can you help us identify these?”. If you have specific suggestions about this tool or its output, I’m interested in hearing that feedback, but it’s probably better to do that in the context of the Pyright github site rather than this forum.

I had a chance yesterday to look more closely at your library (psycopg). I can immediately see a few small changes to the guidance and to Pyright’s current implementation that will mitigate some of your concerns. I will work on those changes. There’s a separate set of potential changes — specific carve-outs for simple type inference cases — that I think are worth exploring. I did some brainstorming with my colleagues (including Guido and the Pylance team), and we came up with some initial ideas. I’ll put together a concrete proposal and post it here at a later date so you and others who are following this discussion can provide feedback.

Let’s all appreciate that Python is a healthy and vibrant ecosystem with evolving tools and standards. It’s through forums and conversations like this one that we can work together to help it evolve in a way that benefits the community as a whole — including Python developers, library authors, and tools providers. I appreciate you taking the time to engage in this discussion.

[hauntsaninja]

Thanks Eric! Just wanted to +1 on additional inference rules being an actionable and desirable thing here, in particular, some special casing for __init__. As an in-IDE user of Pylance, I've been tripped up by the loss of inference of attributes when going from working on a library to using it. Looking forward to hearing your suggestions!

[dvarrazzo]

Thank you very much @erictraut. Looking forward for your followup.

[erictraut]

@dvarrazzo, thanks for your patience. I've done some additional analysis and proposed some changes that I think will mitigate some of your concerns. I've posted my analysis and recommendations in the form of a PR so we can use github's code review tools for comments and discussions. Here's a link: https://github.com/microsoft/pyright/pull/3051/files. Anyone who is interested in this thread is welcome to read the document and provide feedback and suggestions. Thanks!

[erictraut]

I've implemented the recommendations in my PR request. Pyright now falls back on type inference if it encounters an unannotated symbol in a "py.typed" source file. This change will also be included in next week's release of pylance.

The "--verifytypes" option within the latest version of pyright (1.1.225) now distinguishes between "known" (annotated) types, "unknown" types (those that are not annotated and cannot be inferred), and "ambiguous types" (those that are missing an annotation and can be inferred by pyright but could be inferred differently by other Python type checkers). The logic for determining whether an unannotated symbol is "ambiguous" takes into account some of the simpler common cases where all type checkers are almost 100% guaranteed to infer the same type.

I've updated the documentation for the "--verifytypes" option in pyright to include additional information about ambiguous types.