typing.IO and io.BaseIO type hierarchies are incompatible

[aucampia]

Python documentation for the open() builtin indicates that open() returns types from the io.IOBase hierarchy:

The type of file object returned by the open() function depends on the mode. When open() is used to open a file in a text mode ('w', 'r', 'wt', 'rt', etc.), it returns a subclass of io.TextIOBase (specifically io.TextIOWrapper). When used to open a file in a binary mode with buffering, the returned class is a subclass of io.BufferedIOBase. The exact class varies: in read binary mode, it returns an io.BufferedReader; in write binary and append binary modes, it returns an io.BufferedWriter, and in read/write mode, it returns an io.BufferedRandom. When buffering is disabled, the raw stream, a subclass of io.RawIOBase, io.FileIO, is returned.

Python documentation for typing.IO[...] suggests that objects returned by the open() builtin are of type typing.IO[...].

class typing.IO

class typing.TextIO

class typing.BinaryIO

Generic type IO[AnyStr] and its subclasses TextIO(IO[str]) and BinaryIO(IO[bytes]) represent the types of I/O streams such as returned by open().

From this, I deduce that if I type a function as accepting either typing.IO[typing.AnyStr] or io.IOBase, it should be able to accept any object returned from open(). And following from this, I would assume that it should be possible to have an object that is of both typing.IO and io.IOBase type, because if objects that are of typing.IO type could not be of typing.IOBase type, and objects of typing.IOBase type could not be of io.IOBase type, then objects returned from open() could not be in line with the documentation I quoted above.

Given this, I would expect the following code to pass type checking without error:

import io
import typing


def accept_text_io(obj: typing.TextIO) -> None:
    assert isinstance(obj, io.TextIOBase)


def accept_binary_io(obj: typing.BinaryIO) -> None:
    assert isinstance(obj, io.RawIOBase) or isinstance(obj, io.BufferedIOBase)

However, when I run mypy on it, I get the following errors.

$ poetry run mypy --show-error-codes --show-error-context --strict --warn-unreachable --warn-unused-configs --python-version 3.7 test_iotypes_000.py
test_iotypes_000.py: note: In function "accept_text_io":
test_iotypes_000.py:6: error: Subclass of "TextIO" and "TextIOBase" cannot exist: would have incompatible method signatures  [unreachable]
test_iotypes_000.py: note: In function "accept_binary_io":
test_iotypes_000.py:10: error: Subclass of "BinaryIO" and "RawIOBase" cannot exist: would have incompatible method signatures  [unreachable]
test_iotypes_000.py:10: error: Subclass of "BinaryIO" and "BufferedIOBase" cannot exist: would have incompatible method signatures  [unreachable]
Found 3 errors in 1 file (checked 1 source file)

To further determine what is going wrong here, why something can't be of both typing.IO types and io.IOBase types, I ran the following code through mypy, which again I would expect to pass without any errors:

import io
import typing


class DummyBaseIO0(typing.IO[typing.AnyStr], io.IOBase):
    pass


class DummyBaseIO1(typing.IO[typing.Any], io.IOBase):
    pass


class DummyTextIO(typing.TextIO, io.TextIOBase):
    pass


class DummyRawIO(typing.BinaryIO, io.RawIOBase):
    pass


class DummyRawIO(typing.BinaryIO, io.BufferedIOBase):
    pass

And for this code mypy reports the following errors, and specifically highlights the incompatibilities:

$ poetry run mypy --show-error-codes --show-error-context --strict --warn-unreachable --warn-unused-configs --python-version 3.7 test_iotypes_010.py
test_iotypes_010.py: note: In class "DummyBaseIO0":
test_iotypes_010.py:5: error: Definition of "readline" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__iter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "readlines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__enter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "writelines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__next__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__iter__" in base class "Iterator" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__next__" in base class "Iterator" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:5: error: Definition of "__iter__" in base class "Iterable" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py: note: In class "DummyBaseIO1":
test_iotypes_010.py:9: error: Definition of "readline" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:9: error: Definition of "__enter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py: note: In class "DummyTextIO":
test_iotypes_010.py:13: error: Definition of "__enter__" in base class "TextIO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "newlines" in base class "TextIO" is incompatible with definition in base class "TextIOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "errors" in base class "TextIO" is incompatible with definition in base class "TextIOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "encoding" in base class "TextIO" is incompatible with definition in base class "TextIOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "readline" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__iter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "readlines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__enter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "writelines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "read" in base class "IO" is incompatible with definition in base class "TextIOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__next__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__iter__" in base class "Iterator" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__next__" in base class "Iterator" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:13: error: Definition of "__iter__" in base class "Iterable" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py: note: In class "DummyRawIO":
test_iotypes_010.py:17: error: Definition of "__enter__" in base class "BinaryIO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:17: error: Definition of "readline" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:17: error: Definition of "write" in base class "IO" is incompatible with definition in base class "RawIOBase"  [misc]
test_iotypes_010.py:17: error: Definition of "__enter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:17: error: Definition of "writelines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py: note: At top level:
test_iotypes_010.py:21: error: Name "DummyRawIO" already defined on line 17  [no-redef]
test_iotypes_010.py: note: In class "DummyRawIO":
test_iotypes_010.py:21: error: Definition of "__enter__" in base class "BinaryIO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:21: error: Definition of "readline" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:21: error: Definition of "write" in base class "IO" is incompatible with definition in base class "BufferedIOBase"  [misc]
test_iotypes_010.py:21: error: Definition of "__enter__" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:21: error: Definition of "writelines" in base class "IO" is incompatible with definition in base class "IOBase"  [misc]
test_iotypes_010.py:21: error: Definition of "read" in base class "IO" is incompatible with definition in base class "BufferedIOBase"  [misc]
Found 37 errors in 1 file (checked 1 source file)

Summary

So in summary, I would expect the io.IOBase and typing.IO[...] hierarchies to be compatible, and all the code in here to pass mypy validation, but it does not. I'm not sure if there is a good explanation for why the two type hierarchies are incompatible.

Practically, to work around this, I have to now type variables as typing.Enum[typing.TextIO,io.TextIOBase] if it should work with TextIO like things. This is maybe partly due to some other issues (see python/mypy#11193, #6061, #6076), but definitely these incompatibilities do make things harder, and seem wrong to me.

Versions

This issue is written for:

• Python 3.7
• mypy 0.910
• Typeshed from mypy 0.910

All code can be found here.

[aucampia]

Somewhat related: Why does mypy complain about TextIOWrapper receiving GzipFile as argument 1?

[srittau]

From this, I deduce that if I type a function as accepting either typing.IO[typing.AnyStr] or io.IOBase, it should be able to accept any object returned from open(). And following from this, I would assume that it should be possible to have an object that is of both typing.IO and io.IOBase type, because if objects that are of typing.IO type could not be of typing.IOBase type, and objects of typing.IOBase type could not be of io.IOBase type, then objects returned from open() could not be in line with the documentation I quoted above.

The Python I/O classes violate the Liskov substitution problem and are a bit of a mess. They don't lend themselves well to typing, which lead to the introduction of the typing.IO class in PEP 484, predating protocols. They can't be made compatible and that has never been the goal. That mess is also why we don't recommend to use those classes for typing anymore if it can be avoided.

If you have any concrete suggestions on how to improve the typing classes in typeshed, please submit a PR so it can be evaluated and its impact can be judged, but I am closing this issue as I don't see anything actionable here.