typing-extensions 4.13.0 breaks pydantic (<2.10) and SQLAlchemy (latest) project when using custom type

[Giuzzilla]

It seems that by upgrading typing-extensions from 4.12.2 to 4.13.0, a project with a simple Pydantic model referencing a custom type breaks.

Reproducible example:

from typing import Literal

from pydantic import BaseModel

type MyType = Literal["test1", "test2"]

class Test(BaseModel):
    a: int
    b: MyType

Throws:

pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for MyType. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.

If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.

I'm on Pydantic 2.9.0 and Python 3.12.9

I'm not sure if this is an issue on pydantic side and whether it should be fixed there, just pretty sure that it starts to break with 4.13.0

EDIT (see conversation):
Pydantic seems fixed on >=2.10

SQLAlchemy is broken on the latest available version (2.0.39) with a very similar case:

from typing import Literal

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

type MyType = Literal["test1", "test2"]


class Base(DeclarativeBase):
    pass


class MyModel(Base):
    __tablename__ = "my_model"

    id: Mapped[int] = mapped_column(primary_key=True)
    my_type: Mapped[MyType]

[Giuzzilla]

Seems to be fine with the combination of Pydantic >=2.10.0 + typing-extensions 4.13.0

[srittau]

This is interesting, as typing-extensions had no Literal-related changes between 4.12 and 4.13. Furthermore, in Python 3.12 it just re-exports typing.Literal, so I'm not sure what changed.

[srittau]

But FWIW, I can reproduce the problem using Python 3.12.7 and pydantic 2.10.6 (working)/2.9.2 (not working).

[Giuzzilla]

Just to add to the strangeness of the issue, upgrading pydantic does fix the error on the example I made above, but let's suppose I use that type also in a Mapped[MyType] on SQLAlchemy (2.0.39), it fails with:

sqlalchemy.exc.ArgumentError: Could not locate SQLAlchemy Core type for Python type MyType inside the 'my_type' attribute Mapped annotation

I can give a full example later (I'm just finding this out because I had indeed a Literal type in my real project used both by a declarative model on SQLAlchemy and on a Pydantic model)

[Giuzzilla]

Here is a reproducible example of a simple SQLAlchemy (latest, 2.0.39) model that fails:

from typing import Literal

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

type MyType = Literal["test1", "test2"]


class Base(DeclarativeBase):
    pass


class MyModel(Base):
    __tablename__ = "my_model"

    id: Mapped[int] = mapped_column(primary_key=True)
    my_type: Mapped[MyType]

sqlalchemy.exc.ArgumentError: Could not locate SQLAlchemy Core type for Python type MyType inside the 'my_type' attribute Mapped annotation.

Again, no issues with typing-extensions 4.12.2

[brianschubert]

The error going-away bisects to pydantic/pydantic#10713 (released with v2.10.0). The history is linked from there. Seems that this was an issue prompted by #477, caught by our CI (#493), and fixed in pydantic before 4.13.0 was released.

[Giuzzilla]

Here is a reproducible example of a simple SQLAlchemy (latest, 2.0.39) model that fails:

from typing import Literal

from sqlalchemy.orm import DeclarativeBase, Mapped, mapped_column

type MyType = Literal["test1", "test2"]


class Base(DeclarativeBase):
    pass


class MyModel(Base):
    __tablename__ = "my_model"

    id: Mapped[int] = mapped_column(primary_key=True)
    my_type: Mapped[MyType]

sqlalchemy.exc.ArgumentError: Could not locate SQLAlchemy Core type for Python type MyType inside the 'my_type' attribute Mapped annotation.

Again, no issues with typing-extensions 4.12.2

PS: This gives a SQLAlchemy's deprecation warning about type_annotation_map, but even if you add it to the type_annotation_map you still get the same error on 4.13.0

[Daraan]

The error originates from the existence of typing_extensions.TypeAliasType which differs from typing.TypeAliasType in 4.13+. Pydantic did not implement checks for both classes until pydantic/pydantic#10713 and takes a wrong branch which leads to your error.
Thanks brianschubert for finding out this so quickly.

---

A fix, without upgrading pydantic, is to not use the type statement that creates a typing.TypeAliasType, but explicitly using typing_extensions.TypeAliasType instead, i.e.:

from typing import Literal
from typing_extensions import TypeAliasType

from pydantic import BaseModel

MyType = TypeAliasType("MyType", Literal["test1", "test2"])

class Test(BaseModel):
    a: int
    b: MyType

[Giuzzilla]

The error originates from the existence of typing_extensions.TypeAliasType which differs from typing.TypeAliasType in 4.13+. Pydantic did not implement checks for both classes until pydantic/pydantic#10713 and takes a wrong branch which leads to your error. Thanks brianschubert for finding out this so quickly.

A fix, without upgrading pydantic, is to not use the type statement that creates a typing.TypeAliasType, but explicitly using typing_extensions.TypeAliasType instead, i.e.:

from typing import Literal
from typing_extensions import TypeAliasType

from pydantic import BaseModel

MyType = TypeAliasType("MyType", Literal["test1", "test2"])

class Test(BaseModel):
a: int
b: MyType

Yes, either removing type or upgrading pydantic seems to work.

However, SQLAlchemy seems to have a similar issue and has no fixed version for what I understood (I'm on the latest version). And I don't know which other packages might have similar issues.

Unless I remove type annotations everywhere I use this pattern (a bit impractical for my use-case), so for now I'm forced to keep typing-extensions at 4.12.2 despite .13.0 being a minor upgrade.

[AlexWaygood]

We try as hard as we can to help packages avoid this kind of issue:

• We document here that other libraries must be careful not to assume that the typing_extensions version of a symbol and the typing version of a symbol will always be the same on any given Python versions, just because they have been in the past
• We run the tests of several other popular libraries every night to check whether we've accidentally broken one of their test suites (example issue that was automatically opened after the tests from pydantic failed one night: Third-party tests failed on Fri Oct 25 2024 #493)
• We always publish a release candidate a week before publishing the release properly, so that maintainers of third-party libraries and users can test the upcoming release in advance (latest example: https://github.com/python/typing_extensions/releases/tag/4.13.0rc1)

It's unfortunate that none of these checks caught the SQLAlchemy issue in advance here. I think a good action item for us would be to add SQLAlchemy to the list of projects we run the test suites for on a nightly basis; that will help ensure that this doesn't happen again. Other than that, however, I'm not sure what we can do here; it seems like the bug is on SQLAlchemy's side rather than our side, unfortunately.

[Giuzzilla]

We try as hard as we can to help packages avoid this kind of issue:

• We document here that other libraries must be careful not to assume that the typing_extensions version of a symbol and the typing version of a symbol will always be the same on any given Python versions, just because they have been in the past
• We run the tests of several other popular libraries every night to check whether we've accidentally broken one of their test suites (example issue that was automatically opened after the tests from pydantic failed one night: Third-party tests failed on Fri Oct 25 2024 #493)
• We always publish a release candidate a week before publishing the release properly, so that maintainers of third-party libraries and users can test the upcoming release in advance (latest example: https://github.com/python/typing_extensions/releases/tag/4.13.0rc1)

It's unfortunate that none of these checks caught the SQLAlchemy issue in advance here. I think a good action item for us would be to add SQLAlchemy to the list of projects we run the test suites for on a nightly basis; that will help ensure that this doesn't happen again. Other than that, however, I'm not sure what we can do here; it seems like the bug is on SQLAlchemy's side rather than our side, unfortunately.

This is totally understandable, I can open an issue on the SQLAlchemy side (even though to be honest I'm not entirely familiar with the specific issue). I thought it would still be worth it to raise it here given that someone else might be impacted and it was not obvious for me which package upgrade caused it.

[srittau]

From the discussion this seems to be a bug in pydantic, which was fixed in 2.10 and SQLAlchemy (sqlalchemy/sqlalchemy#12473), where a fix is under way (thanks, @Daraan!). I'm closing this here.