msgspec support (#481)

* msgspec first pass

* Fix typing import

* Test carefully for PyPy

* Docs

* Fix typing wrapper

* Fix PyPy CI some more

* Remove unused paramspec

* Use msgspec's datetime structurer

* More msgspec

* Ignore _cpython tests on PyPy

* More msgspec

* More doc work

* Fix

* Docs

* Fix test

* More msgspec work

* Pass through mapping to msgspec

* Fix counters

Author: Tinche

HISTORY.md

@@ -17,6 +17,9 @@ Our backwards-compatibility policy can be found [here](https://github.com/python
   ([#473](https://github.com/python-attrs/cattrs/pull/473))
 - Introduce {meth}`BaseConverter.get_structure_hook` and {meth}`BaseConverter.get_unstructure_hook` methods.
   ([#432](https://github.com/python-attrs/cattrs/issues/432) [#472](https://github.com/python-attrs/cattrs/pull/472))
+- Introduce the [_msgspec_](https://jcristharif.com/msgspec/) {mod}`preconf converter <cattrs.preconf.msgspec>`.
+  Only JSON is supported for now, with other formats supported by _msgspec_ to come later.
+  ([#481](https://github.com/python-attrs/cattrs/pull/481))
 - The default union handler now properly takes renamed fields into account.
   ([#472](https://github.com/python-attrs/cattrs/pull/472))
 - The default union handler now also handles dataclasses.
@@ -25,6 +28,8 @@ Our backwards-compatibility policy can be found [here](https://github.com/python
   ([#452](https://github.com/python-attrs/cattrs/pull/452))
 - The `include_subclasses` strategy now fetches the member hooks from the converter (making use of converter defaults) if overrides are not provided, instead of generating new hooks with no overrides.
   ([#429](https://github.com/python-attrs/cattrs/issues/429) [#472](https://github.com/python-attrs/cattrs/pull/472))
+- The preconf `make_converter` factories are now correctly typed.
+  ([#481](https://github.com/python-attrs/cattrs/pull/481))
 - The {class}`orjson preconf converter <cattrs.preconf.orjson.OrjsonConverter>` now passes through dates and datetimes to orjson while unstructuring, greatly improving speed.
   ([#463](https://github.com/python-attrs/cattrs/pull/463))
 - `cattrs.gen` generators now attach metadata to the generated functions, making them introspectable.

docs/_static/custom.css

@@ -72,7 +72,7 @@ span:target ~ h6:first-of-type {
 
 div.article-container > article {
   font-size: 17px;
-  line-height: 31px;
+  line-height: 29px;
 }
 
 div.admonition {
@@ -89,7 +89,7 @@ p.admonition-title {
 
 article > li > a {
   font-size: 19px;
-  line-height: 31px;
+  line-height: 29px;
 }
 
 div.tab-set {

docs/cattrs.preconf.rst

@@ -41,6 +41,14 @@ cattrs.preconf.msgpack module
    :undoc-members:
    :show-inheritance:
 
+cattrs.preconf.msgspec module
+-----------------------------
+
+.. automodule:: cattrs.preconf.msgspec
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
 cattrs.preconf.orjson module
 ----------------------------
 

docs/customizing.md

@@ -2,7 +2,7 @@
 
 This section describes customizing the unstructuring and structuring processes in _cattrs_.
 
-## Manual Un/structuring Hooks
+## Custom Un/structuring Hooks
 
 You can write your own structuring and unstructuring functions and register them for types using {meth}`Converter.register_structure_hook() <cattrs.BaseConverter.register_structure_hook>` and {meth}`Converter.register_unstructure_hook() <cattrs.BaseConverter.register_unstructure_hook>`.
 This approach is the most flexible but also requires the most amount of boilerplate.

docs/index.md

@@ -2,22 +2,40 @@
 ---
 maxdepth: 2
 hidden: true
+caption: Introduction
 ---
 
 self
 basics
 defaulthooks
+```
+
+```{toctree}
+---
+maxdepth: 2
+hidden: true
+caption: User Guide
+---
+
 customizing
 strategies
 validation
 preconf
 unions
 usage
 indepth
+```
+
+```{toctree}
+---
+maxdepth: 2
+hidden: true
+caption: Dev Guide
+---
+
 history
 benchmarking
 contributing
-API <modules>
 ```
 
 ```{include} ../README.md

docs/preconf.md

@@ -44,6 +44,9 @@ Optional install targets should match the name of the {mod}`cattrs.preconf` modu
 # Using pip
 $ pip install cattrs[ujson]
 
+# Using pdm
+$ pdm add cattrs[orjson]
+
 # Using poetry
 $ poetry add --extras tomlkit cattrs
 ```
@@ -56,15 +59,6 @@ Found at {mod}`cattrs.preconf.json`.
 Bytes are serialized as base 85 strings. Counters are serialized as dictionaries. Sets are serialized as lists, and deserialized back into sets. `datetime` s and `date` s are serialized as ISO 8601 strings.
 
 
-## _ujson_
-
-Found at {mod}`cattrs.preconf.ujson`.
-
-Bytes are serialized as base 85 strings. Sets are serialized as lists, and deserialized back into sets. `datetime` s and `date` s are serialized as ISO 8601 strings.
-
-`ujson` doesn't support integers less than -9223372036854775808, and greater than 9223372036854775807, nor does it support `float('inf')`.
-
-
 ## _orjson_
 
 Found at {mod}`cattrs.preconf.orjson`.
@@ -77,6 +71,61 @@ _orjson_ doesn't support integers less than -9223372036854775808, and greater th
 _orjson_ only supports mappings with string keys so mappings will have their keys stringified before serialization, and destringified during deserialization.
 
 
+## _msgspec_
+
+Found at {mod}`cattrs.preconf.msgspec`.
+Only JSON functionality is currently available, other formats supported by msgspec to follow in the future.
+
+[_msgspec_ structs](https://jcristharif.com/msgspec/structs.html) are supported, but not composable - a struct will be handed over to _msgspec_ directly, and _msgspec_ will handle and all of its fields, recursively.
+_cattrs_ may get more sophisticated handling of structs in the future.
+
+[_msgspec_ strict mode](https://jcristharif.com/msgspec/usage.html#strict-vs-lax-mode) is used by default.
+This can be customized by changing the {meth}`encoder <cattrs.preconf.msgspec.MsgspecJsonConverter.encoder>` attribute on the converter.
+
+What _cattrs_ calls _unstructuring_ and _structuring_, _msgspec_ calls [`to_builtins` and `convert`](https://jcristharif.com/msgspec/converters.html).
+What _cattrs_ refers to as _dumping_ and _loading_, _msgspec_ refers to as [`encoding` and `decoding`](https://jcristharif.com/msgspec/usage.html).
+
+Compatibility notes:
+- Bytes are un/structured as base 64 strings directly by _msgspec_ itself.
+- _msgspec_ [encodes special float values](https://jcristharif.com/msgspec/supported-types.html#float) (`NaN, Inf, -Inf`) as `null`.
+- `datetime` s and `date` s are passed through to be unstructured into RFC 3339 by _msgspec_ itself.
+- _attrs_ classes, dataclasses and sequences are handled directly by _msgspec_ if possible, otherwise by the normal _cattrs_ machinery.
+This means it's possible the validation errors produced may be _msgspec_ validation errors instead of _cattrs_ validation errors.
+
+This converter supports {meth}`get_loads_hook() <cattrs.preconf.msgspec.MsgspecJsonConverter.get_loads_hook>` and {meth}`get_dumps_hook() <cattrs.preconf.msgspec.MsgspecJsonConverter.get_loads_hook>`.
+These are factories for dumping and loading functions (as opposed to unstructuring and structuring); the hooks returned by this may be further optimized to offload as much work as possible to _msgspec_.
+
+```python
+>>> from cattrs.preconf.msgspec import make_converter
+
+>>> @define
+... class Test:
+...     a: int
+
+>>> converter = make_converter()
+>>> dumps = converter.get_dumps_hook(A)
+
+>>> dumps(Test(1))  # Will use msgspec directly.
+b'{"a":1}'
+```
+
+Due to its complexity, this converter is currently _provisional_ and may slightly change as the best integration patterns are discovered.
+
+_msgspec_ doesn't support PyPy.
+
+```{versionadded} 24.1.0
+
+```
+
+## _ujson_
+
+Found at {mod}`cattrs.preconf.ujson`.
+
+Bytes are serialized as base 85 strings. Sets are serialized as lists, and deserialized back into sets. `datetime` s and `date` s are serialized as ISO 8601 strings.
+
+_ujson_ doesn't support integers less than -9223372036854775808, and greater than 9223372036854775807, nor does it support `float('inf')`.
+
+
 ## _msgpack_
 
 Found at {mod}`cattrs.preconf.msgpack`.
@@ -90,10 +139,6 @@ When parsing msgpack data from bytes, the library needs to be passed `strict_map
 
 ## _cbor2_
 
-```{versionadded} 23.1.0
-
-```
-
 Found at {mod}`cattrs.preconf.cbor2`.
 
 _cbor2_ implements a fully featured CBOR encoder with several extensions for handling shared references, big integers, rational numbers and so on.
@@ -112,6 +157,9 @@ Use keyword argument `canonical=True` for efficient encoding to the smallest bin
 Floats can be forced to smaller output by casting to lower-precision formats by casting to `numpy` floats (and back to Python floats).
 Example: `float(np.float32(value))` or `float(np.float16(value))`
 
+```{versionadded} 23.1.0
+
+```
 
 ## _bson_
 

pdm.lock

@@ -94,6 +94,9 @@ cbor2 = [
 bson = [
     "pymongo>=4.4.0",
 ]
+msgspec = [
+    "msgspec>=0.18.5",
+]
 
 [tool.pytest.ini_options]
 addopts = "-l --benchmark-sort=fullname --benchmark-warmup=true --benchmark-warmup-iterations=5  --benchmark-group-by=fullname"

pyproject.toml

@@ -1,5 +1,4 @@
 from cattrs.gen import (
-    AttributeOverride,
     make_dict_structure_fn,
     make_dict_unstructure_fn,
     make_hetero_tuple_unstructure_fn,
@@ -8,6 +7,7 @@
     make_mapping_unstructure_fn,
     override,
 )
+from cattrs.gen._consts import AttributeOverride
 
 __all__ = [
     "AttributeOverride",

src/cattr/gen.py

@@ -750,12 +750,10 @@ def _get_dis_func(
     ) -> Callable[[Any], type]:
         """Fetch or try creating a disambiguation function for a union."""
         union_types = union.__args__
-        if NoneType in union_types:  # type: ignore
+        if NoneType in union_types:
             # We support unions of attrs classes and NoneType higher in the
             # logic.
-            union_types = tuple(
-                e for e in union_types if e is not NoneType  # type: ignore
-            )
+            union_types = tuple(e for e in union_types if e is not NoneType)
 
         # TODO: technically both disambiguators could support TypedDicts and
         # dataclasses...

src/cattrs/converters.py

@@ -166,7 +166,9 @@ def make_dict_unstructure_fn(
                             # type of the default to dispatch on.
                             t = a.default.__class__
                         try:
-                            handler = converter._unstructure_func.dispatch(t)
+                            handler = converter.get_unstructure_hook(
+                                t, cache_result=False
+                            )
                         except RecursionError:
                             # There's a circular reference somewhere down the line
                             handler = converter.unstructure
@@ -293,9 +295,6 @@ def make_dict_structure_fn(
             mapping = generate_mapping(base, mapping)
             break
 
-    if isinstance(cl, TypeVar):
-        cl = mapping.get(cl.__name__, cl)
-
     cl_name = cl.__name__
     fn_name = "structure_" + cl_name
 
@@ -677,7 +676,7 @@ def make_iterable_unstructure_fn(
         # We don't know how to handle the TypeVar on this level,
         # so we skip doing the dispatch here.
         if not isinstance(type_arg, TypeVar):
-            handler = converter._unstructure_func.dispatch(type_arg)
+            handler = converter.get_unstructure_hook(type_arg, cache_result=False)
 
     globs = {"__cattr_seq_cl": unstructure_to or cl, "__cattr_u": handler}
     lines = []
@@ -706,7 +705,8 @@ def make_hetero_tuple_unstructure_fn(
 
     # We can do the dispatch here and now.
     handlers = [
-        converter._unstructure_func.dispatch(type_arg) for type_arg in type_args
+        converter.get_unstructure_hook(type_arg, cache_result=False)
+        for type_arg in type_args
     ]
 
     globs = {f"__cattr_u_{i}": h for i, h in enumerate(handlers)}
@@ -761,11 +761,11 @@ def make_mapping_unstructure_fn(
             # Probably a Counter
             key_arg, val_arg = args, Any
         # We can do the dispatch here and now.
-        kh = key_handler or converter._unstructure_func.dispatch(key_arg)
+        kh = key_handler or converter.get_unstructure_hook(key_arg, cache_result=False)
         if kh == identity:
             kh = None
 
-        val_handler = converter._unstructure_func.dispatch(val_arg)
+        val_handler = converter.get_unstructure_hook(val_arg, cache_result=False)
         if val_handler == identity:
             val_handler = None
 
@@ -833,11 +833,11 @@ def make_mapping_structure_fn(
         is_bare_dict = val_type is Any and key_type is Any
         if not is_bare_dict:
             # We can do the dispatch here and now.
-            key_handler = converter.get_structure_hook(key_type)
+            key_handler = converter.get_structure_hook(key_type, cache_result=False)
             if key_handler == converter._structure_call:
                 key_handler = key_type
 
-            val_handler = converter.get_structure_hook(val_type)
+            val_handler = converter.get_structure_hook(val_type, cache_result=False)
             if val_handler == converter._structure_call:
                 val_handler = val_type