Skip to content

Commit 34d19dc

Browse files
JulianJulian
committed
Add some prose documentation on the new referencing API.
1 parent fcea5ad commit 34d19dc

File tree

6 files changed

+322
-54
lines changed

6 files changed

+322
-54
lines changed

‎docs/faq.rst

+1-51
Original file line numberDiff line numberDiff line change
@@ -88,6 +88,7 @@ The JSON object ``{}`` is simply the Python `dict` ``{}``, and a JSON Schema lik
8888
Specifically, in the case where `jsonschema` is asked to resolve a remote reference, it has no choice but to assume that the remote reference is serialized as JSON, and to deserialize it using the `json` module.
8989

9090
One cannot today therefore reference some remote piece of YAML and have it deserialized into Python objects by this library without doing some additional work.
91+
See `Resolving References to Schemas Written in YAML <referencing:Resolving References to Schemas Written in YAML>` for details.
9192

9293
In practice what this means for JSON-like formats like YAML and TOML is that indeed one can generally schematize and then validate them exactly as if they were JSON by simply first deserializing them using libraries like ``PyYAML`` or the like, and passing the resulting Python objects into functions within this library.
9394

@@ -99,57 +100,6 @@ In such cases one is recommended to first pre-process the data such that the res
99100
In the previous example, if the desired behavior is to transparently coerce numeric properties to strings, as Javascript might, then do the conversion explicitly before passing data to this library.
100101

101102

102-
How do I configure a base URI for $ref resolution using local files?
103-
--------------------------------------------------------------------
104-
105-
`jsonschema` supports loading schemas from the filesystem.
106-
107-
The most common mistake when configuring reference resolution to retrieve schemas from the local filesystem is to specify a base URI which points to a directory, but forget to add a trailing slash.
108-
109-
For example, given a directory ``/tmp/foo/`` with ``bar/schema.json`` within it, you should use something like:
110-
111-
.. code-block:: python
112-
113-
from pathlib import Path
114-
115-
import jsonschema.validators
116-
117-
path = Path("/tmp/foo")
118-
resolver = jsonschema.validators.RefResolver(
119-
base_uri=f"{path.as_uri()}/",
120-
referrer=True,
121-
)
122-
jsonschema.validate(
123-
instance={},
124-
schema={"$ref": "bar/schema.json"},
125-
resolver=resolver,
126-
)
127-
128-
where note:
129-
130-
* the base URI has a trailing slash, even though
131-
`pathlib.PurePath.as_uri` does not add it!
132-
* any relative refs are now given relative to the provided directory
133-
134-
If you forget the trailing slash, you'll find references are resolved a
135-
directory too high.
136-
137-
You're likely familiar with this behavior from your browser. If you
138-
visit a page at ``https://example.com/foo``, then links on it like
139-
``<a href="./bar">`` take you to ``https://example.com/bar``, not
140-
``https://example.com/foo/bar``. For this reason many sites will
141-
redirect ``https://example.com/foo`` to ``https://example.com/foo/``,
142-
i.e. add the trailing slash, so that relative links on the page will keep the
143-
last path component.
144-
145-
There are, in summary, 2 ways to do this properly:
146-
147-
* Remember to include a trailing slash, so your base URI is
148-
``file:///foo/bar/`` rather than ``file:///foo/bar``, as shown above
149-
* Use a file within the directory as your base URI rather than the
150-
directory itself, i.e. ``file://foo/bar/baz.json``, which will of course
151-
cause ``baz.json`` to be removed while resolving relative URIs
152-
153103
Why doesn't my schema's default property set the default on my instance?
154104
------------------------------------------------------------------------
155105

‎docs/index.rst

+1
Original file line numberDiff line numberDiff line change
@@ -12,6 +12,7 @@ Contents
1212

1313
validate
1414
errors
15+
referencing
1516
creating
1617
faq
1718
api/index

‎docs/referencing.rst

+314
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
=========================
2+
JSON (Schema) Referencing
3+
=========================
4+
5+
The JSON Schema :kw:`$ref` and :kw:`$dynamicRef` keywords allow schema authors to combine multiple schemas (or subschemas) together for reuse or deduplication.
6+
7+
The `referencing <referencing:index>` library was written in order to provide a simple, well-behaved and well-tested implementation of this kind of reference resolution [1]_.
8+
It has its own documentation, but this page serves as a quick introduction which is tailored more specifically to JSON Schema, and even more specifically to how to configure `referencing <referencing:index>` for use with `Validator` objects in order to customize the behavior of :kw:`$ref` and friends in your schemas.
9+
10+
Configuring `jsonschema` for custom referencing behavior is essentially a two step process:
11+
12+
* Create a `referencing.Registry` object that behaves the way you wish
13+
14+
* Pass the `referencing.Registry` to your `Validator` when instantiating it
15+
16+
The examples below essentially follow these two steps.
17+
18+
.. [1] One that in fact is independent of this `jsonschema` library itself, and may some day be used by other tools or implementations.
19+
20+
21+
Common Scenarios
22+
----------------
23+
24+
.. _in-memory-schemas:
25+
26+
Making Additional In-Memory Schemas Available
27+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
28+
29+
The most common scenario one is likely to encounter is the desire to include a small number of additional in-memory schemas, making them available for use during validation.
30+
31+
For instance, imagine the below schema for non-negative integers:
32+
33+
.. code:: json
34+
35+
{
36+
"$schema": "https://json-schema.org/draft/2020-12/schema",
37+
"type": "integer",
38+
"minimum": 0
39+
}
40+
41+
We may wish to have other schemas we write be able to make use of this schema, and refer to it as ``http://example.com/nonneg-int-schema`` and/or as ``urn:nonneg-integer-schema``.
42+
43+
To do so we make use of APIs from the referencing library to create a `referencing.Registry` which maps the URIs above to this schema:
44+
45+
.. code:: python
46+
47+
from referencing import Registry, Resource
48+
schema = Resource.from_contents(
49+
{
50+
"$schema": "https://json-schema.org/draft/2020-12/schema",
51+
"type": "integer",
52+
"minimum": 0,
53+
},
54+
)
55+
registry = Registry().with_resources(
56+
[
57+
("http://example.com/nonneg-int-schema", schema),
58+
("urn:nonneg-integer-schema", schema),
59+
],
60+
)
61+
62+
What's above is likely mostly self-explanatory, other than the presence of the `referencing.Resource.from_contents` function.
63+
Its purpose is to convert a piece of "opaque" JSON (or really a Python `dict` containing deserialized JSON) into an object which indicates what *version* of JSON Schema the schema is meant to be interpreted under.
64+
Calling it will inspect a :kw:`$schema` keyword present in the given schema and use that to associate the JSON with an appropriate `specification <referencing.Specification>`.
65+
If your schemas do not contain ``$schema`` dialect identifiers, and you intend for them to be interpreted always under a specific dialect -- say Draft 2020-12 of JSON Schema -- you may instead use e.g.:
66+
67+
.. code:: python
68+
69+
from referencing import Registry, Resource
70+
from referencing.jsonschema import DRAFT2020212
71+
schema = DRAFT202012.create_resource({"type": "integer", "minimum": 0})
72+
registry = Registry().with_resources(
73+
[
74+
("http://example.com/nonneg-int-schema", schema),
75+
("urn:nonneg-integer-schema", schema),
76+
],
77+
)
78+
79+
which has the same functional effect.
80+
81+
You can now pass this registry to your `Validator`, which allows a schema passed to it to make use of the aforementioned URIs to refer to our non-negative integer schema.
82+
Here for instance is an example which validates that instances are JSON objects with non-negative integral values:
83+
84+
.. code:: python
85+
86+
from jsonschema import Draft202012Validator
87+
validator = Draft202012Validator(
88+
{
89+
"type": "object",
90+
"additionalProperties": {"$ref": "urn:nonneg-integer-schema"},
91+
},
92+
registry=registry, # the critical argument, our registry from above
93+
)
94+
validator.validate({"foo": 37})
95+
validator.validate({"foo": -37}) # Uh oh!
96+
97+
.. _ref-filesystem:
98+
99+
Resolving References from the File System
100+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
101+
102+
Another common request from schema authors is to be able to map URIs to the file system, perhaps while developing a set of schemas in different local files.
103+
The referencing library supports doing so dynamically by configuring a callable which can be used to retrieve any schema which is *not* already pre-loaded in the manner described `above <in-memory-schemas>`.
104+
105+
Here we resolve any schema beginning with ``http://localhost`` to a directory ``/tmp/schemas`` on the local filesystem (note of course that this will not work if run directly unless you have populated that directory with some schemas):
106+
107+
.. code:: python
108+
109+
from pathlib import Path
110+
import json
111+
112+
from referencing import Registry, Resource
113+
from referencing.exceptions import NoSuchResource
114+
115+
SCHEMAS = Path("/tmp/schemas")
116+
117+
def retrieve_from_filesystem(uri: str):
118+
if not uri.startswith("http://localhost/"):
119+
raise NoSuchResource(ref=uri)
120+
path = SCHEMAS / Path(uri.removeprefix("http://localhost/"))
121+
contents = json.loads(path.read_text())
122+
return Resource.from_contents(contents)
123+
124+
registry = Registry(retrieve=retrieve_from_filesystem)
125+
126+
Such a registry can then be used with `Validator` objects in the same way shown above, and any such references to URIs which are not already in-memory will be retrieved from the configured directory.
127+
128+
We can mix the two examples above if we wish for some in-memory schemas to be available in addition to the filesystem schemas, e.g.:
129+
130+
.. code:: python
131+
132+
from referencing.jsonschema import DRAFT7
133+
registry = Registry(retrieve=retrieve_from_filesystem).with_resource(
134+
"urn:non-empty-array", DRAFT7.create_resource({"type": "array", "minItems": 1}),
135+
)
136+
137+
where we've made use of the similar `referencing.Registry.with_resource` function to add a single additional resource.
138+
139+
Resolving References to Schemas Written in YAML
140+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
141+
142+
Generalizing slightly, the retrieval function provided need not even assume that it is retrieving JSON.
143+
As long as you deserialize what you have retrieved into Python objects, you may equally be retrieving references to YAML documents or any other format.
144+
145+
Here for instance we retrieve YAML documents in a way similar to the `above <ref-filesystem>` using PyYAML:
146+
147+
.. code:: python
148+
149+
from pathlib import Path
150+
import yaml
151+
152+
from referencing import Registry, Resource
153+
from referencing.exceptions import NoSuchResource
154+
155+
SCHEMAS = Path("/tmp/yaml-schemas")
156+
157+
def retrieve_yaml(uri: str):
158+
if not uri.startswith("http://localhost/"):
159+
raise NoSuchResource(ref=uri)
160+
path = SCHEMAS / Path(uri.removeprefix("http://localhost/"))
161+
contents = yaml.safe_load(path.read_text())
162+
return Resource.from_contents(contents)
163+
164+
registry = Registry(retrieve=retrieve_yaml)
165+
166+
.. note::
167+
168+
Not all YAML fits within the JSON data model.
169+
170+
JSON Schema is defined specifically for JSON, and has well-defined behavior strictly for Python objects which could have possibly existed as JSON.
171+
172+
If you stick to the subset of YAML for which this is the case then you shouldn't have issue, but if you pass schemas (or instances) around whose structure could never have possibly existed as JSON (e.g. a mapping whose keys are not strings), all bets are off.
173+
174+
One could similarly imagine a retrieval function which switches on whether to call ``yaml.safe_load`` or ``json.loads`` by file extension (or some more reliable mechanism) and thereby support retrieving references of various different file formats.
175+
176+
.. _http:
177+
178+
Automatically Retrieving Resources Over HTTP
179+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
180+
181+
In the general case, the JSON Schema specifications tend to `discourage <https://json-schema.org/draft/2020-12/json-schema-core.html#name-loading-a-referenced-schema>`_ implementations (like this one) from automatically retrieving references over the network, or even assuming such a thing is feasible (as schemas may be identified by URIs which are strictly identifiers, and not necessarily downloadable from the URI even when such a thing is sensical).
182+
183+
However, if you as a schema author are in a situation where you indeed do wish to do so for convenience (and understand the implications of doing so), you may do so by making use of the ``retrieve`` argument to `referencing.Registry`.
184+
185+
Here is how one would configure a registry to automatically retrieve schemas from the `JSON Schema Store <https://www.schemastore.org>`_ on the fly using the `httpx <https://www.python-httpx.org/>`_:
186+
187+
.. code:: python
188+
189+
from referencing import Registry, Resource
190+
import httpx
191+
192+
def retrieve_via_httpx(uri: str):
193+
response = httpx.get(uri)
194+
return Resource.from_contents(response.json())
195+
196+
registry = Registry(retrieve=retrieve_via_httpx)
197+
198+
Given such a registry, we can now, for instance, validate instances against schemas from the schema store by passing the ``registry`` we configured to our `Validator` as in previous examples:
199+
200+
.. code:: python
201+
202+
from jsonschema import Draft202012Validator
203+
Draft202012Validator(
204+
{"$ref": "https://json.schemastore.org/pyproject.json"},
205+
registry=registry,
206+
).validate({"project": {"name": 12}})
207+
208+
which should in this case indicate the example data is invalid:
209+
210+
.. code:: python
211+
212+
Traceback (most recent call last):
213+
File "example.py", line 14, in <module>
214+
).validate({"project": {"name": 12}})
215+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
216+
File "jsonschema/validators.py", line 345, in validate
217+
raise error
218+
jsonschema.exceptions.ValidationError: 12 is not of type 'string'
219+
220+
Failed validating 'type' in schema['properties']['project']['properties']['name']:
221+
{'pattern': '^([a-zA-Z\\d]|[a-zA-Z\\d][\\w.-]*[a-zA-Z\\d])$',
222+
'title': 'Project name',
223+
'type': 'string'}
224+
225+
On instance['project']['name']:
226+
12
227+
228+
Retrieving resources from a SQLite database or some other network-accessible resource should be more or less similar, replacing the HTTP client with one for your database of course.
229+
230+
.. warning::
231+
232+
Be sure you understand the security implications of the reference resolution you configure.
233+
And if you accept untrusted schemas, doubly sure!
234+
235+
You wouldn't want a user causing your machine to go off and retrieve giant files off the network by passing it a ``$ref`` to some huge blob, or exploiting similar vulnerabilities in your setup.
236+
237+
238+
Migrating From ``RefResolver``
239+
------------------------------
240+
241+
Older versions of `jsonschema` used a different object -- `_RefResolver` -- for reference resolution, which you a schema author may already be configuring for your own use.
242+
243+
`_RefResolver` is now fully deprecated and replaced by the use of `referencing.Registry` as shown in examples above.
244+
245+
If you are not already constructing your own `_RefResolver`, this change should be transparent to you (or even recognizably improved, as the point of the migration was to improve the quality of the referencing implementation and enable some new functionality).
246+
247+
If you *were* configuring your own `_RefResolver`, here's how to migrate to the newer APIs:
248+
249+
The ``store`` argument
250+
~~~~~~~~~~~~~~~~~~~~~~
251+
252+
`_RefResolver`\ 's ``store`` argument was essentially the equivalent of `referencing.Registry`\ 's in-memory schema storage.
253+
254+
If you currently pass a set of schemas via e.g.:
255+
256+
.. code:: python
257+
258+
from jsonschema import Draft202012Validator, RefResolver
259+
resolver = RefResolver.from_schema(
260+
schema={"title": "my schema"},
261+
store={"http://example.com": {"type": "integer"}},
262+
)
263+
validator = Draft202012Validator(
264+
{"$ref": "http://example.com"},
265+
resolver=resolver,
266+
)
267+
validator.validate("foo")
268+
269+
you should be able to simply move to something like:
270+
271+
.. code:: python
272+
273+
from referencing import Registry
274+
from referencing.jsonschema import DRAFT202012
275+
276+
from jsonschema import Draft202012Validator
277+
278+
registry = Registry().with_resource(
279+
"http://example.com",
280+
DRAFT202012.create_resource({"type": "integer"}),
281+
)
282+
validator = Draft202012Validator(
283+
{"$ref": "http://example.com"},
284+
registry=registry,
285+
)
286+
validator.validate("foo")
287+
288+
Handlers
289+
~~~~~~~~
290+
291+
The ``handlers`` functionality from `_RefResolver` was a way to support additional HTTP schemes for schema retrieval.
292+
293+
Here you should move to a custom ``retrieve`` function which does whatever you'd like.
294+
E.g. in pseudocode:
295+
296+
.. code:: python
297+
298+
from urllib.parse import urlsplit
299+
300+
def retrieve(uri: str):
301+
parsed = urlsplit(uri)
302+
if parsed.scheme == "file":
303+
...
304+
elif parsed.scheme == "custom":
305+
...
306+
307+
registry = Registry(retrieve=retrieve)
308+
309+
310+
Other Key Functional Differences
311+
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
312+
313+
Whilst `_RefResolver` *did* automatically retrieve remote references (against the recommendation of the spec, and in a way which therefore could lead to questionable security concerns when combined with untrusted schemas), `referencing.Registry` does *not* do so.
314+
If you rely on this behavior, you should follow the `above example of retrieving resources over HTTP <http>`.

0 commit comments

Comments
 (0)