Need more details of annotation collection

[handrews]

There should be pseudocode and a sample/recommend output format, as there is with Hyper-Schema. This should include guidance on how annotation collection can be implemented as part of the same evaluation process as validation.

• Should explain that annotations are dropped whenever a validation check fails; the lack of annotations from not and non-validating branches of oneOf, if, etc. is a natural consequence of that
• Should cover how to indicate the schema from which the annotation was contributed- see example below for concerns around $ref
• Should note that there are both static annotations (all of the ones in the validation spec) and dynamic annotations that change based on the instance's contents (links and base in the hyper-schema spec)
• Should perhaps give some guidance around complex object annotations (e.g. links): how do they fit into the overall JSON Schema model? Are there any particular limitations?
• Should provide guidance on whether keywords can only register annotations under the keyword name or can define different annotation names. The use case for different names would include defining an annotation to which multiple keywords contribute, and wanting to avoid associating the annotation with one specific keyword

Some of the above may get split out into their own issues.

The following example shows the motivation for indicating where we were in the schema when the annotation was contributed. A common request is to be able to figure out which title or description to use, with a typical desired heuristic is that values in parent schemas are preferred over values in child schemas. Note that "child" in this cases is what me might call the logical child, following $refs, instead of the physical child (a nested subschema within the current schema object).

So let's have a schema:

{
    "type": "object",
    "properties": {
        "foo": {
            "title": "Title adjacent to reference",
            "description": "Lots of text",
            "readOnly": true,
            "$ref": "#/definitions/fooDef"
        }
    },
    "definitions": {
        "fooDef": {
            "title": "Title in reference target",
            "description": "Even more text"
        }
    }
}

With an instance of:

{
    "foo": "whatever"
}

The annotation results might look like this (I literally made up how $ref appears in the JSON Pointer-ish thing while typing this example, so don't take that as set in stone):

{
    "/foo": {
        "title": {
            "/properties/foo/$ref/title": "Title in reference target",
            "/properties/foo/title": "Title adjacent to reference"
        },
        "description": {
            "/properties/foo/$ref/description": "Even more text",
            "/properties/foo/description": "Lots of text"
        },
        "readOnly": true
    }
}

"/foo" is the instance pointer- this is the place in the instance to which the annotations apply.

Under that, the property names are the names of each annotation. All of these are also the keyword names that produced them, but in theory you could do something that doesn't map directly (please allow me to wave my hands on that- it's related to a complicated proposal that's under discussion and may not end up being relevant).

Under each annotation, the format is determined by the annotation.

readOnly just has a boolean, because it's explicitly stated (as of draft-07) that the overall value of readOnly is a logical OR of all applicable values. So there's no need to keep track of which value came from where- just let the application know the final outcome.

On the other hand, the way "title" and "description" are shown here is probably the default behavior. For those, "/properties/foo/... are schema pointers, more or less. Pointing across $ref is a bit weird and I need to figure out if that's the right approach. But the idea is to be able to tell that one of those is a runtime-parent of the other, and allow applications to make a decision based on that.

It might be better to use an array of terms instead of a not-quite-JSON Pointer. And it's not yet clear to me whether the keyword itself should be at the end of the pointer-ish thing / array. That also has to do with how much we want to abstract away the keyword source of an annotation. In the case of something like title or readOnly, we really do just want literally what the keyword says.

[epoberezkin]

Are collected annotations defined if the validation of the instance fails? Or because the whole schema fails, they should all be dropped (they are defined but empty)?

[handrews]

@epoberezkin I think the conceptual algorithm is something like:

• Recurse down through applicators
• Evaluate assertions
  • If any assertion fails, the only result is the assertion failure
  • Otherwise, collect annotations and return assertion success + annotations
• As you go back up through parent schemas, if any assertion fails, drop all annotations from that schema and all subschemas

So the reason that you never get annotations from not is that either:

• The not subschema failed an assertion, so all of its annotations and its subschema annotations are dropped before the result is negated to a validation pass
• The not subschema passed, but then not negates that result to a fail, and therefore drops all annotations from its subschema

Of course, in practice you would probably check assertions like type before recursing down through applicators in order to avoid evaluating subschemas when not needed.

But I think if we conceptualize this as collecting annotations as we work from "leaf" schemas up to the root schema, and dropping those annotations whenever we reach an assertion failure, that's the right simplified mental model.

I like it a lot more than "collect annotations except under keywords X, Y, Z"

Does that make sense?

[handrews]

@epoberezkin I had not thought about whether dropping means "defined but empty" or "not present at all". I would lean towards "not present at all". If you don't have annotations, you shouldn't know that you could maybe have had annotations. That's a gut reaction, though, I need to think on exactly why I think that's the correct behavior.

[gregsdennis]

@handrews I agree with @epoberezkin on the "defined but empty" part. It makes more sense to me that an empty array says "evaluation completed without errors, but nothing was found."

Also, as a very minor point, JSONPath contains verbiage that allows for an empty array to be returned in the case that nothing was found, but the primary return in these cases is false. I used the empty array because of the reason above.

Still, either way (null or []) is fine with me.

[handrews]

@gregsdennis, if I'm reading @epoberezkin correctly, the only time you have these placeholder values is if annotation values were collected, but were dropped due to a schema failure. What this would give you is the ability to determine whether information was collected and then revoked as not relevant, vs simply never having been collected.

What is the use case for that distinction?

In my view, the fact that a schema that failed validation happened to have some annotations in it is an implementation detail that should not be leaked. Dropping annotations due to a failed assertion means dropping them as if they were never collected.

Consider the following schema:

{
    "allOf": [
        {"type": "array", "minItems": 10, "description": "foo"},
        {"oneOf": [A, B, C]}
    ]
}

where A, B, and C are complex subschemas with various annotations.

Consider the case where the first thing that is checked is "minItems": 10. Ideally, we do not bother to process the oneOf or any of its subschemas. We already know that validation has failed, and therefore no annotations (not "description": "foo" and not anything in any of the oneOf subschemas) are relevant.

However, if "dropping" annotations means keeping the annotation name but clearing the data to null, [], or {}, then we are forced to process everything, collect all possible annotations, and then clear out the data as it becomes invalid.

Why do all of that extra work? What benefit does it bring?

Plus, in general with JSON (and JavaScript, but the important part is JSON), things that aren't relevant are usually simply not present. In JSON Schema, absent keywords never cause validation to fail, and there are no required placeholder values. That is the general philosophy that JSON Schema, as a JSON-based system, should follow.

[gregsdennis]

@handrews I see the distinction and your confusion.

I don't mean that we should continue processing annotations if we know validation will fail, only to remove those annotations. I'm simply talking about the return value. It makes more sense to me to return an empty array (or object if your example is to be followed) when there are no annotations rather than to return null.

If I know at some point during validation that it will fail, I can still shortcut collection in my implementation and return the empty array/object.

Coming from a strongly typed language, it's nice for me to know what type of entity I'm going to get. It's a question of getting JSON object vs JSON null (which is very different than .Net null).

[epoberezkin]

@handrews another question is whether you've considered collecting all keywords in the passed (sub)schemas as annotations. i.e., when the validation fails, it's usually obvious how it did. When it passed, it's usually not as obvious which branch made it to pass. In theory, while only some keywords are assertions, all keywords can be seen as annotations (e.g. required) and knowing which of them have been used to pass the validation can be useful.

The question is prompted by this: ajv-validator/ajv#661
TLDR: how from the schema and the data we know what are required fields when 1) the data passes validation 2) fails validation (that's why I asked whether some annotations can be returned when validation fails, but it probably complicates things).

[Relequestual]

@epoberezkin I think that's interesting, but a separate issue.
I think we should move forward with this issue as explained in the opening explanation.

[handrews]

@gregsdennis I think that the question of how to represent an empty set of annotations is mostly an implementation-specific thing.

Following the conventions of JSON (which come from JavaScript), it is preferable for something to be absent than to be a literal JSON null. This is how all JSON Schema keywords work. And other JSON technologies like JSON Merge Patch make use of the relative rarity of null in JSON.

However, when it comes to working with schemas and schema results in-memory, an implementation may choose to represent that absence however it wants, and should do so in a way that is as idiomatic as possible for the implementation language. Maybe that means using a .NET null? I'm very unfamiliar with .NET, and last looked at it, um... 15 years ago? And not in depth then :-D

[EDIT: I realize the I misread the question and it is about using an empty data structure]

As far as an empty data structure, I'll have to think on that. It might be reasonable, or it might be that strongly typed languages should just use an empty data structure where other languages may not use anything. It's so variable- in JavaScript empty data structures are truth-y, but in Python they are false-y.

Ideally JSON Schema is idiomatic JSON, and in-memory representations are as close to idiomatic for that language as is practical.