`format` should not change behavior based on its vocabulary value

[gregsdennis]

The true or false value of the vocabulary declaration governs the implementation requirements necessary to process a schema that uses "format", and the behaviors on which schema authors can rely.

This is just wrong. It precludes an ability to assert format when supported but not require assertion when not supported. The spec still allows for configuration of this, but the practical side of that configuration becomes quite difficult and confusing:

• For : false, my configuration needs to say "format should be an assertion."
• For : true, my configuration needs to say "format should be an annotation."

This is hard to do and confusing for clients (users of the implementation). The configuration should always work one direction. If my implementation offers a "format behavior" configuration with values of "assert" and "annotate," setting either value only works for one of the vocab cases. The only way to get the desired behavior is to have a configuration that says "use the non-default behavior," which changes its meaning depending on the schema it's processing.

What we SHOULD have is format as an annotation ALWAYS, but configurable to be assertion.

I agree with @karenetheridge that it'd be nice to have a way for the schema itself to indicate how format should be processed, and I think that changing it based on the vocab value was an attempt at that. But the vocab value and the behavior of format are orthogonal concerns. The spec is conflating them unnecessarily.

[handrews]

I'm not going to get involved in the resolution of this, but just to make sure we're all on the same page (because everything about or related to format is a confusing dumpster fire):

The egregiously complex interplay of the boolean $vocabulary value for format and the configuration of the implementation processing the schema is 100% the result of trying to preserve the prior behavior (as typically implemented rather than as specified) of format. Specifically, that most implementations up through draft-07 do a best-effort validation on format.

The only combination directly related to that is:

• implementation only supports best effort, not full syntax validation (whatever full syntax validation even means, as people argue over that)
• the value for the format vocabulary in $vocabulary is false
• the implementation is configured to support format validation, which is not the default configuration

This allows for best effort validation of format. It is a weird case where, due to the implementation configuration flag, a false value actually sort-of turns on some functionality. This never comes up for any other vocabulary as nothing else has ever specified an implementation flag for functionality control.

What we SHOULD have is format as an annotation ALWAYS, but configurable to be assertion.

@gregsdennis I believe this is the case? The table in the section of the release notes to which I linked above should make that clear, please let me know if there is confusion on this point.

Since we changed the behavior for unrecognized keywords to be "collect as annotation" instead of "ignore", format always behaves as an annotation even if it is not recognized (either not supported, or someone used the keyword without declaring the vocabulary). Declaring the vocabulary with true works like any other vocabulary- in practice it's weird because very few implementations fully support format syntax validation, which is why it's false in the default meta-schema.

Really, if we get rid of the half-ass "best effort" case, then format stops being so damn weird. It would probably also stop being supported, but I would not consider that a bad thing. Either way, it's up to y'all now :-) As everyone's probably tired of hearing by now, I'd have ripped it out entirely and told people to make vocabularies if I thought I could have gotten away with it.

[gregsdennis]

Declaring the vocabulary with true works like any other vocabulary...

But for format (as it's currently written), it doesn't behave like other vocabularies. format explicitly states that it changes from annotation to validation when the vocabulary is required (value is true).

From https://github.com/json-schema-org/json-schema-spec/blob/master/jsonschema-validation.xml#L569

The true or false value of the vocabulary declaration governs the implementation requirements necessary to process a schema that uses "format", and the behaviors on which schema authors can rely.

From https://github.com/json-schema-org/json-schema-spec/blob/master/jsonschema-validation.xml#L599

The assertion evaluation behavior when the option is not explicitly specified depends on the vocabulary declaration's boolean value.

This is what I have a problem with. This implies that format automatically acts as an assertion when the vocab value is true. This is bad. It should always be an annotation unless configured to act as an assertion.

[gregsdennis]

Regarding the (2019-09) table, the rows that concerns me are the

best-effort/full-syntax | default (off) | true | vocabulary error

If the configuration is off but there is some kind of support, annotations should be generated; the implementation shouldn't error.

[gregsdennis]

I'll try to rework things to make the "annotation unless configured" behavior a bit more clear.

[handrews]

But for format (as it's currently written), it doesn't behave like other vocabularies. format explicitly states that it changes from annotation to validation when the vocabulary is required (value is true).

No, it changes from best-effort validation (if available and configured) to complete validation.

If the configuration is off but there is some kind of support, annotations should be generated; the implementation shouldn't error.

The goal was to make true function normally and consistently, so what I wrote is correct.

The config option, which is only present for historical reasons, is only relevant for false. You have to break one or the other: either the config option is ignored for true or format is the one and only vocabulary that doesn't generate an error on true if it is not supported.

In my view ALL of the bizarre inconsistent behavior should be in the backwards compatibility case. Everything else should function normally, so that means that with true the config option is irrelevant. It's just ignored.

Because the semantics of true are this MUST function as validation or else the schema will not function correctly. So forget the configuration on the implementation side. It's not relevant. You can't turn vocabularies off like that, ever, and it would be a really bad idea to set a precedent allowing that.

Implementation-side configuration options are a bad idea in general because they change the behavior of the schema to something other than how it was written. Don't make that problem worse, please. (although if you do I'm not going to do anything about it).

[handrews]

Honestly, the simplest option is to get rid of the legacy bullshit- no config option, and you either implement it fully or you don't. But people will complain about that so you have to decide whether you're ok with it. I'm not sure more people would complain about that than complain about format already TBH.

[gregsdennis]

The config option... is only relevant for false.

My argument is that this is wrong. It's relevant for both true and false.

The problem space has three variables: configuration of the library (assert or annotate), whether the library knows about the vocabulary (yes or no), and the value of the vocabulary entry in $vocabulary (true or false). This gives 8 scenarios.

Here's a new table:

configuration	vocab known	vocab value	result
annotate	yes	false	annotation
annotate	no	false	annotation
annotate	yes	true	annotation
annotate	no	true	error
assert	yes	false	assertion
assert	no	false	assertion
assert	yes	true	assertion
assert	no	true	error

Implementations should be configured to annotate by default.

I don't see how this is affected by a best-effort or full validation. The implementation either supports the keyword or it doesn't. The degree to which it implements the keyword is irrelevant.

In my view ALL of the bizarre inconsistent behavior should be in the backwards compatibility case. Everything else should function normally, so that means that with true the config option is irrelevant. It's just ignored.

How can either the true or false case have been an issue of backward compatibility when $vocabulary was being written? There was no precedence for either value because the keyword didn't exist.

Because the semantics of true are this MUST function as validation or else the schema will not function correctly.

Where is this sematic defined? $vocabulary doesn't define this anywhere:

If the value is true, then implementations that do not recognize the vocabulary MUST refuse to process any schemas that declare this meta-schema with "$schema". If the value is false, implementations that do not recognize the vocabulary SHOULD proceed with processing such schemas.

This is the only text that defines what the behavior should be for a vocabulary with value of true (aside from the previously quoted text for format, which is currently under scrutiny).

---

format is merely an assertion annotation keyword that has a concession that implementations MAY be configured to be an assertion. This configuration is orthogonal to whether the implementation knows about the keyword's vocab or whether the implementation knows about the vocab at all.

[Relequestual]

format is merely an assertion keyword that
@gregsdennis

I assume you meant "format is merely an ANNOTATION keyword that..."?

[Relequestual]

After reading the above, 3 times, and the current preview of the spec format vocabulary, at least 4 times, I'm leaning more towards as it currently stands. Hear me out.

99.99 % of schema authors will never write their own vocabulary or use a non-standard dialect (Bar the OAS dialect).

As such, we should make the upgrade path of previous schemas as easy as possible, and the current docs and articles on JSON Schema as least out of date as possible.

I'm not advocating we should just do things as we've always done because "reasons". That would be bad. However, we should provide minimal friction for users, which is what the current state of 2020-11 preview and 2019-09 affords.

As an aside, 2019-09 is over a year old, and NO ONE has asked about this. Not one person has come and asked us, at least.

As such, we SHOULD keep the status quo for now, and tackle format all in one go. In the next draft.

It SOUNDS confusing from an implementation perspective, but from a schema author perspective, it actually makes a lot of sense.

---

format, now (2019-09 and 2020-11-rc-0), by default is an annotation only. If I want SOME best effort validation, I can enable it using some kind of implementation option.
I can post-process the annotations to do validation specific to me if I want.
I haven't had to create a new dialect or meta-schema.
No more surprises.

If I want others using my schema to always use format as an assertion, and perform "FULL" validation, I create a new dialect which sets the format vocab to true. An implementation receiving that schema (with a new dialect), now knows it MUST perform "FULL" validation if it can, or throw an error. The config option is moot because the schema author has defined "this is really really required, so just do it or else".
No more surprises.

The goal is no more surprises (removing ambiguity) and ease of use, and I think that was met in the approach laid out, as per 2019-09, and as per 2020-11 preview.

---

Remember, the vocabulary object boolean doesn't enable or disable a vocabulary, but asserts if the implementation MUST understand the vocabulary in order to correctly process the schema. If it's set to false, it MAY still process using the vocabulary if it understands it.

The config option is really a sticky plaster over a problem we need to solve, so, let's assume we're going to remove it at some point.

With the config option removed, a vocabulary object boolean of true for format is clear, do the really good validation, or die. However, if it's false, you MIGHT get just annotations (because that's the default behaviour of all keywords), or you MIGHT get full validation... This is now "ta-da surprise" land, and we've established this is bad for everyone.

You could argue that in such an event as we properly define format, that it should be default be enabled, and YES I think we all agree... buttttt it's super hard to agree on exactly how that sort of validation works. I mean look at regex for email... (Some would argue it's easy, others not).

So, given we can't fully qualify the exact validation for all the format values right now, and we want to avoid surprises, can you agree that the current situation is best-effort given constraints and logically derived?

[gregsdennis]

99.99 % of schema authors will never write their own vocabulary or use a non-standard dialect (Bar the OAS dialect).

99.99% of schema authors aren't even using 2019-09 (all the questions I see have draft 7 schemas, despite the suggestions to use 2019-09, and only a handful of validators even support it; even VS Code doesn't support it), and the ones that do aren't using format, so they have no set expectations around the vocab being true influencing the behavior of format. For them, vocabs don't exist, and this is a new concept. There are no questions about how format works in 2019-09 because no one is using 2019-09.

We need to fix this now when adoption is at its lowest; it'll be much harder to fix later.

With the config option removed, a vocabulary object boolean of true for format is clear, do the really good validation, or die.

Why do you assume that the default behavior for format should be assertion without the configuration option? My proposal is to leave the vocab true/false meaning intact and maintain format as an annotation, just like any other annotation keyword (e.g. title). The option leaves an opening for implementations to be configured (separately from the schema) to treat it as an assertion completely separately from the vocab value.

If you want the schema to specify whether it should be an assertion...

If I want others using my schema to always use format as an assertion, and perform "FULL" validation, I create a new dialect which sets the format vocab to true.

This is the wrong way to do this. (I described it in the slack thread.) If you want the schema to be able to specify that format is an assertion, that's fine, but conflating the meaning of specifying a vocab as true is the wrong way to do it. (e.g. Another option is the introduction of a new "configuration" type of keyword that can set this behavior.) Furthermore it sets a precedence that this is an okay thing to do, suggesting that other vocab authors may do the same. That is bad.

The current definition conflates two separate issues: vocab value meaning and configurability. These ideas need to stay separate.

The goal is no more surprises (removing ambiguity) and ease of use, and I think that was met in the approach laid out, as per 2019-09, and as per 2020-11 preview.

"No more surprises" would mean that format isn't special. This whole dependent behavior is a hard gotcha. It's the only keyword that behaves this way, and we will be better off (now and in the long run) aligning it with the behavior of the other keywords.

"Ease of use" would mean that this keyword behaves like any other keyword with respect to the vocab value.

you MIGHT get just annotations (because that's the default behaviour of all keywords), or you MIGHT get full validation... This is now "ta-da surprise" land, and we've established this is bad for everyone.

This is what documentation is for. Literally no one is going to have 100% validation on all of these keywords. No one can. There will always be edge cases, no matter how strengently a format validation is coded. You are therefore relegated to mandating that everyone documenting their level of support.

Secondarily, if you must configure the implementation separately, you'll never have that "ta-da suprise" moment because you have to configure it properly. You'll never get assertions when you want annotation, and vice versa. As it's written, that could happen. As it's written, I can't require format annotations or opt into format assertion. As it's written I have two options: MAYBE get annotations (which I can look up support for format in the docs) or DEFINITELY get format assertions.

This is the gotcha that we have to remove. My PR does exactly that. With my PR, you can opt in to whether you get assertion or annotation behavior through configuration, and the schema specifies whether format support (at all) is a hard or soft requirement.