Skip to content

Commit 1ed8b65

Browse files
committed
Make JSON schema available for verification under https:// URIs
After updating gojsonschema to include xeipuuv/gojsonschema#171 , tests fail with > unable to validate: Could not read schema from HTTP, response status is 404 Not Found Before that gojsonschema change, "$ref" links were interpreted by taking the current schema source file's URI as a base, and treating "$ref" as relative to this. For example, starting with the [file://]/image-manifest-schema.json URI, as used by Validator.Validate (based on the "specs" map), the > "$ref": "content-descriptor.json" reference used to evaluate to file:///content-descriptor.json. gojsonschema.jsonReferenceLoader would then load these file:///*.json URIs via _escFS. After the gojsonschema change, "$ref" links are evaluated relative to a URI base specified by the "id" attribute inside the schema source, regardless of the "external" URI passed to the gojsonschema.JSONLoader. This is consistent with http://json-schema.org/latest/json-schema-core.html#rfc.section.8 and http://json-schema.org/latest/json-schema-core.html#rfc.section.9.2 (apart from the "id" vs. "$id" attribute name). In the same example, [file://]/image-manifest-schema.json URI contains > "id": "https://opencontainers.org/schema/image/manifest", so the same > "$ref": "content-descriptor.json" now evaluates to "https://opencontainers.org/schema/image/content-descriptor.json", which is not found by gojsonschema.jsonReferenceLoader (it uses _escFS only for file:/// URIs), resulting in the 404 quoted above. This is a minimal fix, making the schema files available to gojsonschema at the https:// URIs, while continuing to read them from _escFS. Because gojsonschema.jsonReferenceLoader can only use the provided fs for file:/// URIs, we are forced to implement our own gojsonschema.JSONLoaderFactory and gojsonschema.JSONLoader; something like this might be more generally useful and should therefore instead be provided by the gojsonschema library. This particular JSONLoader{Factory,} implementation, though, is image-spec specific because it locally works around various inconsistencies in the image-spec JSON schemas, and thus is not suitable for gojsonschema as is. Namely, the specs/*.json schema files use URIs with two URI path prefixes, https://opencontainers.org/schema/{,image/} in the top-level "id" attributes, and the nested "id" attributes along with "$ref" references use _several more_ URI path prefixes, e.g. > "id": "https://opencontainers.org/schema/image/manifest/annotations", > "$ref": "defs-descriptor.json#/definitions/annotations" in image-manifest-schema.json specifies the https://opencontainers.org/schema/image/manifest/defs-descriptor.json URI. In fact, defs-descriptor.json references use all of the following URIs: > https://opencontainers.org/schema/defs-descriptor.json > https://opencontainers.org/schema/image/defs-descriptor.json > https://opencontainers.org/schema/image/descriptor/defs-descriptor.json > https://opencontainers.org/schema/image/index/defs-descriptor.json > https://opencontainers.org/schema/image/manifest/defs-descriptor.json So, this commit introduces a loader which preserves the original _escFS layout by recognizing and stripping all of these prefixes, and using the same /*.json paths for _escFS lookups as before; this is clearly unsuitable for gojsonschema inclusion. Finally, the reason this commit uses such a fairly hacky loader is that merely changing the _escFS structure is still not sufficient to get consistent schema: the schema/*.json paths in this repository, and the "$ref" values, do not match the "id" values inside the schemas at all. E.g. image-manifest-schema.json refers to https://opencontainers.org/schema/image/manifest/content-descriptor.json , while content-descriptor.json identifies itself as https://opencontainers.org/schema/descriptor , matching neither the path prefix nor the file name. Overall, it is completely unclear to me which of the URIs is the canonical URI of the "content descriptor" schema, and the owner of the URI namespace needs to decide on the canonical schema URIs. Only afterwards can the code be cleanly modified to match the specification; until then, this commit at least keeps the tests passing, and the validator usable by external callers who want to use the public image-spec/schema.ValidateMediaType*.Validate() API. Signed-off-by: Miloslav Trmač <[email protected]>
1 parent f2b7079 commit 1ed8b65

File tree

3 files changed

+157
-7
lines changed

3 files changed

+157
-7
lines changed

schema/loader.go

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
// Copyright 2018 The Linux Foundation
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
package schema
16+
17+
import (
18+
"bytes"
19+
"encoding/json"
20+
"fmt"
21+
"io"
22+
"io/ioutil"
23+
"net/http"
24+
"strings"
25+
26+
"github.com/xeipuuv/gojsonreference"
27+
"github.com/xeipuuv/gojsonschema"
28+
)
29+
30+
// fsLoaderFactory implements gojsonschema.JSONLoaderFactory by reading files under the specified namespaces from the root of fs.
31+
type fsLoaderFactory struct {
32+
namespaces []string
33+
fs http.FileSystem
34+
}
35+
36+
// newFSLoaderFactory returns a fsLoaderFactory reading files under the specified namespaces from the root of fs.
37+
func newFSLoaderFactory(namespaces []string, fs http.FileSystem) *fsLoaderFactory {
38+
return &fsLoaderFactory{
39+
namespaces: namespaces,
40+
fs: fs,
41+
}
42+
}
43+
44+
func (factory *fsLoaderFactory) New(source string) gojsonschema.JSONLoader {
45+
return &fsLoader{
46+
factory: factory,
47+
source: source,
48+
}
49+
}
50+
51+
// refContents returns the contents of ref, if available in fsLoaderFactory.
52+
func (factory *fsLoaderFactory) refContents(ref gojsonreference.JsonReference) ([]byte, error) {
53+
refStr := ref.String()
54+
path := ""
55+
for _, ns := range factory.namespaces {
56+
if strings.HasPrefix(refStr, ns) {
57+
path = "/" + strings.TrimPrefix(refStr, ns)
58+
break
59+
}
60+
}
61+
if path == "" {
62+
return nil, fmt.Errorf("Schema reference %#v unexpectedly not available in fsLoaderFactory with namespaces %#v", path, factory.namespaces)
63+
}
64+
65+
f, err := factory.fs.Open(path)
66+
if err != nil {
67+
return nil, err
68+
}
69+
defer f.Close()
70+
71+
return ioutil.ReadAll(f)
72+
}
73+
74+
// fsLoader implements gojsonschema.JSONLoader by reading the document named by source from a fsLoaderFactory.
75+
type fsLoader struct {
76+
factory *fsLoaderFactory
77+
source string
78+
}
79+
80+
// JsonSource implements gojsonschema.JSONLoader.JsonSource. The "Json" capitalization needs to be maintained to conform to the interface.
81+
func (l *fsLoader) JsonSource() interface{} { // nolint: golint
82+
return l.source
83+
}
84+
85+
func (l *fsLoader) LoadJSON() (interface{}, error) {
86+
// Based on gojsonschema.jsonReferenceLoader.LoadJSON.
87+
reference, err := gojsonreference.NewJsonReference(l.source)
88+
if err != nil {
89+
return nil, err
90+
}
91+
92+
refToURL := reference
93+
refToURL.GetUrl().Fragment = ""
94+
95+
body, err := l.factory.refContents(refToURL)
96+
if err != nil {
97+
return nil, err
98+
}
99+
100+
return decodeJSONUsingNumber(bytes.NewReader(body))
101+
}
102+
103+
// decodeJSONUsingNumber returns JSON parsed from an io.Reader
104+
func decodeJSONUsingNumber(r io.Reader) (interface{}, error) {
105+
// Copied from gojsonschema.
106+
var document interface{}
107+
108+
decoder := json.NewDecoder(r)
109+
decoder.UseNumber()
110+
111+
err := decoder.Decode(&document)
112+
if err != nil {
113+
return nil, err
114+
}
115+
116+
return document, nil
117+
}
118+
119+
// JsonReference implements gojsonschema.JSONLoader.JsonReference. The "Json" capitalization needs to be maintained to conform to the interface.
120+
func (l *fsLoader) JsonReference() (gojsonreference.JsonReference, error) { // nolint: golint
121+
return gojsonreference.NewJsonReference(l.JsonSource().(string))
122+
}
123+
124+
func (l *fsLoader) LoaderFactory() gojsonschema.JSONLoaderFactory {
125+
return l.factory
126+
}

schema/schema.go

Lines changed: 30 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -35,13 +35,37 @@ var (
3535
// having the OCI JSON schema files in root "/".
3636
fs = _escFS(false)
3737

38-
// specs maps OCI schema media types to schema files.
38+
// schemaNamespaces is a set of URI prefixes which are treated as containing the schema files of fs.
39+
// This is necessary because *.json schema files in this directory use "id" and "$ref" attributes which evaluate to such URIs, e.g.
40+
// ./image-manifest-schema.json URI contains
41+
// "id": "https://opencontainers.org/schema/image/manifest",
42+
// and
43+
// "$ref": "content-descriptor.json"
44+
// which evaluates as a link to
45+
// "https://opencontainers.org/schema/image/content-descriptor.json",
46+
//
47+
// To support such links without accessing the network (and trying to load content which is not hosted at these URIs),
48+
// fsLoaderFactory accepts any URI starting with one of the schemaNamespaces below,
49+
// and uses _escFS to load them from the root of its in-memory filesystem tree.
50+
//
51+
// (Note that this must contain subdirectories before its parent directories for fsLoaderFactory.refContents to work.)
52+
schemaNamespaces = []string{
53+
"https://opencontainers.org/schema/image/descriptor/",
54+
"https://opencontainers.org/schema/image/index/",
55+
"https://opencontainers.org/schema/image/manifest/",
56+
"https://opencontainers.org/schema/image/",
57+
"https://opencontainers.org/schema/",
58+
}
59+
60+
// specs maps OCI schema media types to schema URIs.
61+
// These URIs are expected to be used only by fsLoaderFactory (which trims schemaNamespaces defined above)
62+
// and should never cause a network access.
3963
specs = map[Validator]string{
40-
ValidatorMediaTypeDescriptor: "content-descriptor.json",
41-
ValidatorMediaTypeLayoutHeader: "image-layout-schema.json",
42-
ValidatorMediaTypeManifest: "image-manifest-schema.json",
43-
ValidatorMediaTypeImageIndex: "image-index-schema.json",
44-
ValidatorMediaTypeImageConfig: "config-schema.json",
64+
ValidatorMediaTypeDescriptor: "https://opencontainers.org/schema/content-descriptor.json",
65+
ValidatorMediaTypeLayoutHeader: "https://opencontainers.org/schema/image/image-layout-schema.json",
66+
ValidatorMediaTypeManifest: "https://opencontainers.org/schema/image/image-manifest-schema.json",
67+
ValidatorMediaTypeImageIndex: "https://opencontainers.org/schema/image/image-index-schema.json",
68+
ValidatorMediaTypeImageConfig: "https://opencontainers.org/schema/image/config-schema.json",
4569
}
4670
)
4771

schema/validator.go

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -67,7 +67,7 @@ func (v Validator) Validate(src io.Reader) error {
6767
}
6868
}
6969

70-
sl := gojsonschema.NewReferenceLoaderFileSystem("file:///"+specs[v], fs)
70+
sl := newFSLoaderFactory(schemaNamespaces, fs).New(specs[v])
7171
ml := gojsonschema.NewStringLoader(string(buf))
7272

7373
result, err := gojsonschema.Validate(sl, ml)

0 commit comments

Comments
 (0)