Skip to content

Commit 7664ab3

Browse files
jmrodrijmrodri
authored
enhancement: External validations for Operator SDK (#98)
* Add motivation and summary Signed-off-by: jesus m. rodriguez <[email protected]> * add goals and motiviation Signed-off-by: jesus m. rodriguez <[email protected]> * add open questions Signed-off-by: jesus m. rodriguez <[email protected]> * Add alternatives and what the current tests do Signed-off-by: jesus m. rodriguez <[email protected]> * DRAFT 2 of the validation EP Signed-off-by: jesus m. rodriguez <[email protected]> * Minor updates Signed-off-by: jesus m. rodriguez <[email protected]> * Final version of the external validation EP Signed-off-by: jesus m. rodriguez <[email protected]> * feedback: reword non-goal; make bullets consistent * feedback: spell out CVP in Summary section * feedback: address validators in other languages * feedback: add a blurb to Risks & Mitigations section * feedback: add example to existing validators * feedback: add examples, non-goal, and clarification.
1 parent 51b42b3 commit 7664ab3

File tree

1 file changed

+380
-0
lines changed

1 file changed

+380
-0
lines changed

‎enhancements/sdk-external-and-pluggable-validations.md

+380
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,380 @@
1+
---
2+
title: sdk-external-and-pluggable-validations
3+
authors:
4+
- "@jmrodri"
5+
reviewers:
6+
- "@camilamacedo86"
7+
- "@joelanford"
8+
- "@bparees"
9+
approvers:
10+
- "@camilamacedo86"
11+
- "@joelanford"
12+
creation-date: 2021-10-01
13+
last-updated: 2021-10-01
14+
status: implementable
15+
see-also:
16+
- "/enhancements/this-other-neat-thing.md"
17+
---
18+
19+
# SDK External and Pluggable Validations
20+
21+
## Release Signoff Checklist
22+
23+
- [X] Enhancement is `implementable`
24+
- [X] Design details are appropriately documented from clear requirements
25+
- [ ] Test plan is defined
26+
- [ ] Graduation criteria for dev preview, tech preview, GA
27+
28+
## Open Questions [optional]
29+
30+
1. Can we use scorecard to replace all of these validations?
31+
* Scorecard uses the cluster to run the tests, these validations typically
32+
run locally or in a pipeline. They are also done before the expensive
33+
operator tests are run.
34+
1. What would it take to convert an existing validator to an executable format?
35+
Simply add a `main.go` to wrap existing validators.
36+
1. Do we need to add `json` tags to [these structs][errors-go]?
37+
38+
## Summary
39+
40+
Today, validations used by Container Verification Pipeline (CVP) are compiled into
41+
the `operator-sdk`. Any changes to the validation rules requires a release of
42+
operator-framework/api followed by a release of `operator-sdk`. This proposal
43+
attempts to design a way where the validations can be hosted in their own repos
44+
as well as updated without requiring new releases of the `operator-sdk`.
45+
46+
The validator is defined in the operator-framework/api, for example here is the
47+
[`OperatorHubValidator`][operator-hub-validator]. In the `operator-sdk`, it is
48+
"registered" as an optional validator.
49+
50+
```go
51+
var optionalValidators = validators{
52+
{
53+
Validator: apivalidation.OperatorHubValidator,
54+
name: "operatorhub",
55+
labels: map[string]string{
56+
nameKey: "operatorhub",
57+
suiteKey: "operatorframework",
58+
},
59+
desc: "OperatorHub.io metadata validation. ",
60+
},
61+
...
62+
```
63+
64+
When the validators are specified on the CLI, we call `run()` on the
65+
[validators][optional-validators]. In that `run()` method we loop through the
66+
validaors specified by `vals`. On each validator we invoke `Validate` with the
67+
list of objects to validate.
68+
69+
```go
70+
...
71+
for _, v := range vals {
72+
if sel.Matches(labels.Set(v.labels)) {
73+
results = append(results, v.Validate(objs...)...)
74+
}
75+
}
76+
...
77+
```
78+
79+
## Motivation
80+
81+
Every time the business changes validation rules, it requires an update to the
82+
operator-framework/api library. A release of said library, then it needs to get
83+
included into the operator-sdk. Then a release of the SDK needs to be cut in
84+
order for the new validation rule change to be usable.
85+
86+
This process slows down the business by having to wait for this release process
87+
to occur for what could be a very small rule change. It also causes downstream
88+
rules to require an immediate upstream operator-sdk release, which may
89+
otherwise be as far as 3 weeks away. It is difficult to explain to the community
90+
why we need a new release for a CVP need.
91+
92+
Having the validators external to the SDK will allow for vendor specific
93+
validators to be created allowing for greater flexibility.
94+
95+
### Goals
96+
97+
* Allow validations to be updated when the business needs them to be
98+
* Allow validations to release when the authors need them to be
99+
* Do not require newer builds of the operator-sdk to get updated rules
100+
* Allow validations to be hosted in their own repos
101+
* Allow validations to be external to operator-sdk
102+
103+
### Non-Goals
104+
105+
* Migrate existing validators to the new format
106+
* The distribution of the external validators
107+
108+
## Proposal
109+
110+
### User Stories [optional]
111+
112+
#### Story 1
113+
* as a CVP admin I would like to change the validation for XXX and have it
114+
useable as soon as I can do a validator release.
115+
116+
#### Story 2
117+
* as a validation author, I can write a validation for a bundle and use it
118+
without having to get a new operator-sdk release
119+
120+
#### Story 3
121+
* Users want to edit/add/remove a set of validation rules for Operator bundle validation.
122+
123+
#### Story 4
124+
* Users can host a set of validation rules either locally or in remote.
125+
126+
#### Story 5
127+
* Users can define the target set of validation rules for SDK's bundle validation command (either from local or remote source).
128+
129+
### Implementation Details/Notes/Constraints [optional]
130+
131+
There are a few alternatives, but I think wrapping the validations in their
132+
own executables makes sense. It aligns with the work we've done with
133+
[Phase 2][phase-2] plugins. It allows the most flexibility in terms of
134+
implementation. It also has a simple API -
135+
input: bundle dir; output: ManifestResult JSON.
136+
137+
Wrapping validations in their own executable simply means that there
138+
needs to be something that the operator-sdk can run like a shell script
139+
or a binary.
140+
141+
The validation executable will need to accept a single input value which
142+
is the bundle directory. The output will need to be a ManifestResult JSON. This
143+
JSON will be parsed by the operator-sdk converted and output as a Result.
144+
145+
For example, the existing validations could easily be wrapped with a `main.go`
146+
file and compiled into a binary. They could be copied to their own repos for
147+
easier release. Now you might be thinking that this means we'll have many
148+
releases still, we actually would only have one (1) release which is the
149+
validator itself. For go based validators this might mean a new binary, but you
150+
could also release your validator in python and make it a single file or even a
151+
shell script.
152+
153+
From the operator-sdk's point of view, we don't care what you use
154+
to create your validator only that we can run it and pass it a bundle directory.
155+
156+
#### Validators
157+
158+
##### Running validators
159+
160+
The [existing validators](#default-validators-run-by-operator-sdk) operate
161+
on a bundle. Each validator should be some executable that accepts a
162+
single argument, the bundle root directory. So `operator-sdk` will pass them
163+
the bundle directory. It will be the validator's responsibility to parse
164+
the bundle to get what it needs.
165+
166+
For example, the validator executable will be invoked from the
167+
`operator-sdk` in the following manner:
168+
169+
```
170+
/path/to/validator/executable /path/to/bundle
171+
```
172+
173+
A concrete example might look like this:
174+
175+
```
176+
./validator-poc /home/user/dev/gatekeeper-operator/bundle
177+
```
178+
179+
The actual implementation is up to the author. Here we have an example of
180+
the [`OperatorHubValidator`][validator-poc1] as a Go binary.
181+
182+
The validators can be written in any language but there must be an executable
183+
entry point that accepts a single bundle path as a CLI argument. For example,
184+
you can write your validator in python but you would want to make the main
185+
python file executable or supply a shell script to be invoked.
186+
187+
##### Validator results
188+
189+
As stated earlier, each validator should be some executable that accepts a
190+
single argument, the bundle root directory. Th validators should also output
191+
ManifestResult JSON to stdout.
192+
193+
Because the [existing validators](#default-validators-run-by-operator-sdk)
194+
currently return a [ManifestResult][manifest-result], it seems logical that we
195+
use the same object as JSON for the output.
196+
197+
The validator executable should exit non-zero ONLY if the entrypoint failed to
198+
run NOT if the bundle validation fails.
199+
200+
For example, let's say the validator is given a path to the gatekeeper bundle.
201+
The validator should validate the given bundle and output the ManifestResult JSON.
202+
Here is an example of a run:
203+
204+
```json
205+
{
206+
"Name": "gatekeeper-operator.v0.2.0-rc.3",
207+
"Errors": null,
208+
"Warnings": [
209+
{
210+
"Type": "CSVFileNotValid",
211+
"Level": "Warning",
212+
"Field": "",
213+
"BadValue": "",
214+
"Detail": "(gatekeeper-operator.v0.2.0-rc.3) csv.Spec.minKubeVersion is not informed. It is recommended you provide this information. Otherwise, it would mean that your operator project can be distributed and installed in any cluster version available, which is not necessarily the case for all projects."
215+
}
216+
]
217+
}
218+
```
219+
220+
The above JSON will be read by the `operator-sdk` during `bundle validate`
221+
command and output the results as it does today. The example below shows what
222+
`operator-sdk bundle validate` would printout if given the ManifestResult from
223+
above.
224+
225+
```json
226+
{
227+
"passed": true,
228+
"outputs": [
229+
{
230+
"type": "warning",
231+
"message": "Warning: Value : (gatekeeper-operator.v0.2.0-rc.3) csv.Spec.minKubeVersion is not informed. It is recommended you provide this information. Otherwise, it would mean that your operator project can be distributed and installed in any cluster version available, which is not necessarily the case for all projects."
232+
}
233+
]
234+
}
235+
```
236+
237+
Allowing the validators to output ManifestResult should make it easier to
238+
transition existing validators to the external format with minimal code.
239+
240+
##### Migrating existing validator to executable
241+
242+
In the short to near term, you create a new `main.go` for each of the validators.
243+
Then import the validator code from operator-framework/api. The `main.go` would
244+
take in one (1) argument, the bundle directory.
245+
246+
Since the [existing validators](#default-validators-run-by-operator-sdk) already
247+
output `ManifestResult`, it's easiest if we simply print that out as JSON to
248+
stdout.
249+
250+
An example POC that takes an existing validator and outputs `ManifestResult` can
251+
be found at [validator-poc][validator-poc1]
252+
253+
Another pair of examples of a migration can be found at
254+
[ocp-olm-catalog-validator][ocp-olm-catalog-valdator] and at
255+
[the k8s-bundle-validator][k8s-bundle-validator]. These particular examples do NOT
256+
yet output `ManifestResult` format.
257+
258+
#### CLI
259+
260+
The big question at hand is how do we indicate to the `operator-sdk` CLI that
261+
we want to run an external validator? Today, the `bundle validate` command takes
262+
in a few flags, here we will discuss how each might need to be changed to work
263+
with external validators.
264+
265+
```
266+
Usage:
267+
operator-sdk bundle validate [flags]
268+
269+
...
270+
271+
Flags:
272+
--alpha-select-external string Selector to select external validators to run. It should be set to a Unix path list ("/path/to/e1.sh:/path/to/e2")
273+
-h, --help help for validate
274+
-b, --image-builder string Tool to pull and unpack bundle images. Only used when validating a bundle image. One of: [docker, podman, none] (default "docker")
275+
--list-optional List all optional validators available. When set, no validators will be run
276+
--optional-values --optional-values=k8s-version=1.22 Inform a []string map of key=values which can be used by the validator. e.g. to check the operator bundle against an Kubernetes version that it is intended to be distributed use --optional-values=k8s-version=1.22 (default [])
277+
-o, --output string Result format for results. One of: [text, json-alpha1]. Note: output format types containing "alphaX" are subject to change and not covered by guarantees of stable APIs. (default "text")
278+
--select-optional string Label selector to select optional validators to run. Run this command with '--list-optional' to list available optional validators
279+
280+
Global Flags:
281+
--plugins strings plugin keys to be used for this subcommand execution
282+
--verbose Enable verbose logging
283+
```
284+
285+
* *--help* works as is
286+
* *--image-builder* works as is
287+
* *--list-optional* would need to be updated to locate external validators.
288+
* *--optional-values* would continue to be passed to the validators
289+
* *--output* indicates how we want to output the results.
290+
* *--select-optional* works as is
291+
* *-- alpha-select-external* added; takes in the location of the executable to run as
292+
the validator, i.e. `/path/to/validator/the-executable:/path/to/another`
293+
294+
### Risks and Mitigations
295+
296+
There is little risk, if this does not pan out we keep going on the current path
297+
of compiling them in.
298+
299+
Another possible risk is users of an airgapped environment will have to install
300+
these external validator executables. We could supply a mechanism for
301+
distributing these external validators as an image. For this first release, we
302+
will ignore the distribution solutions leaving it to the user to copy the
303+
validators onto the system running `operator-sdk`.
304+
305+
## Design Details
306+
307+
### Test Plan
308+
309+
- unit tests for the feature will be added in the implementation
310+
- QE would need to create custom validations to be called by the new `--alpha-select-external` flag.
311+
312+
### Graduation Criteria
313+
314+
This feature will come out as alpha accessible via the `--alpha-select-external`
315+
flag to the `bundle validate` command. We will accept feedback on possible
316+
changes to the feature.
317+
318+
After a period of time, we can graduate to a GA with a more permanent flag, i.e.
319+
`--select-external`.
320+
321+
### Version Skew Strategy
322+
323+
* The version of the validators can change however the validator author sees fit.
324+
* The API or contract between `operator-sdk` and validators will be
325+
* input to validator: bundle directory
326+
* output from validator: `ManifestResult` JSON
327+
328+
## Implementation History
329+
330+
N/A
331+
332+
## Drawbacks
333+
334+
Validators would have to be their own executables which could result in a
335+
compilation step being needed depending on the language used to implement them.
336+
337+
## Alternatives
338+
339+
* put validations in their own images
340+
* need to define "API" contract what is the entrypoint and what parameters do
341+
we give it
342+
* pro:
343+
* already have a precendence for running things from images
344+
* familiar tech
345+
* con:
346+
* would need a cluster to run these validations
347+
* authors would have to create binaries of their validations anyway
348+
349+
* use a language like JavaScript or CUE to define all validations
350+
* validations could be run from a git repo, i.e. operator-sdk could pull it
351+
and then evaluate it
352+
* pro:
353+
* simpler delivery, expose via a gitrepo and done
354+
* con:
355+
* all existing validations would have to be re-written in a new language
356+
structure which could introduce new bugs
357+
* unproven technology
358+
* would have to write the engine to know how to execute these
359+
360+
* use scorecard to do the validations
361+
* create validations written in scorecard as custom tests
362+
* pro:
363+
* infrastructure required to run is already built within scorecard
364+
* con:
365+
* would need a cluster to run these validations
366+
* because of cluster, tests would require 3 minutes to be executed
367+
* would make it very time consuming and difficult to run from Audit Tool
368+
369+
## Infrastructure Needed [optional]
370+
371+
N/A
372+
373+
[phase-2]: https://github.com/kubernetes-sigs/kubebuilder/blob/master/designs/extensible-cli-and-scaffolding-plugins-phase-2.md
374+
[manifest-result]: https://github.com/operator-framework/api/blob/master/pkg/validation/errors/error.go#L9-L16
375+
[validator-poc1]: https://github.com/jmrodri/validator-poc/tree/poc1-manifestresults
376+
[errors-go]: https://github.com/operator-framework/api/blob/master/pkg/validation/errors/error.go#L9-L16
377+
[operator-hub-validator]: https://github.com/operator-framework/api/blob/master/pkg/validation/internal/operatorhub.go
378+
[optional-validator]: https://github.com/operator-framework/operator-sdk/blob/master/internal/cmd/operator-sdk/bundle/validate/optional.go#L130-L156
379+
[ocp-olm-catalog-validator]: https://github.com/redhat-openshift-ecosystem/ocp-olm-catalog-validator
380+
[k8s-bundle-validator]: https://github.com/k8s-operatorhub/bundle-validator

0 commit comments

Comments
 (0)