Add AppWrapper v1beta2 CRD and controllers to Codeflare operator #543

dgrove-oss · 2024-04-22T22:14:57Z

Replaces #491.

This assumes/includes #541.

dgrove-oss · 2024-04-26T17:46:58Z

rebased to resolve conflict in Dockerfile

main.go

Srihari1192 · 2024-05-06T11:53:18Z

@dgrove-oss Appwrapper instance creation is failing with below error

Error "failed calling webhook "mappwrapper.kb.io": failed to call webhook: the server could not find the requested resource" for field "undefined".

dgrove-oss · 2024-05-06T15:20:54Z

I make an additional adjustment to the startup logic so that the AppWrapper webhooks will be registered as soon as the certificates are ready if the operator config enables AppWrappers.

However, the AppWrapper CRD is still installed unconditionally. If AppWrapper is disabled in the config, this will result in the user getting a cryptic error like:

Error from server (InternalError): error when creating "../appwrapper/samples/wrapped-pod.yaml": Internal error occurred: failed calling webhook "mappwrapper.kb.io": failed to call webhook: the server could not find the requested resource

when they create or edit an AppWrapper.

If AppWrappers are enabled in the config, then you should see the following behavior (if Kueue is already installed on your cluster, only step 3 below is relevant).

Initial startup of make deploy ENV=e2e (trimming noise from cert rotation). AppWrappers are enabled, but Kueue is not installed in the cluster

2024-05-06T14:44:36Z	INFO	setup	Build info	{"operatorVersion": "", "appwrapperVersion": "UNKNOWN", "date": "2024-05-06 14:38"}
2024-05-06T14:44:36Z	INFO	setup	setting up health endpoints
2024-05-06T14:44:36Z	INFO	setup	setting up RayCluster controller
2024-05-06T14:44:36Z	INFO	We detected being on Vanilla Kubernetes!
2024-05-06T14:44:36Z	INFO	setup	setting up AppWrapper components
2024-05-06T14:44:36Z	INFO	setup	Workload API not available; setting up waiter for Workload API availability
2024-05-06T14:44:36Z	INFO	setup	starting manager
2024-05-06T14:44:36Z	INFO	controller-runtime.metrics	Starting metrics server
2024-05-06T14:44:36Z	INFO	controller-runtime.metrics	Serving metrics server	{"bindAddress": ":8080", "secure": false}
2024-05-06T14:44:36Z	INFO	starting server	{"kind": "health probe", "addr": "[::]:8081"}
2024-05-06T14:44:36Z	INFO	setup	API workloads.kueue.x-k8s.io not available, setting up retry watcher
2024-05-06T14:44:36Z	INFO	setup	API rayclusters.ray.io not available, setting up retry watcher
2024-05-06T14:44:36Z	INFO	Starting workers	{"controller": "cert-rotator", "worker count": 1}
2024-05-06T14:44:38Z	INFO	setup	Setting up AppWrapper webhook
2024-05-06T14:44:38Z	INFO	controller-runtime.builder	Registering a mutating webhook	{"GVK": "workload.codeflare.dev/v1beta2, Kind=AppWrapper", "path": "/mutate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:44:38Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/mutate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:44:38Z	INFO	controller-runtime.builder	Registering a validating webhook	{"GVK": "workload.codeflare.dev/v1beta2, Kind=AppWrapper", "path": "/validate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:44:38Z	INFO	controller-runtime.webhook	Registering webhook	{"path": "/validate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:44:38Z	INFO	controller-runtime.webhook	Starting webhook server
2024-05-06T14:44:38Z	INFO	controller-runtime.certwatcher	Updated current TLS certificate
2024-05-06T14:44:38Z	INFO	controller-runtime.webhook	Serving webhook server	{"host": "", "port": 9443}
2024-05-06T14:44:38Z	INFO	controller-runtime.certwatcher	Starting certificate watcher
2024-05-06T14:47:06Z	INFO	admission	Applying defaults	{"webhookGroup": "workload.codeflare.dev", "webhookKind": "AppWrapper", "AppWrapper": {"name":"sample-job","namespace":"default"}, "namespace": "default", "name": "sample-job", "resource": {"group":"workload.codeflare.dev","version":"v1beta2","resource":"appwrappers"}, "user": "kubernetes-admin", "requestID": "53545468-844c-43a0-8bc3-3649a124da80", "job": {"apiVersion": "workload.codeflare.dev/v1beta2", "kind": "AppWrapper", "namespace": "default", "name": "sample-job"}}
2024-05-06T14:47:06Z	INFO	admission	Validating create	{"webhookGroup": "workload.codeflare.dev", "webhookKind": "AppWrapper", "AppWrapper": {"name":"sample-job","namespace":"default"}, "namespace": "default", "name": "sample-job", "resource": {"group":"workload.codeflare.dev","version":"v1beta2","resource":"appwrappers"}, "user": "kubernetes-admin", "requestID": "35624b2e-ebb7-43a1-8989-dc02e826908f", "job": {"apiVersion": "workload.codeflare.dev/v1beta2", "kind": "AppWrapper", "namespace": "default", "name": "sample-job"}}

do a make kueue-e2e to install Kueue; the codeflare operator should restart

2024-05-06T14:51:39Z	INFO	setup	API workloads.kueue.x-k8s.io installed, invoking deferred action
2024-05-06T14:51:39Z	INFO	setup	Workload API now available; triggering controller restart
...
2024-05-06T14:51:39Z	INFO	Wait completed, proceeding to shutdown the manager

On restart, AppWrappers should be fully enabled


2024-05-06T14:52:03Z    INFO    setup   Build info      {"operatorVersion": "", "appwrapperVersion": "UNKNOWN", "date": "2024-05-06 14:38"}
2024-05-06T14:52:03Z    INFO    setup   setting up health endpoints
2024-05-06T14:52:03Z    INFO    setup   setting up RayCluster controller
2024-05-06T14:52:03Z    INFO    We detected being on Vanilla Kubernetes!
2024-05-06T14:52:03Z    INFO    setup   setting up AppWrapper components
2024-05-06T14:52:03Z    INFO    setup   Workload API available; enabling AppWrappers
2024-05-06T14:52:03Z    INFO    setup   Waiting for certificate generation to complete
2024-05-06T14:52:03Z    INFO    setup   starting manager
2024-05-06T14:52:03Z    INFO    controller-runtime.metrics      Starting metrics server
2024-05-06T14:52:03Z    INFO    starting server {"kind": "health probe", "addr": "[::]:8081"}
2024-05-06T14:52:03Z    INFO    controller-runtime.metrics      Serving metrics server  {"bindAddress": ":8080", "secure": false}
2024-05-06T14:52:03Z    INFO    setup   API rayclusters.ray.io not available, setting up retry watcher
2024-05-06T14:52:03Z    INFO    cert-rotation   starting cert rotator controller
2024-05-06T14:52:03Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *v1.Secret"}
2024-05-06T14:52:03Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-05-06T14:52:03Z    INFO    Starting EventSource    {"controller": "cert-rotator", "source": "kind source: *unstructured.Unstructured"}
2024-05-06T14:52:03Z    INFO    Starting Controller     {"controller": "cert-rotator"}
2024-05-06T14:52:03Z    INFO    cert-rotation   no cert refresh needed
2024-05-06T14:52:03Z    INFO    cert-rotation   certs are ready in /tmp/k8s-webhook-server/serving-certs
2024-05-06T14:52:03Z    INFO    Starting workers        {"controller": "cert-rotator", "worker count": 1}
2024-05-06T14:52:03Z    INFO    cert-rotation   no cert refresh needed
2024-05-06T14:52:03Z    INFO    cert-rotation   Ensuring CA cert        {"name": "codeflare-operator-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration", "name": "codeflare-operator-validating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=ValidatingWebhookConfiguration"}
2024-05-06T14:52:03Z    INFO    cert-rotation   Ensuring CA cert        {"name": "codeflare-operator-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration", "name": "codeflare-operator-mutating-webhook-configuration", "gvk": "admissionregistration.k8s.io/v1, Kind=MutatingWebhookConfiguration"}
2024-05-06T14:52:05Z    INFO    cert-rotation   CA certs are injected to webhooks
2024-05-06T14:52:05Z    INFO    setup   Setting up AppWrapper webhook
2024-05-06T14:52:05Z    INFO    setup   Setting up AppWrapper controller
2024-05-06T14:52:05Z    INFO    Starting Controller     {"controller": "AppWrapperChildWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper"}
2024-05-06T14:52:05Z    INFO    Starting workers        {"controller": "AppWrapperChildWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "worker count": 1}
2024-05-06T14:52:05Z    INFO    Starting EventSource    {"controller": "AppWrapperChildWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "source": "kind source: *v1beta2.AppWrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.builder      Registering a mutating webhook  {"GVK": "workload.codeflare.dev/v1beta2, Kind=AppWrapper", "path": "/mutate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/mutate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.builder      Registering a validating webhook        {"GVK": "workload.codeflare.dev/v1beta2, Kind=AppWrapper", "path": "/validate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.webhook      Registering webhook     {"path": "/validate-workload-codeflare-dev-v1beta2-appwrapper"}
2024-05-06T14:52:05Z    INFO    Starting EventSource    {"controller": "AppWrapperWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "source": "kind source: *v1beta2.AppWrapper"}
2024-05-06T14:52:05Z    INFO    Starting EventSource    {"controller": "AppWrapperWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "source": "kind source: *v1beta1.Workload"}
2024-05-06T14:52:05Z    INFO    Starting Controller     {"controller": "AppWrapperWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.webhook      Starting webhook server
2024-05-06T14:52:05Z    INFO    Starting EventSource    {"controller": "AppWrapper", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "source": "kind source: *v1beta2.AppWrapper"}
2024-05-06T14:52:05Z    INFO    Starting EventSource    {"controller": "AppWrapper", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "source": "kind source: *v1.Pod"}
2024-05-06T14:52:05Z    INFO    Starting Controller     {"controller": "AppWrapper", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper"}
2024-05-06T14:52:05Z    INFO    controller-runtime.certwatcher  Updated current TLS certificate
2024-05-06T14:52:05Z    INFO    controller-runtime.webhook      Serving webhook server  {"host": "", "port": 9443}
2024-05-06T14:52:05Z    INFO    controller-runtime.certwatcher  Starting certificate watcher
2024-05-06T14:52:05Z    INFO    Starting workers        {"controller": "AppWrapperWorkload", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "worker count": 1}
2024-05-06T14:52:05Z    INFO    Starting workers        {"controller": "AppWrapper", "controllerGroup": "workload.codeflare.dev", "controllerKind": "AppWrapper", "worker count": 1}

dgrove-oss · 2024-05-06T15:23:08Z

To document the expectation, if AppWrappers are disabled in the config your log should look like this:

2024-05-06T15:06:36Z	INFO	setup	Build info	{"operatorVersion": "", "appwrapperVersion": "UNKNOWN", "date": "2024-05-06 14:38"}
2024-05-06T15:06:36Z	INFO	setup	setting up health endpoints
2024-05-06T15:06:36Z	INFO	setup	setting up RayCluster controller
2024-05-06T15:06:36Z	INFO	We detected being on Vanilla Kubernetes!
2024-05-06T15:06:36Z	INFO	setup	setting up AppWrapper components
2024-05-06T15:06:36Z	INFO	setup	AppWrappers are disabled by operator configuration
2024-05-06T15:06:36Z	INFO	setup	starting manager
...

dgrove-oss · 2024-05-06T20:35:08Z

I made further adjustments. Now if AppWrappers are completely disabled by the config, we setup a webhook that generates an error when AppWrappers are created.

Error from server (Forbidden): error when creating "../appwrapper/samples/wrapped-job.yaml": admission webhook "vappwrapper.kb.io" denied the request: AppWrappers disabled by CodeFlare operator configuration

dgrove-oss · 2024-05-07T17:16:37Z

I've ported the e2e tests from #491 to this PR as well now.

…perator

Define mock appwrapper webhooks to enable reasonable error messages for users when AppWrappers are disabled by configuration.

pkg/config/config.go

astefanutti · 2024-05-16T16:42:46Z

/lgtm

astefanutti · 2024-05-17T09:50:47Z

/approve

openshift-ci · 2024-05-17T09:50:53Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: astefanutti

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [astefanutti]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci bot requested review from astefanutti and dimakis April 22, 2024 22:15

dgrove-oss force-pushed the appwrapper branch from 800b6b6 to 628c021 Compare April 22, 2024 22:24

openshift-merge-robot added the needs-rebase label Apr 26, 2024

dgrove-oss force-pushed the appwrapper branch from 80a8b44 to 4f6aa2e Compare April 26, 2024 17:46

openshift-merge-robot removed the needs-rebase label Apr 26, 2024

dgrove-oss force-pushed the appwrapper branch 2 times, most recently from a67dd1f to a940fa0 Compare April 26, 2024 21:01

dgrove-oss commented Apr 26, 2024

View reviewed changes

main.go Outdated Show resolved Hide resolved

dgrove-oss commented Apr 26, 2024

View reviewed changes

main.go Outdated Show resolved Hide resolved

dgrove-oss changed the title ~~Add AppWrapper CRD and controllers to Codeflare operator~~ Add AppWrapper v1beta2 CRD and controllers to Codeflare operator Apr 29, 2024

dgrove-oss force-pushed the appwrapper branch 3 times, most recently from 0dd33f7 to 143b507 Compare April 29, 2024 18:23

Srihari1192 self-requested a review April 30, 2024 10:08

dgrove-oss mentioned this pull request May 2, 2024

Upgrade from AppWrapper v1beta1 to v1beta2 #491

Closed

4 tasks

dgrove-oss force-pushed the appwrapper branch from 6e8beb4 to 3b3fa8d Compare May 6, 2024 13:56

dgrove-oss force-pushed the appwrapper branch from db0d3a9 to 248726a Compare May 7, 2024 15:36

openshift-merge-robot added the needs-rebase label May 10, 2024

dgrove-oss force-pushed the appwrapper branch from f024fa8 to 77a5c28 Compare May 10, 2024 15:33

openshift-merge-robot added needs-rebase and removed needs-rebase labels May 10, 2024

dgrove-oss force-pushed the appwrapper branch from 2aa306c to fead99b Compare May 14, 2024 01:08

openshift-merge-robot removed the needs-rebase label May 14, 2024

dgrove-oss added 16 commits May 16, 2024 09:47

upgrade appwrapper to v0.9.0

7c20552

adjust appwrapper controller startup

80c378b

graceful handling of Kueue not being installed before the codeflare o…

34236a9

…perator

update to appwrapper 0.10.0

4721c96

configure appwrapper webhooks

56171b5

restructure AppWrapper startup to ensure all-or-nothing setup

664c926

AppWrapper webhook can be registered without Kueue APIs

1fd6972

improved appwrapper webhook configuration

38a9dee

Define mock appwrapper webhooks to enable reasonable error messages for users when AppWrappers are disabled by configuration.

update to appwrapper v0.11.0

d7bafdc

adjust e2e test so that RayCluster really is managed by Kueue

322aca7

add AppWrapper tests

816a81e

update to appwrapper v0.11.1

5d5adb0

address smaller review comments

d022ef4

simplify mock AppWrapper webhooks

3a6e338

inline AppWrapperConfig struct

948a7ce

missed makefile adjustment for kueue in tag-and-build

55fe025

dgrove-oss force-pushed the appwrapper branch from f562a78 to 55fe025 Compare May 16, 2024 13:47

openshift-merge-robot removed the needs-rebase label May 16, 2024

dgrove-oss added 2 commits May 16, 2024 09:59

remove kustomize patches/config for conversion webhook

444c9d3

drop mock appwrapper webhook

f087f6a

astefanutti reviewed May 16, 2024

View reviewed changes

pkg/config/config.go Outdated Show resolved Hide resolved

inline appwrapper config struct at the json level

e9e54b6

openshift-ci bot assigned astefanutti May 16, 2024

openshift-ci bot added the lgtm label May 16, 2024

openshift-ci bot added the approved label May 17, 2024

openshift-merge-bot bot merged commit 92e6eb5 into project-codeflare:main May 17, 2024
8 checks passed

dgrove-oss deleted the appwrapper branch May 19, 2024 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add AppWrapper v1beta2 CRD and controllers to Codeflare operator #543

Add AppWrapper v1beta2 CRD and controllers to Codeflare operator #543

dgrove-oss commented Apr 22, 2024 •

edited

Loading

dgrove-oss commented Apr 26, 2024

Srihari1192 commented May 6, 2024

dgrove-oss commented May 6, 2024 •

edited

Loading

dgrove-oss commented May 6, 2024

dgrove-oss commented May 6, 2024 •

edited

Loading

dgrove-oss commented May 7, 2024

astefanutti commented May 16, 2024

astefanutti commented May 17, 2024

openshift-ci bot commented May 17, 2024

Add AppWrapper v1beta2 CRD and controllers to Codeflare operator #543

Add AppWrapper v1beta2 CRD and controllers to Codeflare operator #543

Conversation

dgrove-oss commented Apr 22, 2024 • edited Loading

dgrove-oss commented Apr 26, 2024

Srihari1192 commented May 6, 2024

dgrove-oss commented May 6, 2024 • edited Loading

dgrove-oss commented May 6, 2024

dgrove-oss commented May 6, 2024 • edited Loading

dgrove-oss commented May 7, 2024

astefanutti commented May 16, 2024

astefanutti commented May 17, 2024

openshift-ci bot commented May 17, 2024

dgrove-oss commented Apr 22, 2024 •

edited

Loading

dgrove-oss commented May 6, 2024 •

edited

Loading

dgrove-oss commented May 6, 2024 •

edited

Loading