Add support for plugins referenced by URI or in external registries #269

amisevsk · 2021-02-17T01:21:09Z

What does this PR do?

Adds support for downloading plugins via URI, enabling support for components.plugin.uri and components.plugin.id when registryUrl is specified. Adds tests to cover new functions.

What issues does this PR fix or reference?

Closes #266

Is it tested? How?

The easiest way to test this PR is probably to deploy the plugin registry from this repo:

# use the deploy-registry.sh script in the repo, or
export NAMESPACE=devworkspace-plugins
kubectl apply -n ${NAMESPACE} -f \
  https://raw.githubusercontent.com/amisevsk/devfile-plugin-registry/2e8f9a211bf8c9e33149627defa4fc5dc137e2c5/deploy/deployment.yaml
kubectl apply -n ${NAMESPACE} -f \
  https://raw.githubusercontent.com/amisevsk/devfile-plugin-registry/master/deploy/service.yaml

# For minikube:
export ROUTING_SUFFIX="$(minikube ip).nip.io"
curl https://raw.githubusercontent.com/amisevsk/devfile-plugin-registry/master/deploy/k8s/ingress.yaml \
  | envsubst \
  | kc apply -n ${NAMESPACE} -f -
export REGISTRY_URL="$(kubectl get ingress -n devworkspace-plugins che-plugin-registry -o json | jq -r '.spec.rules[0].host')/v3"

# For OpenShift
kubectl apply -n ${NAMESPACE} -f \
  https://raw.githubusercontent.com/amisevsk/devfile-plugin-registry/master/deploy/os/route.yaml
export REGISTRY_URL="http://$(kubectl get route -n devworkspace-plugins che-plugin-registry -o json | jq -r '.spec.host')/v3"

Apply a devworkspace referring to this registry:

cat <<EOF | kubectl apply -f -
kind: DevWorkspace
apiVersion: workspace.devfile.io/v1alpha2
metadata:
  name: theia
spec:
  started: true
  template:
    projects:
      - name: project
        git:
          remotes:
            origin: "https://github.com/che-samples/web-nodejs-sample.git"
    components:
      - name: theia
        plugin:
          id: eclipse/che-theia/next
          registryUrl: ${REGISTRY_URL}
      - name: terminal
        plugin:
          id: eclipse/che-machine-exec-plugin/nightly/
          registryUrl: ${REGISTRY_URL}
EOF

JPinkney

Everything makes sense to me, the only problem I run into when testing is:

reconnecting-websocket-mjs.js:518 WebSocket connection to 'wss://workspace8ea81d13d3e24820.apps-crc.testing/theia/services' failed: Error during WebSocket handshake: Unexpected response code: 404
(anonymous) @ reconnecting-websocket-mjs.js:518

so I'm not actually able to get into theia and test that everything is working correctly, the loader just keeps spinning

JPinkney · 2021-02-17T15:37:29Z

pkg/library/flatten/flatten.go

+	plugin *devworkspace.PluginComponent,
+	tools ResolverTools) (resolvedPlugin *devworkspace.DevWorkspaceTemplateSpec, pluginLabels map[string]string, err error) {
+
+	if plugin.Uri == "" {


Can this piece be hit? It looks like on line ~135 ish you have case plugin.Uri != ""

You're right, it can't -- I suppose it's moreso paranoia on my part from constantly having to check component.Plugin != nil 😄

I'll remove the check (this also increases our test coverage 😉 !)

JPinkney · 2021-02-17T15:46:58Z

pkg/library/flatten/network/devfile.go

+)
+
+const (
+	SupportedDevfileSchemaVersion = "2.0.0" // TODO what should this actually be at this point?


// TODO what should this actually be at this point?

I'd like to know the answer on that as well, I've just been using 2.0.0 on the dashboard side 😆

2.0.0 don't support setting cpuLimits, while 2.1.0 supports )

Probably we need support all and convert to the latest.

Realistically, it should be something like 2\..*\..*, assuming there's no backwards-incompatible changes.

Replaced this with a basic ^2\..* regexp.

sleshchenko

I followed the instructions for OpenShift and my workspace is RUNNING but I'm not able to access Che Theia due 404 -> Cannot GET /theia/
I have RHPDS cluster, but this issue is related more to Che Theia or single host per workspace, which I implemented and merged long time ago. So, I don't see how it's related to the current PR.

The changes LGTM, my testing RHPDS as almost successful, plugins are fetched and workspace pod is correctly created

pkg/library/flatten/network/fetch.go

sleshchenko · 2021-02-18T08:33:19Z

pkg/library/flatten/network/fetch.go

+	Get(location string) (*http.Response, error)
+}
+
+func FetchDevWorkspaceTemplate(location string, httpClient HTTPGetter) (*dw.DevWorkspaceTemplateSpec, map[string]string, error) {


this method does not have doc and even return params are not named. At least it's not clear what map is supposed to contain.
Consider fixing one of them or both

Removed any handling of plugin metadata until we decide a plan for how to do it.

sleshchenko · 2021-02-18T08:36:28Z

pkg/library/flatten/network/devfile.go

+)
+
+const (
+	SupportedDevfileSchemaVersion = "2.0.0" // TODO what should this actually be at this point?


2.0.0 don't support setting cpuLimits, while 2.1.0 supports )

Probably we need support all and convert to the latest.

openshift-ci-robot · 2021-02-18T10:58:43Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: amisevsk, JPinkney, sleshchenko

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [JPinkney,amisevsk,sleshchenko]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2021-02-19T16:57:11Z

New changes are detected. LGTM label has been removed.

amisevsk · 2021-02-19T16:58:14Z

/test v5-devworkspaces-operator-e2e

amisevsk · 2021-02-19T23:00:27Z

/test v5-devworkspaces-operator-e2e

Signed-off-by: Angel Misevski <[email protected]>

Avoid processing plugin metadata as there's still not defined way to translate metadata from a devfile into a DevWorkspace[Template] Signed-off-by: Angel Misevski <[email protected]>

amisevsk · 2021-02-22T22:05:25Z

/test v5-devworkspaces-operator-e2e

Had to rebase over #270 and update imports to include v2

amisevsk requested review from sleshchenko and JPinkney February 17, 2021 01:21

openshift-ci-robot added the approved label Feb 17, 2021

amisevsk mentioned this pull request Feb 17, 2021

Investigate devfile flattening + devworkspace startup workflow #227

Closed

amisevsk changed the title ~~Plugin uris~~ Add support for plugins referenced by URI or in external registries Feb 17, 2021

JPinkney approved these changes Feb 17, 2021

View reviewed changes

openshift-ci-robot assigned JPinkney Feb 17, 2021

openshift-ci-robot added the lgtm label Feb 17, 2021

sleshchenko approved these changes Feb 18, 2021

View reviewed changes

openshift-ci-robot assigned sleshchenko Feb 18, 2021

openshift-ci-robot removed the lgtm label Feb 19, 2021

sleshchenko mentioned this pull request Feb 22, 2021

Che Controller Sprint 197 eclipse-che/che#18991

Closed

23 tasks

amisevsk force-pushed the plugin-uris branch from e244837 to 98bc81a Compare February 22, 2021 21:53

amisevsk added 7 commits February 22, 2021 16:58

Implement retrieving plugins from URI and remote registry

a874114

Signed-off-by: Angel Misevski <[email protected]>

Add support for plugins defined in registries and via URI

cd0da2a

Signed-off-by: Angel Misevski <[email protected]>

Add HTTPGetter interface to allow mocking in tests

1993472

Signed-off-by: Angel Misevski <[email protected]>

Add test cases to cover downloading plugins from URI

b65b327

Signed-off-by: Angel Misevski <[email protected]>

Move testutils to internal package to avoid exporting

8e46c07

Signed-off-by: Angel Misevski <[email protected]>

Check devfile schemaVersion via regexp to support all 2.x devfiles

a103cab

Signed-off-by: Angel Misevski <[email protected]>

Removing handling of plugin metadata

69aa54c

Avoid processing plugin metadata as there's still not defined way to translate metadata from a devfile into a DevWorkspace[Template] Signed-off-by: Angel Misevski <[email protected]>

amisevsk force-pushed the plugin-uris branch from 98bc81a to 69aa54c Compare February 22, 2021 22:04

amisevsk merged commit e0e3b62 into devfile:master Feb 23, 2021

amisevsk deleted the plugin-uris branch February 23, 2021 01:00

amisevsk mentioned this pull request Feb 23, 2021

Add support for ephemeral volumes #281

Merged

amisevsk mentioned this pull request Mar 19, 2021

Add library package for resolving devfile parents and plugins devfile/library#74

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add support for plugins referenced by URI or in external registries #269

Add support for plugins referenced by URI or in external registries #269

amisevsk commented Feb 17, 2021 •

edited by sleshchenko

Loading

JPinkney left a comment

JPinkney Feb 17, 2021

amisevsk Feb 17, 2021

JPinkney Feb 17, 2021

sleshchenko Feb 18, 2021

amisevsk Feb 18, 2021

amisevsk Feb 19, 2021

sleshchenko left a comment

sleshchenko Feb 18, 2021 •

edited

Loading

amisevsk Feb 18, 2021

amisevsk Feb 19, 2021

sleshchenko Feb 18, 2021

openshift-ci-robot commented Feb 18, 2021

openshift-ci-robot commented Feb 19, 2021

amisevsk commented Feb 19, 2021

amisevsk commented Feb 19, 2021

amisevsk commented Feb 22, 2021

Add support for plugins referenced by URI or in external registries #269

Add support for plugins referenced by URI or in external registries #269

Conversation

amisevsk commented Feb 17, 2021 • edited by sleshchenko Loading

What does this PR do?

What issues does this PR fix or reference?

Is it tested? How?

JPinkney left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sleshchenko left a comment

Choose a reason for hiding this comment

sleshchenko Feb 18, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openshift-ci-robot commented Feb 18, 2021

openshift-ci-robot commented Feb 19, 2021

amisevsk commented Feb 19, 2021

amisevsk commented Feb 19, 2021

amisevsk commented Feb 22, 2021

amisevsk commented Feb 17, 2021 •

edited by sleshchenko

Loading

sleshchenko Feb 18, 2021 •

edited

Loading