[ide] use code flag to install extensions #8313

mustard-mh · 2022-02-18T13:18:23Z

Description

Track readiness for content ide ide-desktop ready

Upstream vscode has provided extension install with flag --install-extension and there are some bugs with our installation code
~~🐛 Remove extensionIdRegex for extensions that install via file path~~
🐛 Support extensions that their id starts with http or https

It's a Golang copy of these codes

And part of extension host timeout #7863 fixes, more detail see internal slack

Related Issue(s)

Relates #7863, fixes #7409

How to test

For test, we use this version of vscode which remove our custom extensions install only gitpod-io/openvscode-server@6d0747d

Just remove support of install with file path

Make sure to use insider vscode https://hw-code-flag.staging.gitpod-dev.com/preferences
Open a workspace, you can use this repo https://hw-code-flag.staging.gitpod-dev.com/#github.com/mustard-mh/test/tree/vs/exthost-no-file
Click Extensions button of vscode check if all your extensions installed
Run ps -efj fww should have a process like

sh /ide/bin/gitpod-code --start-server --install-builtin-extension github.vscode-pull-request-github --install-extension bierner.color-info --install-extension [email protected] --install-extension magicstack.magicpython --install-extension formulahendry.code-runner --port 23000 --host 0.0.0.0 --without-connection-token --server-data-dir /workspace/.vscode-remote

Go to segment.com search with supervisor_readiness to see event tracking

Release Notes

NONE

Documentation

/werft analytics=segment|TEZnsG4QbLSxLfHfNieLYGF4cDwyFWoe

codecov · 2022-02-18T13:22:09Z

Codecov Report

Merging #8313 (8d31c75) into main (42d529c) will increase coverage by 18.73%.
The diff coverage is 0.00%.

❗ Current head 8d31c75 differs from pull request most recent head c8ec22e. Consider uploading reports for the commit c8ec22e to get more accurate results

@@             Coverage Diff             @@
##             main    #8313       +/-   ##
===========================================
+ Coverage   12.31%   31.05%   +18.73%     
===========================================
  Files          20       37       +17     
  Lines        1161     5842     +4681     
===========================================
+ Hits          143     1814     +1671     
- Misses       1014     3890     +2876     
- Partials        4      138      +134

Flag	Coverage Δ
components-gitpod-cli-app	`11.17% <ø> (ø)`
components-local-app-app-darwin-amd64	`?`
components-local-app-app-darwin-arm64	`?`
components-local-app-app-linux-amd64	`?`
components-local-app-app-linux-arm64	`?`
components-local-app-app-windows-386	`?`
components-local-app-app-windows-amd64	`?`
components-local-app-app-windows-arm64	`?`
components-supervisor-app	`35.12% <0.00%> (?)`

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
components/supervisor/pkg/supervisor/supervisor.go	`5.95% <0.00%> (ø)`
components/local-app/pkg/auth/pkce.go
components/local-app/pkg/auth/auth.go
components/supervisor/pkg/ports/served-ports.go	`80.76% <0.00%> (ø)`
components/supervisor/pkg/terminal/ring-buffer.go	`45.65% <0.00%> (ø)`
components/supervisor/pkg/ports/ports-config.go	`80.00% <0.00%> (ø)`
components/supervisor/pkg/supervisor/tasks.go	`58.56% <0.00%> (ø)`
...mponents/supervisor/pkg/supervisor/notification.go	`83.95% <0.00%> (ø)`
components/supervisor/pkg/supervisor/services.go	`21.52% <0.00%> (ø)`
components/supervisor/pkg/supervisor/ssh.go	`0.00% <0.00%> (ø)`
... and 12 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 42d529c...c8ec22e. Read the comment docs.

components/ide/code/extensionhelper/main.go

jeanp413 · 2022-02-18T15:15:04Z

@akosyakov we should keep supporting GITPOD_EXTERNAL_EXTENSIONS variable right? As typefox folks are using it context
cc @mustard-mh

akosyakov · 2022-02-18T15:16:41Z

@jeanp413 for now we should preserve the same semantic like in in customisations to VS Code Server. For cleaning up let's file a separate issue and it will need some research and planning.

components/ide/code/extensionhelper/main.go

mustard-mh · 2022-02-18T21:07:37Z

Updated, need re-review 😄

components/ide/code/extensionhelper/main.go

akosyakov · 2022-02-19T05:52:37Z

@jeanp413 @mustard-mh I found #4630 which I filed last year. I think this PR goes in right direction if it does not rely on env vars anymore. I think it is fine to remove both GITPOD_EXTERNAL_EXTENSIONS and GITPOD_RESOLVED_EXTENSIONS. They were never intended to be used for other purposes. External extensions means VSIX files from .gitpod.yml. We already cover them with this PR.

We should do #4630 as a follow-up after all workspaces are running this PR. It is important since it will improve startup of workspace on the server.

akosyakov · 2022-02-19T05:57:00Z

@mustard-mh @jeanp413 the trouble with this PR that it will harm perceived performance, before a user could access VS Code UI before actually content was loaded or extensions are installed, now everything will be delayed. Is not there a way to start a server without extensions and then install them in parallel?

It is in regards to .gitpod.yml parsing and downloading vsix files (since it requires content ready), for other cases like adding built-in extensions it is alright. Could you check whether WorkspaceContext already has a info about initially parser gitpod.yml maybe we are better to fetch it instead of parsing .gitpod.yml from the disk?

mustard-mh · 2022-02-19T07:34:53Z

We can start code first then command extentions management to install extensions

mustard-mh · 2022-02-19T07:39:43Z

But we should wait until code exthost ready, if cli command didn't await
cc @akosyakov @jeanp413

mustard-mh · 2022-02-19T08:15:07Z

Extensions that supervisor provided, only serve once and it's reading .gitpod.yml file with github API, and save it to DB.

If we use it, user start and stop worksapces to trigger extensions intall not work

So we should

🅰️

Start code first
Read .gitpod.yml from disk after content init ready
And after exthost ready, install extensions. Wait by watch vscode.lock file? Or @jeanp413 have other ideas?

🅱️

Save extensions again in DB before workspace stopped, (need discuss with WebApp team why it designed like this, and if changes break somthing)
Install extensions
(a) Start code with install extensions
(b) Same with 🅰️ step3

💡 Other ideas...

cc @akosyakov

akosyakov · 2022-02-19T14:54:10Z

I think .gitpod.yml already parsed and stored in db as a part of Workspace. We could fetch it from the server with getWorkspace or better server could install an env var like GITPOD_VSCODE_CONFIG. So we can access it faster. It will remove requirements to wait for content load and we could immediately start server. We will still need to wait for vsix files to download, but it should be rather a corner case.

jeanp413 · 2022-02-19T15:04:48Z

I was thinking about this yesterday too, I'll write some questions I did myself:

What does contentAvailable mean?, does it mean the whole /workspace folder is mounted/restored? does it mean only /workspace/repo folder is available after cloning? does it mean both?
If it means whole /workspace is restored then we really want to wait for it as .vscode-remote is inside /workspace folder
If not, then I think we should not wait for it (I think we already do this in vscode workbench.ts right?) nor reading .gitpod.yml from filesystem as it will delay server startup, I see the server already knows .gitpod.yml contents by using github/gitlab/etc filesystem API, so I would vote for keep using env variables or calling server api to get it's contents (I think we do this to create task terminals?)
Regarding installing extensions from URL, we first need to download it so we can pass a path to the vsix file to the server cli, but the downloading will delay server startup too, so I think we should look into other options:
- maybe create an internal task that uses gp await 23000 to ensure server started, download them and install them using server cli
- install them using gitpod extension once extension host started

akosyakov · 2022-02-19T15:11:54Z

If it means whole /workspace is restored then we really want to wait for it as .vscode-remote is inside /workspace folder

It is a good point, contentAvailable means that everything under /workspace was restored from backup for instance. 😬

Maybe we could move it somehow in server? i.e. to delay file system access internally?

If not, then I think we should not wait for it (I think we already do this in vscode workbench.ts right?) nor reading .gitpod.yml from filesystem as it will delay server startup, I see the server already knows .gitpod.yml contents by using github/gitlab/etc filesystem API, so I would vote for keep using env variables or calling server api to get it's contents (I think we do this to create task terminals?)

I think we still should do it in regardless of contentAvailable via env var is fastest.

Regarding installing extensions from URL, we first need to download it so we can pass a path to the vsix file to the server cli, but the downloading will delay server startup too, so I think we should look into other options:

I think we should ignore it for now and delay. I don't think that it is very popular feature. Maybe we should add some analytics for server startup time and usage of vsix files in .gitpod.yml?

jeanp413 · 2022-02-19T15:22:47Z

Maybe we could move it somehow in server? i.e. to delay file system access internally?

You mean /ide folder? does it persist data? if not it would be the same as moving it back to home folder, nice thing about having it in workspace folder is that we can look into logs of previous run and extensions already installed on workspace restart

I think we still should do it in regardless of contentAvailable via env var is fastest.

Agree 👍

I think we should ignore it for now and delay. I don't think that it is very popular feature. Maybe we should add some analytics for server startup time and usage of vsix files in .gitpod.yml?

👍 to adding some analytics, also small thing about options I listed is we don't need to worry about race condition in await extensionsReady in server on startup

akosyakov · 2022-02-19T15:37:41Z

You mean /ide folder? does it persist data? if not it would be the same as moving it back to home folder, nice thing about having it in workspace folder is that we can look into logs of previous run and extensions already installed on workspace restart

As for now only /workspace folder is persisted. I meant in VS Code Server remote file system service, but it will require changing ms code.

Let's start with contentAvailable check but in parallel with resolving extensions? Plus adding analytics for startup times with info about extensions in .gitpod.yml. In one month we could make a better decision then.

jeanp413 · 2022-02-19T19:03:47Z

Let's start with contentAvailable check but in parallel with resolving extensions?

If we resolve extensions via env variable, contentAvailable check should be done in supervisor?

components/ide/code/codehelper/main.go

components/supervisor/pkg/supervisor/supervisor.go

jeanp413 · 2022-02-26T03:03:24Z

components/ide/code/codehelper/main.go

+		log.WithError(err).WithField("wsInfo", wsInfo).WithField("cstate", contentStatus).Error("resolve workspace info failed")
+		return
+	}
+	log.WithField("cost", time.Now().Local().Sub(startTime).Milliseconds()).Info("content available")


Where can I see this logs?

Open a workspace in this preview env, get URL like https://mustardmh-test-fibq5bqd3jm.ws-dev.hw-code-flag.staging.gitpod-dev.com/, copy prefix mustardmh-test-fibq5bqd3jm

Make sure you setup your GCP auth, see notion

Open a gitpod-io/gitpod workspace

Run kubens staging-hw-code-flag to switch to the namespace

Run kubectl get pods -l metaID=mustardmh-test-fibq5bqd3jm to get correct workspace

Run kubectl logs <workspace_pod_name> -f to access and follow the logs

Or you can go to GCP. Click correct workspace pod to access logs

jeanp413 · 2022-02-26T03:09:58Z

Also need to add --do-not-sync to the server arguments in both arrays to avoid syncing this extensions

mustard-mh · 2022-02-26T18:31:49Z

I can't find this flag --do-not-sync @jeanp413

jeanp413 · 2022-02-26T21:48:17Z

Not all options appear in help but it's there and it's used here

jeanp413 · 2022-02-28T17:32:21Z

LGTM, left a minor comment in the gp-code branch gitpod-io/openvscode-server@6d0747d#r67646084

@mustard-mh does this still needs to be done? if not could you squash so we can merge it

need to clean up custom insider for test and debug log for track

akosyakov · 2022-03-01T10:58:13Z

WORKSPACE.yaml

@@ -7,7 +7,7 @@ defaultArgs:
  jbMarketplacePublishTrigger: "false"
  publishToJBMarketplace: true
  localAppVersion: unknown
-  codeCommit: 0175d2fd7b619fb1c34dcf2a0e17a909acd1be3e
+  codeCommit: 6d0747dbcd685cfe47251d56a452b619c9da444b


It does not make it stable, right?

This is for test only

akosyakov · 2022-03-01T10:59:50Z

@mustard-mh @jeanp413 Please add a new event here: https://www.notion.so/gitpod/dd900d5177944fc38d22bb8c7d3d2cce?v=54cdae8ff0f54b4d8235429ab9893e49

akosyakov · 2022-03-01T11:22:59Z

components/supervisor/pkg/supervisor/supervisor.go

+		Timestamp           int64  `json:"timestamp,omitempty"`
+	}
+	trackFn := func(ctx context.Context, gitpodService *gitpod.APIoverJSONRPC, cfg *Config, kind string) {
+		gitpodService.TrackEvent(ctx, &gitpod.RemoteTrackMessage{


it does not return err or something like that to log?

I print this error in a new commit, but when this reproduce, no error is returned and there is no useful information in the log, see mustardmh-test-aabar7hcoyk

akosyakov · 2022-03-01T12:50:41Z

Does it really fix #7863 ? I think it is just a clean up we did not really identify a root cause?

mustard-mh · 2022-03-01T13:08:14Z

@jeanp413 found out that there is some race when installing extensions internal chat

This is because our custom extension install code contains both id and URL of extension internal chat

This PR is related to #7863

Our plans to fix #7863 are:

Using server arguments (This PR)
Remove our custom extension install code in openvscode-server

cc @akosyakov

akosyakov · 2022-03-01T13:15:52Z

Yeah, I seen internal chat, ok, let's close it for now and keep eyes on it.

jeanp413

LGTM

jeanp413 · 2022-03-01T17:45:36Z

/unhold

mustard-mh · 2022-03-01T20:51:16Z

I print this error in a new commit, but when this reproduce, no error is returned and there is no useful information in the log, see mustardmh-test-aabar7hcoyk

File an issue about segment #8516

mustard-mh requested a review from a team February 18, 2022 13:18

roboquat added release-note size/L labels Feb 18, 2022

github-actions bot added the team: IDE label Feb 18, 2022

akosyakov reviewed Feb 18, 2022

View reviewed changes

components/ide/code/extensionhelper/main.go Outdated Show resolved Hide resolved

akosyakov reviewed Feb 18, 2022

View reviewed changes

components/ide/code/extensionhelper/main.go Outdated Show resolved Hide resolved

akosyakov requested a review from jeanp413 February 18, 2022 14:59

jeanp413 requested changes Feb 18, 2022

View reviewed changes

components/ide/code/extensionhelper/main.go Outdated Show resolved Hide resolved

mustard-mh force-pushed the hw/code-flag branch from 83f51b6 to 5cd78ab Compare February 18, 2022 20:30

roboquat added size/XXL and removed size/L labels Feb 18, 2022

mustard-mh force-pushed the hw/code-flag branch from 5cd78ab to 0735f11 Compare February 18, 2022 20:35

mustard-mh requested review from jeanp413 and akosyakov February 18, 2022 21:07

akosyakov reviewed Feb 19, 2022

View reviewed changes

components/ide/code/extensionhelper/main.go Outdated Show resolved Hide resolved

jeanp413 reviewed Feb 26, 2022

View reviewed changes

components/ide/code/codehelper/main.go Outdated Show resolved Hide resolved

jeanp413 reviewed Feb 26, 2022

View reviewed changes

components/ide/code/codehelper/main.go Outdated Show resolved Hide resolved

jeanp413 reviewed Feb 26, 2022

View reviewed changes

components/supervisor/pkg/supervisor/supervisor.go Outdated Show resolved Hide resolved

jeanp413 reviewed Feb 26, 2022

View reviewed changes

akosyakov reviewed Mar 1, 2022

View reviewed changes

mustard-mh force-pushed the hw/code-flag branch from 8d31c75 to bf60177 Compare March 1, 2022 17:00

mustard-mh added 2 commits March 1, 2022 17:03

[ide] use code flag to install extensions

acd580c

[supervisor] add readiness track

c8ec22e

mustard-mh force-pushed the hw/code-flag branch from bf60177 to c8ec22e Compare March 1, 2022 17:04

jeanp413 approved these changes Mar 1, 2022

View reviewed changes

roboquat removed the do-not-merge/hold label Mar 1, 2022

roboquat merged commit c6cc1ab into main Mar 1, 2022

roboquat deleted the hw/code-flag branch March 1, 2022 17:46

mustard-mh mentioned this pull request Mar 1, 2022

Sometimes the track event is not shown in the segment #8516

Closed

roboquat added deployed: IDE IDE change is running in production deployed Change is completely running in production labels Mar 2, 2022

jeanp413 mentioned this pull request Mar 4, 2022

[vscode-web] extension host process times out on startup #7863

Closed

mustard-mh mentioned this pull request Mar 29, 2022

One frame running phase for every workspace start #8990

Closed

mustard-mh mentioned this pull request Jul 29, 2022

[supervisor]: gitpod config changed analytics #11653

Merged

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[ide] use code flag to install extensions #8313

[ide] use code flag to install extensions #8313

mustard-mh commented Feb 18, 2022 •

edited

Loading

codecov bot commented Feb 18, 2022 •

edited

Loading

jeanp413 commented Feb 18, 2022

akosyakov commented Feb 18, 2022

mustard-mh commented Feb 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

akosyakov commented Feb 19, 2022

akosyakov commented Feb 19, 2022 •

edited

Loading

mustard-mh commented Feb 19, 2022

mustard-mh commented Feb 19, 2022

mustard-mh commented Feb 19, 2022 •

edited

Loading

akosyakov commented Feb 19, 2022 •

edited

Loading

jeanp413 commented Feb 19, 2022

akosyakov commented Feb 19, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Feb 19, 2022 •

edited

Loading

akosyakov commented Feb 19, 2022

jeanp413 commented Feb 19, 2022

jeanp413 Feb 26, 2022

mustard-mh Feb 26, 2022

mustard-mh Feb 26, 2022

jeanp413 commented Feb 26, 2022

mustard-mh commented Feb 26, 2022 •

edited

Loading

jeanp413 commented Feb 26, 2022 •

edited

Loading

jeanp413 commented Feb 28, 2022

akosyakov Mar 1, 2022

mustard-mh Mar 1, 2022

akosyakov commented Mar 1, 2022

akosyakov Mar 1, 2022

mustard-mh Mar 1, 2022

akosyakov commented Mar 1, 2022

mustard-mh commented Mar 1, 2022 •

edited

Loading

akosyakov commented Mar 1, 2022

jeanp413 left a comment

jeanp413 commented Mar 1, 2022

mustard-mh commented Mar 1, 2022

[ide] use code flag to install extensions #8313

[ide] use code flag to install extensions #8313

Conversation

mustard-mh commented Feb 18, 2022 • edited Loading

Description

Related Issue(s)

How to test

Release Notes

Documentation

codecov bot commented Feb 18, 2022 • edited Loading

Codecov Report

jeanp413 commented Feb 18, 2022

akosyakov commented Feb 18, 2022

mustard-mh commented Feb 18, 2022 • edited by werft-gitpod-dev-com bot Loading

akosyakov commented Feb 19, 2022

akosyakov commented Feb 19, 2022 • edited Loading

mustard-mh commented Feb 19, 2022

mustard-mh commented Feb 19, 2022

mustard-mh commented Feb 19, 2022 • edited Loading

akosyakov commented Feb 19, 2022 • edited Loading

jeanp413 commented Feb 19, 2022

akosyakov commented Feb 19, 2022 • edited by werft-gitpod-dev-com bot Loading

jeanp413 commented Feb 19, 2022 • edited Loading

akosyakov commented Feb 19, 2022

jeanp413 commented Feb 19, 2022

jeanp413 Feb 26, 2022

Choose a reason for hiding this comment

mustard-mh Feb 26, 2022

Choose a reason for hiding this comment

mustard-mh Feb 26, 2022

Choose a reason for hiding this comment

jeanp413 commented Feb 26, 2022

mustard-mh commented Feb 26, 2022 • edited Loading

jeanp413 commented Feb 26, 2022 • edited Loading

jeanp413 commented Feb 28, 2022

akosyakov Mar 1, 2022

Choose a reason for hiding this comment

mustard-mh Mar 1, 2022

Choose a reason for hiding this comment

akosyakov commented Mar 1, 2022

akosyakov Mar 1, 2022

Choose a reason for hiding this comment

mustard-mh Mar 1, 2022

Choose a reason for hiding this comment

akosyakov commented Mar 1, 2022

mustard-mh commented Mar 1, 2022 • edited Loading

akosyakov commented Mar 1, 2022

jeanp413 left a comment

Choose a reason for hiding this comment

jeanp413 commented Mar 1, 2022

mustard-mh commented Mar 1, 2022

mustard-mh commented Feb 18, 2022 •

edited

Loading

codecov bot commented Feb 18, 2022 •

edited

Loading

mustard-mh commented Feb 18, 2022 •

edited by werft-gitpod-dev-com bot

Loading

akosyakov commented Feb 19, 2022 •

edited

Loading

mustard-mh commented Feb 19, 2022 •

edited

Loading

akosyakov commented Feb 19, 2022 •

edited

Loading

akosyakov commented Feb 19, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jeanp413 commented Feb 19, 2022 •

edited

Loading

mustard-mh commented Feb 26, 2022 •

edited

Loading

jeanp413 commented Feb 26, 2022 •

edited

Loading

mustard-mh commented Mar 1, 2022 •

edited

Loading