[bitbucket-server] support for projects and prebuilds #8896

AlexTugarev · 2022-03-19T05:23:57Z

Description

This enables prebuilds for Bitbucket Server integrations.

Breaking Changes

Admin scopes are required to make the webhook creation work. Full set of scopes to enable for the OAuth App: PUBLIC_REPOS, REPO_READ, REPO_WRITE, REPO_ADMIN, PROJECT_ADMIN
Due to a change in Identity.authName mapping, users need to re-login to get that updated, otherwise repository services wont be able to read permissions properly.

Related Issue(s)

Fixes #8455

How to test

Make sure to create a new account in the preview environment
Make sure to create an account in https://bitbucket.gitpod-self-hosted.com (hint: log in as "roboquat" with the credentials in 1password, then create a new user account for yourself)
Try to add repos with admin permission as projects. Test with personal repos as well as repos in repos.
- Verify that webhooks are created on the BBS.
Open workspaces for those repos and make changes.
- Verify that prebuilds are triggered.
- Verify that prebuilds ran on the proper branch for example.

TODO

manage and use webhook secret to verify events

Known issues or out of scope

No support for Issue and PR contexts
No UI for setting up a BBS integration (this has to be done via config at the moment)

Release Notes

Adding support for Projects and Prebuilds for Bitbucket Server.

csweichel · 2022-03-29T11:30:54Z

Awesome work - we really need this change.
/cc @geropl what do we need to do to get this merged?

geropl · 2022-03-30T06:57:33Z

@AlexTugarev Can you raise this within the team to get a review this week? Maybe from Jan?

jldec · 2022-03-30T08:15:19Z

/werft run

👍 started the job as gitpod-build-at-bbs.14

jldec · 2022-03-30T09:51:27Z

Validated adding integration as a user by opening a workspace on a prefixed URL.
Works as expected ✅

layout of the scopes is a bit wide, but not a big problem (look similar to existing BB integration)

jldec · 2022-03-30T09:59:06Z

resolved

New project flow fails to fetch branches (upper-case/lower-case issue?) for repo at
https://bitbucket.gitpod-self-hosted.com/projects/JLDEC/repos/jldec-repo-march-30/browse

jldec · 2022-03-30T10:20:16Z

Project on personal repo is able to fetch branches. ✅
Branch list formatting is off (because of avatar - see similar issue for GHE in #8755)

jldec · 2022-03-30T10:21:44Z

Project on same personal repo shows error when looking for .gitpod.yml under configuration page.

jldec · 2022-03-30T11:36:02Z

error in ~username context urls. ACK'd, but fixup is not yet pushed to not disturb reviewers.

The button to start a new workspace after creating a new project on a personal repo, appears to create a broken context url since it uses the clone url path scm/~jldec/test-repo2.git. Replacing that context url with the actual url path from BBS does work users/jldec/repos/test-repo2/browse

jldec · 2022-03-30T11:38:12Z

fixed

There does not appear to be any installed webhook on any of the repos for which I have created a project.

AlexTugarev · 2022-03-31T16:31:13Z

@jldec, please check again 🙏🏻

Also, pay attention to the Breaking Changes, esp. to the re-login step. Casing of the auth name was wrong in my account. I've added tests for that, it should meet the expectations in both directions and match the permission queries.

In your case, which is assumingely uncommon, there is a project and a user account named equally. Something you would not expect to be possible. I've change the workaround to try projects first, this should resolve the issues from above as well.

Let's see if we need a 3rd round 🤞🏻

AlexTugarev · 2022-04-01T06:47:37Z

Just found an issue with the New Project wizard, which was actually reported several times, but we seem to not have found a repro for: User.identities contain a Gitpod identity, which AFAIU is used to authorize webhook event handler (i.e. for prebuild events) and this is not handled nor expected.

I'll make a drive-by-change to exclude them from the results of GitpodService.getAuthProviders as it's likely unexpected/unhandled in other places.

easyCZ · 2022-04-01T07:42:37Z

AT: that's a configuration issue no BBS. they are rendering an error page. I've improved the error message propagation, so if you try to start a workspace, you'll see a helpful message. BTW: we've discussed a general error component in the dashboard to show system/project errors, THIS could be used to provide instant feedback on the branches overview.

I'm unable to see prebuilds for my project:

Steps:

Create a bitbucket project
Import https://gitlab.com/gitpod-milan/gitpod-large-image as a repo
Create gitpod project
Navigate to Branches - nothing visible

5. Try `Run prebuild` - nothing happens, request response contains

Request triggerPrebuild failed with message: 404 / Not Found

Project Settings also seem to not have access to the repo properly

easyCZ · 2022-04-01T07:50:01Z

components/server/ee/src/prebuilds/bitbucket-server-service.ts

+    @inject(BitbucketServerContextParser) protected contextParser: BitbucketServerContextParser;
+
+    async getRepositoriesForAutomatedPrebuilds(user: User): Promise<ProviderRepository[]> {
+        const repos = await this.api.getRepos(user, { limit: 100, permission: "REPO_ADMIN" });


[nit] Would be good to comment that we're requesting REPO_ADMIN because that's the minimum role required to create webhooks.

easyCZ · 2022-04-01T07:52:07Z

components/server/ee/src/prebuilds/bitbucket-server-service.ts

+        if (permission && ["REPO_ADMIN", "PROJECT_ADMIN"].indexOf(permission) !== -1) {
+            return true;
+        }


[nit] It can be sometimes helpful to communicate the reasoning for a particular check though a method name. For example:

if (hasPermissionToCreateWebhooks(permission) {

components/server/src/bitbucket-server/bitbucket-server-api.ts

AlexTugarev · 2022-04-01T08:15:49Z

@easyCZ, please invite me to your project to reproduce #8896 (comment)

jankeromnes · 2022-04-01T09:58:47Z

Many thanks for implementing BBS! 🏭

Gave this a spin, and was able to successfully add a Bitbucket project! 🎉

Minor: The UI looks a little bit unpolished still --

1. No BBS provider icon? 2. Should BBS really be visible by default on Login?	3. Uppercase scopes? 4. Without scope descriptions?	5. Why the `~` in my name? 6. Shouldn't the project name be vertically centered?

1. Include a button icon on the login page for Bitbucket Server provider #8603
2. for this preview env the config is injected manually(!) as there is no alternative ATM.
2. Configuration UI to register Bitbucket Server instance #9010
3. 4. 🛹. also, we need to figure out if scope elevation is wanted, or if PROJECT_ADMIN is required.
5. ~username is to separate from username which would point to a non-personal, i.e. regular, project. weird.

Also, I wasn't able to start a workspace for some reason:

AT: this is tackled in most recent changes. see comments bellow.

Add empty repo, click on New Workspace	Fails with 404? 😳

AlexTugarev · 2022-04-01T10:16:23Z

@jankeromnes thanks for asking on the ~

Why the ~ in my name?

That indicated personal projects. As @jldec managed to create a personal and a regular project of the same name, I'd consider that a "feature"! Seriously, I've no idea of how to treat that properly. Ambiguity of project name is hard. Let's have the ~ until we learn, how if there is a better way to handle it. Having that said, I see no way in removing it without breaking the other project kinds. Ok?

jankeromnes · 2022-04-01T10:16:45Z

AT: that's the "no default branch" situation to be resolved on BBS. latest change introduced better error message propagation, which explains the problem.

Okay, I was now able to push changes to my BBS repository. Unfortunately, no Prebuilds are being triggered.

When I try the Run Prebuild button, it fails with the same 404 message. I guess this is the ~ problem as well?

easyCZ

Thanks, I've tested this as far as I can and seems to work. There are still some rough edges on the integration (largely due to BitBucket Server being a bit crazy) but I think this is a good enough 🛹 .

Code-wise, looks sensible but I don't yet have full context to know if there are skeletons.

easyCZ · 2022-04-01T14:37:07Z

/hold

jankeromnes · 2022-04-01T15:22:29Z

Nice! Feels like this is closing in on the finish line. 🏁

I've noticed a few remaining rough edges though:

This button looks awkward (in "New Project", but I guess it's the same as Include a button icon on the login page for Bitbucket Server provider #8603)

When I add my project "YOLO" (https://bitbucket.gitpod-self-hosted.com/scm/~jan/yolo.git) as a new project, it fails with:

(The project is actually still created, but I guess without a webhook, because prebuilds do not trigger automatically on pushes)

jldec · 2022-04-04T05:41:20Z

I was a bit surprised to get the extra login button when I start a totally new session - if we ship this and configure our BBS integration for testing, we don't want everyone else (non-gitpod) who lands on gitpod.io for the first time to see the extra button. Can you confirm that this will not be the case for SaaS?

jldec · 2022-04-04T05:45:30Z

Regarding the button formatting - the lack of image, and the way the button text bleeds to the edges does look unpolished. I understand that it's cosmestic, but the first impression users see is important as well.

AlexTugarev · 2022-04-04T06:14:12Z

Can you confirm that this will not be the case for SaaS?

@jldec, don't worry, one need to do a backflip to get it running on self-hosted, there is no path for this to get initialized on SaaS.

AlexTugarev · 2022-04-04T06:21:28Z

Thanks @jankeromnes, overlooked your issue with the 404 error. I could reproduce it on a fork. Trying to get this sorted out before proceeding.

edit

Awesome! That was indeed a bug in "can install webhook" logic for repos in personal accounts. There is an API which should not be called for personal accounts, but it was.

I changed and updated the branch.

AlexTugarev · 2022-04-04T08:24:25Z

/unhold

roboquat added release-note do-not-merge/work-in-progress size/XXL labels Mar 19, 2022

AlexTugarev removed the do-not-merge/work-in-progress label Mar 28, 2022

AlexTugarev marked this pull request as ready for review March 28, 2022 13:24

AlexTugarev requested a review from a team March 28, 2022 13:24

github-actions bot added the team: webapp Issue belongs to the WebApp team label Mar 28, 2022

AlexTugarev force-pushed the at/bbs branch from 2b61f1c to ad98675 Compare March 31, 2022 14:08

easyCZ reviewed Apr 1, 2022

View reviewed changes

components/server/src/bitbucket-server/bitbucket-server-api.ts Show resolved Hide resolved

easyCZ reviewed Apr 1, 2022

View reviewed changes

components/server/src/bitbucket-server/bitbucket-server-api.ts Show resolved Hide resolved

easyCZ reviewed Apr 1, 2022

View reviewed changes

components/server/src/bitbucket-server/bitbucket-server-api.ts Show resolved Hide resolved

AlexTugarev requested a review from jankeromnes April 1, 2022 09:17

AlexTugarev requested a review from jldec April 1, 2022 13:30

easyCZ previously approved these changes Apr 1, 2022

View reviewed changes

roboquat added the do-not-merge/hold label Apr 1, 2022

AlexTugarev dismissed easyCZ’s stale review via cc5df03 April 4, 2022 07:27

AlexTugarev force-pushed the at/bbs branch from 086c15d to cc5df03 Compare April 4, 2022 07:27

[bitbucket-server] support for projects and prebuilds

a676418

AlexTugarev force-pushed the at/bbs branch from cc5df03 to a676418 Compare April 4, 2022 08:07

roboquat removed the do-not-merge/hold label Apr 4, 2022

easyCZ approved these changes Apr 4, 2022

View reviewed changes

roboquat merged commit 76b51bc into main Apr 4, 2022

roboquat deleted the at/bbs branch April 4, 2022 08:37

roboquat added deployed: webapp Meta team change is running in production deployed Change is completely running in production labels Apr 5, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[bitbucket-server] support for projects and prebuilds #8896

[bitbucket-server] support for projects and prebuilds #8896

AlexTugarev commented Mar 19, 2022 •

edited by jankeromnes

Loading

csweichel commented Mar 29, 2022

geropl commented Mar 30, 2022

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

AlexTugarev commented Mar 31, 2022 •

edited

Loading

AlexTugarev commented Apr 1, 2022

easyCZ commented Apr 1, 2022 •

edited by AlexTugarev

Loading

easyCZ Apr 1, 2022

easyCZ Apr 1, 2022

AlexTugarev commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

AlexTugarev commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

easyCZ left a comment

easyCZ commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

jldec commented Apr 4, 2022

jldec commented Apr 4, 2022

AlexTugarev commented Apr 4, 2022

AlexTugarev commented Apr 4, 2022 •

edited

Loading

AlexTugarev commented Apr 4, 2022

[bitbucket-server] support for projects and prebuilds #8896

[bitbucket-server] support for projects and prebuilds #8896

Conversation

AlexTugarev commented Mar 19, 2022 • edited by jankeromnes Loading

Description

Breaking Changes

Related Issue(s)

How to test

TODO

Known issues or out of scope

Release Notes

csweichel commented Mar 29, 2022

geropl commented Mar 30, 2022

jldec commented Mar 30, 2022 • edited by werft-gitpod-dev-com bot Loading

jldec commented Mar 30, 2022 • edited by werft-gitpod-dev-com bot Loading

jldec commented Mar 30, 2022 • edited by AlexTugarev Loading

jldec commented Mar 30, 2022 • edited by werft-gitpod-dev-com bot Loading

jldec commented Mar 30, 2022

jldec commented Mar 30, 2022 • edited by AlexTugarev Loading

jldec commented Mar 30, 2022 • edited by AlexTugarev Loading

AlexTugarev commented Mar 31, 2022 • edited Loading

AlexTugarev commented Apr 1, 2022

easyCZ commented Apr 1, 2022 • edited by AlexTugarev Loading

easyCZ Apr 1, 2022

Choose a reason for hiding this comment

easyCZ Apr 1, 2022

Choose a reason for hiding this comment

AlexTugarev commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 • edited by AlexTugarev Loading

AlexTugarev commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 • edited by AlexTugarev Loading

easyCZ left a comment

Choose a reason for hiding this comment

easyCZ commented Apr 1, 2022

jankeromnes commented Apr 1, 2022 • edited by AlexTugarev Loading

jldec commented Apr 4, 2022

jldec commented Apr 4, 2022

AlexTugarev commented Apr 4, 2022

AlexTugarev commented Apr 4, 2022 • edited Loading

edit

AlexTugarev commented Apr 4, 2022

AlexTugarev commented Mar 19, 2022 •

edited by jankeromnes

Loading

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

jldec commented Mar 30, 2022 •

edited by werft-gitpod-dev-com bot

Loading

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

jldec commented Mar 30, 2022 •

edited by AlexTugarev

Loading

AlexTugarev commented Mar 31, 2022 •

edited

Loading

easyCZ commented Apr 1, 2022 •

edited by AlexTugarev

Loading

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

jankeromnes commented Apr 1, 2022 •

edited by AlexTugarev

Loading

AlexTugarev commented Apr 4, 2022 •

edited

Loading