From 64ba888da5d3c54d8e325d4e5ca600bf3f4ddaa8 Mon Sep 17 00:00:00 2001 From: Gero Posmyk-Leinemann Date: Thu, 15 May 2025 14:01:55 +0000 Subject: [PATCH 1/2] [memory-bank] Overhaul based on current usage - merge learning-journal.md into existing docs - introduce public-api.md - introduce components.md for more reliable lookup --- memory-bank/components.md | 49 +++++ memory-bank/components/content-service-api.md | 7 - memory-bank/components/gitpod-db.md | 2 +- memory-bank/components/gitpod-protocol.md | 1 + memory-bank/components/ide-metrics-api.md | 9 - memory-bank/components/ide-service-api.md | 8 - memory-bank/components/image-builder-api.md | 8 - memory-bank/components/local-app-api.md | 7 - memory-bank/components/public-api-server.md | 2 +- memory-bank/components/public-api.md | 191 ++++++++++++++++++ memory-bank/components/registry-facade-api.md | 7 - memory-bank/components/server.md | 10 +- memory-bank/components/spicedb.md | 16 +- memory-bank/components/supervisor-api.md | 8 - memory-bank/components/usage-api.md | 8 - memory-bank/components/ws-daemon-api.md | 8 - memory-bank/components/ws-manager-api.md | 8 - .../components/ws-manager-bridge-api.md | 7 - memory-bank/systemPatterns.md | 167 +++++++++++++-- memory-bank/techContext.md | 3 +- 20 files changed, 403 insertions(+), 123 deletions(-) create mode 100644 memory-bank/components.md create mode 100644 memory-bank/components/public-api.md diff --git a/memory-bank/components.md b/memory-bank/components.md new file mode 100644 index 00000000000000..0b6aabcb321dd3 --- /dev/null +++ b/memory-bank/components.md @@ -0,0 +1,49 @@ +# Gitpod Components Index + +This file serves as index for the per-component documentation in individual md files in the `memory-bank/components` directory. + +- [server](components/server.md) +- [content-service](components/content-service.md) +- [ws-manager-mk2](components/ws-manager-mk2.md) +- [ws-daemon](components/ws-daemon.md) +- [image-builder-mk3](components/image-builder-mk3.md) +- [content-service-api](components/content-service-api.md) +- [ws-manager-api](components/ws-manager-api.md) +- [ws-daemon-api](components/ws-daemon-api.md) +- [image-builder-api](components/image-builder-api.md) +- [supervisor-api](components/supervisor-api.md) +- [registry-facade-api](components/registry-facade-api.md) +- [ide-service-api](components/ide-service-api.md) +- [ide-metrics-api](components/ide-metrics-api.md) +- [ws-manager-bridge-api](components/ws-manager-bridge-api.md) +- [public-api](components/public-api.md) +- [usage-api](components/usage-api.md) +- [local-app-api](components/local-app-api.md) +- [dashboard](components/dashboard.md) +- [ide](components/ide.md) +- [proxy](components/proxy.md) +- [registry-facade](components/registry-facade.md) +- [blobserve](components/blobserve.md) +- [ipfs](components/ipfs.md) +- [openvsx-proxy](components/openvsx-proxy.md) +- [scheduler-extender](components/scheduler-extender.md) +- [gitpod-db](components/gitpod-db.md) +- [gitpod-protocol](components/gitpod-protocol.md) +- [supervisor](components/supervisor.md) +- [workspacekit](components/workspacekit.md) +- [ws-proxy](components/ws-proxy.md) +- [ide-proxy](components/ide-proxy.md) +- [ide-service](components/ide-service.md) +- [ide-metrics](components/ide-metrics.md) +- [docker-up](components/docker-up.md) +- [common-go](components/common-go.md) +- [service-waiter](components/service-waiter.md) +- [node-labeler](components/node-labeler.md) +- [scrubber](components/scrubber.md) +- [spicedb](components/spicedb.md) +- [public-api-server](components/public-api-server.md) +- [gitpod-cli](components/gitpod-cli.md) +- [local-app](components/local-app.md) +- [ws-manager-bridge](components/ws-manager-bridge.md) +- [image-builder-bob](components/image-builder-bob.md) +- [usage](components/usage.md) diff --git a/memory-bank/components/content-service-api.md b/memory-bank/components/content-service-api.md index f31f07cd434a07..91437b3f55cbf0 100644 --- a/memory-bank/components/content-service-api.md +++ b/memory-bank/components/content-service-api.md @@ -98,13 +98,6 @@ This script performs the following actions: - Generates TypeScript code - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`, which provides common functionality for all API components: - -- `install_dependencies`: Installs required protoc plugins -- `protoc_buf_generate`: Uses the `buf` tool to generate code based on the configuration in `buf.gen.yaml` -- `update_license`: Updates license headers in generated files - ### Building After Code Generation After regenerating the code, you may need to rebuild components that depend on the Content Service API. This typically involves: diff --git a/memory-bank/components/gitpod-db.md b/memory-bank/components/gitpod-db.md index 65df6941f3bd03..c221027c0de88c 100644 --- a/memory-bank/components/gitpod-db.md +++ b/memory-bank/components/gitpod-db.md @@ -49,7 +49,7 @@ The Gitpod DB component defines numerous entities that map to database tables, i - **User**: User accounts and profiles - **Workspace**: Workspace metadata and configuration - **WorkspaceInstance**: Running workspace instances -- **Team**: Team organization and membership +- **Team**: Represents an "Organization" within Gitpod, storing its core details, membership, and organization-level state. The entity is named `DBTeam` for historical reasons. - **Project**: Project configuration and settings - **Identity**: User identity and authentication - **Token**: Authentication tokens and credentials diff --git a/memory-bank/components/gitpod-protocol.md b/memory-bank/components/gitpod-protocol.md index 6c76e24d2605ba..8ba8a7129a52a7 100644 --- a/memory-bank/components/gitpod-protocol.md +++ b/memory-bank/components/gitpod-protocol.md @@ -50,6 +50,7 @@ The Gitpod Protocol defines numerous core data structures, including: - `User`: User account information - `Identity`: Authentication provider identity - `Token`: Authentication tokens +- `Organization` (defined in `src/teams-projects-protocol.ts`): Represents a team/organization, including its settings - `GitpodToken`: API tokens - `AuthProviderInfo`: Authentication provider metadata - `AuthProviderEntry`: Authentication provider configuration diff --git a/memory-bank/components/ide-metrics-api.md b/memory-bank/components/ide-metrics-api.md index 876f30af9944d9..4cd3e9c6ee471d 100644 --- a/memory-bank/components/ide-metrics-api.md +++ b/memory-bank/components/ide-metrics-api.md @@ -86,15 +86,6 @@ This script performs the following actions: - Generates Java code - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh` and defines some custom functions: - -- `install_dependencies`: Installs required protoc plugins -- `local_go_protoc`: Generates Go code with specific include paths -- `go_protoc_gateway`: Generates gRPC Gateway code for REST endpoints -- `local_java_protoc`: Generates Java code -- `update_license`: Updates license headers in generated files - The IDE Metrics API is unique in that it generates both gRPC and REST API endpoints using the gRPC Gateway, and it also generates Java code for use in JetBrains IDE plugins. ### Building After Code Generation diff --git a/memory-bank/components/ide-service-api.md b/memory-bank/components/ide-service-api.md index d9c45233d2af9a..91f97208d5db54 100644 --- a/memory-bank/components/ide-service-api.md +++ b/memory-bank/components/ide-service-api.md @@ -105,14 +105,6 @@ This script performs the following actions: - Generates TypeScript code using `ts_proto` - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `go_protoc`: Generates Go code -- `typescript_ts_protoc`: Generates TypeScript code using ts_proto -- `update_license`: Updates license headers in generated files - The IDE Service API generates TypeScript code using the ts_proto plugin, which creates more modern TypeScript interfaces compared to the standard protoc TypeScript generator. ### Building After Code Generation diff --git a/memory-bank/components/image-builder-api.md b/memory-bank/components/image-builder-api.md index 17b65354c85123..abf39db8c01a2c 100644 --- a/memory-bank/components/image-builder-api.md +++ b/memory-bank/components/image-builder-api.md @@ -109,14 +109,6 @@ This script performs the following actions: - Patches the generated TypeScript code for compatibility - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `go_protoc`: Generates Go code -- `typescript_protoc`: Generates TypeScript code -- `update_license`: Updates license headers in generated files - Additionally, the script: - Generates mock implementations using `mockgen` for testing - Patches the generated TypeScript code using a script from the content-service-api diff --git a/memory-bank/components/local-app-api.md b/memory-bank/components/local-app-api.md index a9f224a5cbf02d..b03aa81b6d2992 100644 --- a/memory-bank/components/local-app-api.md +++ b/memory-bank/components/local-app-api.md @@ -74,13 +74,6 @@ This script performs the following actions: - Generates Go code using `protoc-gen-go` and `protoc-gen-go-grpc` - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh` and defines some custom functions: - -- `install_dependencies`: Installs required protoc plugins -- `local_go_protoc`: Generates Go code with specific include paths for third-party dependencies -- `update_license`: Updates license headers in generated files - The Local App API has a dependency on the Supervisor API for tunnel visibility definitions, which is included in the protoc generation process. ### Building After Code Generation diff --git a/memory-bank/components/public-api-server.md b/memory-bank/components/public-api-server.md index 47539c862e91ef..e75d224f3fd8e3 100644 --- a/memory-bank/components/public-api-server.md +++ b/memory-bank/components/public-api-server.md @@ -120,7 +120,7 @@ The Public API Server is configured through a JSON configuration file: ## Integration Points The Public API Server integrates with: -1. **Gitpod Server**: For core Gitpod functionality +1. **Gitpod Server**: For core Gitpod functionality. The Public API Server often acts as a gRPC gateway, proxying requests to the Gitpod Server (TypeScript component) where the business logic for many `gitpod.v1` services (like `OrganizationService`) is implemented. 2. **Database**: For persistent storage 3. **Redis**: For caching and session management 4. **Billing Service**: For billing-related operations diff --git a/memory-bank/components/public-api.md b/memory-bank/components/public-api.md new file mode 100644 index 00000000000000..056b5e5f5855b7 --- /dev/null +++ b/memory-bank/components/public-api.md @@ -0,0 +1,191 @@ +# Public API + +## Overview +The Public API defines the gRPC interfaces for programmatic access to Gitpod functionality. It serves as the canonical way for external integrations, automation, and third-party tools to interact with Gitpod's core services. The API is structured into two packages (stable and experimental) with different compatibility guarantees and is designed to be backward compatible, well-documented, and follow modern API design principles. + +## Purpose +This API provides a standardized interface for: +- Programmatically managing workspaces (create, start, stop, delete) +- Accessing and managing user information +- Working with organizations and teams +- Managing projects and repositories +- Integrating with source code management systems +- Configuring editors and IDEs +- Authenticating via OpenID Connect +- Managing personal access tokens +- Automating Gitpod workflows + +## Architecture +The Public API is implemented as a set of gRPC services defined in Protocol Buffer files. These definitions are used to generate client and server code in Go, TypeScript, and Java. The API is exposed on `api.gitpod.io` or `api.` for Dedicated installations. + +The API is structured into two main packages: + +1. **Stable (v1)**: + - Located in `gitpod/v1/` + - Provides compatibility guarantees + - Services, calls, types, and fields are not removed without following a deprecation policy + - Services, calls, types, and fields are not renamed + - Non-successful responses are described exhaustively + - **Implementation**: Directly implemented in the server component using Connect + +2. **Experimental**: + - Located in `gitpod/experimental/v1/` + - Provides no compatibility guarantees + - May change frequently + - **Implementation**: Handled in the public-api-server component, and either: + - Implemented directly in Go + - Forwarded to the old websocket API in the server component + +## Implementation Patterns + +### Stable API Implementation +The stable API (v1) is implemented directly in the server component using Connect. This means: +- The server component handles the business logic for these API endpoints +- The implementation is in TypeScript using Connect + +### Experimental API Implementation +The experimental API is handled in the public-api-server component in two ways: + +1. **Direct Implementation**: + - Some services are implemented directly in Go within the public-api-server + - These implementations handle the business logic directly + - They may interact with the database or other services directly + +2. **Forwarded Implementation**: + - Other services forward requests to the old websocket API in the server component + - The public-api-server acts as a proxy, translating gRPC requests to websocket API calls + - The server component handles the actual business logic + - This approach is often used for functionality that already exists in the server component + +## Key Services + +### Stable API (v1) +All implemented in server component using Connect: +- WorkspaceService +- OrganizationService +- UserService +- TokenService +- SCMService +- AuthProviderService +- ConfigurationService +- EnvVarService +- InstallationService +- PrebuildService +- SSHService +- VerificationService + +### Experimental API +Implemented in public-api-server: +- WorkspacesService (Forwarded to server) +- TeamsService (Directly implemented in Go) +- ProjectsService (Forwarded to server) +- EditorService (Directly implemented in Go) +- IDEClientService (Directly implemented in Go) +- OIDCService (Directly implemented in Go) +- IdentityProviderService (Directly implemented in Go) +- TokensService (Forwarded to server) +- UserService (Forwarded to server) +- StatsService (Directly implemented in Go) + +## Communication Patterns +- The API uses gRPC for efficient, typed communication +- Connect is used for the stable API implementation +- Requests include authentication tokens for identifying the user +- Pagination is supported for listing operations +- Streaming is used for real-time updates (e.g., workspace status changes) +- Field masks are used to specify which fields to return or update + +### Implementation-specific Patterns +- **Stable API (v1)**: + - Requests are handled by the server component using Connect + - The public-api-server routes requests to the server component + +- **Experimental API**: + - Directly implemented services handle requests in the public-api-server + - Forwarded services translate gRPC requests to websocket API calls to the server component + +## Dependencies +- **Server Component**: Implements the stable API and handles forwarded experimental API requests +- **Public-api-server Component**: Implements experimental APIs directly or forwards to server +- **Database**: Stores user, workspace, and organization data +- **Redis**: Used for caching and session management +- **gRPC and Connect**: Used for API implementation +- **Protocol Buffers**: Used for API definition and code generation + +## Usage Examples +- Creating and managing workspaces programmatically +- Building custom dashboards and management tools +- Integrating Gitpod with CI/CD pipelines +- Automating workspace provisioning +- Building IDE extensions that interact with Gitpod +- Implementing custom authentication flows +- Integrating with third-party services + +## Version Compatibility +The API uses Protocol Buffers version 3 (proto3) syntax, which provides forward and backward compatibility features. + +### Stable API (v1) +- Services, calls, types, and fields are not removed without following a deprecation policy +- Services, calls, types, and fields are not renamed +- Non-successful responses are described exhaustively +- Changes require an API User Experience review + +### Experimental API +- No compatibility guarantees +- May change frequently +- Should not be relied upon for production use + +## Code Generation and Building + +### Regenerating Code from Protobuf Definitions +The Public API uses Protocol Buffers and gRPC for defining interfaces. When changes are made to the `.proto` files, the corresponding code in Go, TypeScript, and Java needs to be regenerated. + +To regenerate the code: + +1. Navigate to the public-api directory: + ```bash + cd components/public-api + ``` + +2. Run the generate script: + ```bash + ./generate.sh + ``` + +3. Rebuild the typescript code: + ```bash + cd typescript-commond && yarn build + ``` + +This script performs the following actions: +- Installs necessary dependencies +- Lints the proto files using buf +- Runs breaking change detection against the main branch +- Removes previously generated files +- Generates Go, TypeScript, and Java code using buf +- Updates license headers +- Runs formatting tools +- Builds the TypeScript package + +### Building After Code Generation +After regenerating the code, you may need to rebuild components that depend on the Public API. This typically involves: + +1. For Go components: + ```bash + cd + go build ./... + ``` + +2. For TypeScript components: + ```bash + cd + yarn install + yarn build + ``` + +3. Using Leeway (for CI/CD): + ```bash + leeway build -D components/:app + ``` + +The Public API is a critical component for enabling programmatic access to Gitpod functionality. It enables third-party integrations, automation, and custom tooling to interact with Gitpod in a standardized, versioned way. diff --git a/memory-bank/components/registry-facade-api.md b/memory-bank/components/registry-facade-api.md index fb46e6299d26fa..634c6644845a9f 100644 --- a/memory-bank/components/registry-facade-api.md +++ b/memory-bank/components/registry-facade-api.md @@ -86,13 +86,6 @@ This script performs the following actions: - Generates Go code using `protoc-gen-go` and `protoc-gen-go-grpc` - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `go_protoc`: Generates Go code -- `update_license`: Updates license headers in generated files - The Registry Facade API is relatively simple compared to other APIs, focusing primarily on defining the structure of image specifications rather than complex service interactions. ### Building After Code Generation diff --git a/memory-bank/components/server.md b/memory-bank/components/server.md index 0aadab361d6f14..24fdc5cb548b47 100644 --- a/memory-bank/components/server.md +++ b/memory-bank/components/server.md @@ -21,7 +21,7 @@ The primary purposes of the Server component are: The Server operates as an Express.js application with several key components: -1. **API Server**: Provides HTTP and WebSocket endpoints for client communication +1. **API Server**: Provides HTTP and WebSocket endpoints for client communication. It also directly hosts and implements `gitpod.v1` gRPC services (defined in `.proto` files within `components/public-api/`) for programmatic access. 2. **Authentication System**: Handles user authentication and session management 3. **Database Interface**: Interacts with the database for persistent storage 4. **WebSocket Manager**: Manages real-time communication with clients @@ -40,6 +40,7 @@ The server is designed as a modular application using dependency injection (Inve - `src/auth/`: Authentication and authorization - `src/workspace/`: Workspace management - `src/user/`: User management +- `src/orgs/`: Organization management - `src/prebuilds/`: Prebuild functionality - `src/billing/`: Billing and subscription management - `src/github/`, `src/gitlab/`, `src/bitbucket/`: SCM integrations @@ -79,8 +80,9 @@ The Server is configured via environment variables and configuration files, incl The Server exposes multiple API endpoints: 1. **User API**: User management, authentication, and preferences -2. **Workspace API**: Workspace creation, management, and access -3. **SCM Integration APIs**: GitHub, GitLab, Bitbucket webhooks and OAuth +2. **Organization API**: Handles organization creation, settings, member management, and administrative functions like maintenance mode. +3. **Workspace API**: Workspace creation, management, and access +4. **SCM Integration APIs**: GitHub, GitLab, Bitbucket webhooks and OAuth 4. **Billing API**: Subscription and payment management 5. **WebSocket API**: Real-time communication with clients 6. **Health and Metrics API**: System health and monitoring @@ -94,7 +96,7 @@ The Server supports multiple authentication methods: 3. **OAuth Integration**: With GitHub, GitLab, Bitbucket, etc. 4. **Personal Access Tokens**: For programmatic access -Authorization is handled through a combination of user roles, permissions, and access controls. +Authorization is handled through a combination of user roles and permissions, leveraging SpiceDB for fine-grained access control checks within its service implementations (including for gRPC services). ## Integration Points diff --git a/memory-bank/components/spicedb.md b/memory-bank/components/spicedb.md index a8702392c41238..ae96d0d0a6d1b9 100644 --- a/memory-bank/components/spicedb.md +++ b/memory-bank/components/spicedb.md @@ -108,17 +108,13 @@ The schema defines various permissions for each entity type: ## Configuration -The SpiceDB component is configured through a YAML schema file: +The SpiceDB component's core configuration is its schema, defined in `components/spicedb/schema/schema.yaml`. This YAML file is central to Gitpod's authorization model and specifies: +- **Object Definitions**: The types of resources being protected (e.g., `user`, `organization`, `project`, `workspace`). +- **Relations**: How different objects can be related to each other (e.g., an `organization` has an `owner` which is a `user`; a `project` belongs to an `org`). +- **Permissions**: What actions can be performed on objects, and how these permissions are derived from relations (e.g., `permission write_settings = owner + installation->admin` for an `organization`). +- **Validation & Assertions**: Rules and tests to ensure the schema's integrity and correctness. -```yaml -schema/schema.yaml -``` - -This file defines: -- Entity types (user, organization, project, workspace) -- Relationships between entities -- Permissions derived from these relationships -- Validation rules and assertions for testing +New permissions (like the `maintenance` permission for organizations) are added by modifying this schema file. Other components, such as the `server` component, then use a SpiceDB client to check these permissions at runtime to authorize operations. ## Integration Points diff --git a/memory-bank/components/supervisor-api.md b/memory-bank/components/supervisor-api.md index 891a429370cfd3..e26e483394b158 100644 --- a/memory-bank/components/supervisor-api.md +++ b/memory-bank/components/supervisor-api.md @@ -149,14 +149,6 @@ This script performs the following actions: - Updates license headers - Removes trailing whitespace from Java files -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh` and defines some custom functions: - -- `install_dependencies`: Installs required protoc plugins -- `local_go_protoc`: Generates Go code with specific include paths for third-party dependencies -- `go_protoc_gateway`: Generates gRPC Gateway code for REST endpoints -- `update_license`: Updates license headers in generated files - The `generate-java.sh` script: - Temporarily modifies proto files to handle Java reserved keywords - Downloads the gRPC Java plugin if needed diff --git a/memory-bank/components/usage-api.md b/memory-bank/components/usage-api.md index f84c2805831220..37be369a38e55b 100644 --- a/memory-bank/components/usage-api.md +++ b/memory-bank/components/usage-api.md @@ -118,14 +118,6 @@ This script performs the following actions: - Generates Go and TypeScript code using buf - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `lint`: Lints the proto files using buf -- `protoc_buf_generate`: Generates code using buf based on the configuration in `buf.gen.yaml` -- `update_license`: Updates license headers in generated files - The `buf.gen.yaml` file configures the code generation: - Generates Go code with appropriate module paths - Generates TypeScript code using ts_proto with specific options for nice-grpc compatibility diff --git a/memory-bank/components/ws-daemon-api.md b/memory-bank/components/ws-daemon-api.md index 6410b637c5238d..47d6d4f0384d53 100644 --- a/memory-bank/components/ws-daemon-api.md +++ b/memory-bank/components/ws-daemon-api.md @@ -119,14 +119,6 @@ This script performs the following actions: - Patches the generated TypeScript code for compatibility - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `go_protoc`: Generates Go code -- `typescript_protoc`: Generates TypeScript code -- `update_license`: Updates license headers in generated files - Additionally, the script: - Generates mock implementations using `mockgen` for testing - Updates JSON tags in the generated Go code using `gomodifytags` diff --git a/memory-bank/components/ws-manager-api.md b/memory-bank/components/ws-manager-api.md index a96e86bd7bd115..a92e1d00b97f2c 100644 --- a/memory-bank/components/ws-manager-api.md +++ b/memory-bank/components/ws-manager-api.md @@ -139,14 +139,6 @@ This script performs the following actions: - Patches the generated TypeScript code for compatibility - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `go_protoc`: Generates Go code -- `typescript_protoc`: Generates TypeScript code -- `update_license`: Updates license headers in generated files - Additionally, the script: - Generates mock implementations using `mockgen` for testing - Patches the generated TypeScript code using a script from the content-service-api diff --git a/memory-bank/components/ws-manager-bridge-api.md b/memory-bank/components/ws-manager-bridge-api.md index 118df93ec2a06e..1187c01ba0b919 100644 --- a/memory-bank/components/ws-manager-bridge-api.md +++ b/memory-bank/components/ws-manager-bridge-api.md @@ -94,13 +94,6 @@ This script performs the following actions: - Generates code using buf based on the configuration in `buf.gen.yaml` - Updates license headers -### Implementation Details -The `generate.sh` script uses functions from the shared script at `scripts/protoc-generator.sh`: - -- `install_dependencies`: Installs required protoc plugins -- `protoc_buf_generate`: Generates code using buf based on the configuration in `buf.gen.yaml` -- `update_license`: Updates license headers in generated files - The `buf.gen.yaml` file configures the code generation: - Generates Go code with appropriate module paths - Generates JavaScript and TypeScript code with gRPC support diff --git a/memory-bank/systemPatterns.md b/memory-bank/systemPatterns.md index fc23decbcf0192..7c12b976674590 100644 --- a/memory-bank/systemPatterns.md +++ b/memory-bank/systemPatterns.md @@ -70,6 +70,28 @@ Gitpod follows a microservices architecture composed of several key components t 4. **Public API**: Provides programmatic access to Gitpod functionality. +## Project Structure Patterns + +- **Component Organization**: The project is organized into components, each with its own directory in the `components/` folder. +- **API Definitions**: API definitions are typically in separate packages with `-api` suffix (e.g., `content-service-api`). +- **Protocol Buffers**: gRPC service definitions use Protocol Buffers (`.proto` files). +- **Build Configuration**: Each component has a `BUILD.yaml` file defining its build configuration. +- **Docker Configuration**: Components that run as containers have a `leeway.Dockerfile`. + +## Code Style Preferences + +### Go Code +- Follow standard Go conventions (gofmt). +- Error handling with explicit checks. +- Context propagation for cancellation. +- Structured logging. + +### TypeScript Code +- Use TypeScript for type safety. +- React for UI components. +- Functional components with hooks. +- ESLint and Prettier for formatting. + ## Design Patterns ### Microservices Pattern @@ -84,12 +106,6 @@ Components communicate through events for asynchronous operations, improving sca ### API Gateway Pattern A central API gateway routes requests to appropriate services, handling cross-cutting concerns like authentication. -### Circuit Breaker Pattern -Services implement circuit breakers to prevent cascading failures when downstream services are unavailable. - -### Sidecar Pattern -The Supervisor component runs alongside workspace applications as a sidecar, providing common functionality. - ### Immutable Infrastructure Workspaces are treated as immutable, with changes to configuration resulting in new environments rather than modifications to existing ones. @@ -97,15 +113,22 @@ Workspaces are treated as immutable, with changes to configuration resulting in ### Workspace Lifecycle -1. User requests a workspace through the dashboard or Git integration -2. Auth service validates the request -3. Workspace Manager creates the workspace specification -4. Image Builder ensures the required image is available -5. Workspace Manager instructs Kubernetes to create the workspace pod -6. Workspace Daemon initializes the workspace environment -7. Supervisor starts within the workspace -8. IDE Service connects the IDE to the workspace -9. Proxy routes user traffic to the workspace +1. User requests a workspace through the dashboard or Git integration. +2. Auth service validates the request. +3. Workspace Manager creates the workspace specification. +4. Image Builder ensures the required image is available. +5. Workspace Manager instructs Kubernetes to create the workspace pod. +6. Workspace Daemon initializes the workspace environment. +7. Supervisor starts within the workspace. +8. IDE Service connects the IDE to the workspace. +9. Proxy routes user traffic to the workspace. + +The key components involved in the workspace lifecycle include: + - workspace-manager-mk2 + - ws-daemon + - ws-manager-bridge + - server + - image-builder-mk3 ### Data Flow @@ -132,16 +155,118 @@ Workspaces are treated as immutable, with changes to configuration resulting in 8. **Kubernetes Deployment Configuration**: All code that defines Kubernetes objects for deployable components lives in `install/installer`. This centralized approach ensures consistent deployment patterns across all components. +9. **Public API Architecture**: External programmatic access is provided via `gitpod.v1` gRPC services. These services are defined using Protocol Buffers (`.proto` files) located in the `components/public-api/gitpod/v1/` directory. The `server` component (TypeScript/Express.js based) directly implements and hosts these gRPC services. The `public-api-server` component (Go based) can act as an external-facing gRPC gateway. + +10. **Authorization with SpiceDB**: Fine-grained, relationship-based access control (ReBAC) is managed by SpiceDB. The authorization schema, defining resources, relationships, and permissions, is located in `components/spicedb/schema/schema.yaml`. Service implementations (e.g., within the `server` component) query SpiceDB to enforce these permissions. + ## Development Workflows ### Product Requirements Document (PRD) Workflow Gitpod uses a structured PRD workflow for feature development to ensure proper planning, documentation, and implementation: -1. **Requirements Gathering** (Plan Mode): Understand the problem, explore existing components, gather information -2. **PRD Creation** (Plan Mode): Create a detailed document in `prd/` with standardized sections -3. **Implementation Planning** (Plan Mode): Identify files to modify and plan the approach -4. **Implementation** (Act Mode): Create/modify necessary files following the plan -5. **Documentation Update** (Act Mode): Update memory bank with new knowledge -6. **Verification**: Ensure implementation meets requirements and documentation is complete +1. **Requirements Gathering** (Plan Mode): Understand the problem, explore existing components, gather information. +2. **PRD Creation** (Plan Mode): Create a detailed document in `prd/` (e.g., `NNN-featurename.md`) with standardized sections: Overview, Background, Requirements, Implementation Details, Testing, Deployment Considerations, Future Improvements +3. **Implementation Planning** (Plan Mode): Identify files to modify and plan the approach +4. **Implementation** (Act Mode): Create/modify necessary files following the plan +5. **Documentation Update** (Act Mode): Update memory bank with new knowledge +6. **Verification**: Ensure implementation meets requirements and documentation is complete This workflow ensures thorough planning, clear documentation, and knowledge preservation for all feature development. + +### Build Approaches + +- **In-tree builds** (for development in a Gitpod workspace): + - TypeScript components: Use `yarn` commands defined in `package.json`. + - `yarn build`: Compile the component. + - `yarn test`: Run tests. + - `yarn lint`: Check code style. + - `yarn watch`: Watch for changes and rebuild. + - Go components: Use standard Go tools. + - `go build ./...`: Build all packages. + - `go test ./...`: Test all packages. + - `go run main.go`: Build and run. +- **Out-of-tree builds** (Leeway, primarily used in CI): + - `leeway build components/component-name:app`: Build a specific component. + - `leeway build -D components/component-name:app`: Build with dependencies. + - `leeway exec --package components/component-name:app -- command`: Run a command for a package. + +### Testing + +- Unit tests alongside code. +- Integration tests in separate directories. +- End-to-end tests in the `test/` directory. +- Component-specific test commands in `package.json` (for TypeScript). +- Go tests use standard `go test` command. + +### Local Development + +- Use Gitpod workspaces for development (dogfooding). +- Components can be run individually for testing. +- Preview environments for testing changes. +- Use in-tree builds for rapid iteration during development. + +## Known Challenges + +- **Build System Complexity**: The Leeway build system has a learning curve. +- **Component Dependencies**: Understanding dependencies between components can be challenging. +- **Testing Environment**: Setting up proper testing environments for all components. + +## Tool Usage Patterns + +- **VS Code**: Primary IDE for TypeScript development. +- **GoLand/IntelliJ**: Often used for Go development. +- **Docker**: Used for containerized development and testing. +- **kubectl**: Used for interacting with Kubernetes clusters. +- **Werft**: CI/CD system for automated builds and tests. + +## Documentation Patterns + +- **README.md**: Each component should have a README explaining its purpose. +- **API Documentation**: Generated from Protocol Buffer definitions. +- **Memory Bank Documentation**: + - Component-specific documentation is stored in `memory-bank/components/` directory. + - Each component gets its own markdown file with detailed information about its purpose, architecture, and implementation. + - Component documentation should focus on both service components that implement business logic and API components that define interfaces. + - Documentation follows a consistent structure with sections for Overview, Purpose, Architecture, Key Features, etc. + - API component documentation should include a "Code Generation and Building" section that explains: + - How to regenerate code from protobuf definitions. + - The implementation details of the generation process. + - How to build components that depend on the API after regeneration. + +## Evolution of Project Decisions + +This section captures how and why certain architectural and design decisions were made. + +### Memory Bank Organization + +- **Component Documentation**: The decision to create separate documentation files for each component in the `memory-bank/components/` directory was made to: + 1. Provide a clear, organized structure for component documentation. + 2. Allow for detailed documentation of each component's purpose, architecture, and implementation. + 3. Make it easier to find information about specific components. + 4. Enable incremental updates to component documentation without affecting other files. +- **API Component Documentation**: The decision to include "*-api" components in separate documentation was made because: + 1. API components define critical interfaces between services. + 2. Understanding these interfaces is essential for developers working across components. + 3. API documentation helps clarify system boundaries and communication patterns. + 4. Separate documentation makes it easier to track API changes and versioning. + +## User Preferences + +This section will be updated as specific user preferences for working with the codebase are identified. + +## Build and Test Information + +When working with components, it is crucial to document the following: + +- **Build Commands**: Document any new or component-specific build commands encountered. +- **Test Commands**: Document how to run tests for each component. +- **Dependencies**: Note any special dependencies required for building or testing. +- **Common Issues**: Document common build or test issues and their solutions. +- **Performance Considerations**: Note any performance considerations for builds. + +Whenever new build patterns or commands are encountered, the following should be updated: +1. The relevant component documentation in `memory-bank/components/`. +2. This `systemPatterns.md` file (previously the learning journal) with general patterns. +3. The `techContext.md` file if it represents a significant pattern. + +This information is critical for effective development work, as being able to build and test components is fundamental to making changes and verifying their correctness. diff --git a/memory-bank/techContext.md b/memory-bank/techContext.md index b5ab1a855d61f5..fb5cb54aad0fba 100644 --- a/memory-bank/techContext.md +++ b/memory-bank/techContext.md @@ -19,7 +19,8 @@ ### Infrastructure - **Kubernetes**: Container orchestration platform - **Docker**: Containerization technology -- **MySQL**: Primary database for persistent data +- **MySQL**: Primary database for persistent data. + - TypeORM is used as the ORM - **Redis**: Caching and ephemeral data storage - **Helm**: Kubernetes package management - **Terraform**: Infrastructure as code for deployment From 7fc4605fa337083ea3a34cf9eb597ad7dc9c5f99 Mon Sep 17 00:00:00 2001 From: Gero Posmyk-Leinemann Date: Fri, 23 May 2025 10:19:30 +0000 Subject: [PATCH 2/2] [memory-bank] Compress --- .clinerules/memory-bank.md | 2 +- memory-bank/activeContext.md | 190 ++++-------------- memory-bank/components.md | 68 ++----- memory-bank/productContext.md | 94 ++++----- memory-bank/progress.md | 257 ++++++------------------ memory-bank/projectbrief.md | 41 ++-- memory-bank/systemPatterns.md | 364 +++++++++++++--------------------- memory-bank/techContext.md | 216 ++++++-------------- 8 files changed, 373 insertions(+), 859 deletions(-) diff --git a/.clinerules/memory-bank.md b/.clinerules/memory-bank.md index a61a3bbbf853cc..abbe14b914bfdd 100644 --- a/.clinerules/memory-bank.md +++ b/.clinerules/memory-bank.md @@ -126,7 +126,7 @@ Note: When triggered by **update memory bank**, I MUST review every memory bank - Short-term memory (activeContext.md, progress.md): Detailed, recent, specific - Long-term memory (systemPatterns.md, techContext.md, projectbrief.md): Compressed, patterns, principles - Apply this strategy on every interaction with the memory bank -- Use "compress memory bank" trigger to perform a compression run +- Use **compress memory bank** trigger to perform a compression run When compressing memory bank files: 1. Focus on patterns over instances diff --git a/memory-bank/activeContext.md b/memory-bank/activeContext.md index ae6b25f3eb1d28..36b2f7bfae0d7d 100644 --- a/memory-bank/activeContext.md +++ b/memory-bank/activeContext.md @@ -1,169 +1,59 @@ # Active Context: Gitpod ## Current Work Focus +Building a comprehensive knowledge base of the Gitpod codebase and architecture for effective development, troubleshooting, and enhancement. -We are focusing on understanding the Gitpod codebase and architecture. The primary goal is to build a comprehensive knowledge base that will allow for effective development, troubleshooting, and enhancement of the Gitpod platform. - -Key areas of focus include: - -1. **System Architecture Understanding**: Mapping out the relationships between components and services -2. **Component Documentation**: Creating detailed documentation for each component -3. **Development Workflow**: Understanding how to effectively develop and test changes -4. **Documentation**: Maintaining a comprehensive memory bank for future reference -5. **Standardized Workflows**: Establishing and documenting standardized workflows for development activities +| Focus Area | Description | +|------------|-------------| +| System Architecture | Component relationships and service interactions | +| Component Documentation | Detailed documentation for each component | +| Development Workflow | Effective development and testing processes | +| Documentation | Comprehensive memory bank maintenance | +| Standardized Workflows | Established development procedures | ## Recent Changes -- Created the initial memory bank structure with core files -- Established standardized workflows for development activities: - - Product Requirements Document (PRD) workflow for feature development -- Added a components subdirectory to the memory bank -- Created detailed documentation for key components: - - blobserve: Service that provides static assets from OCI images - - content-service: Manages various types of content within the platform - - dashboard: Web-based user inSterface for Gitpod - - ws-manager-mk2: Kubernetes controller for managing workspace lifecycle - - supervisor: Init process that runs inside each workspace container - - ws-daemon: Node-level daemon for workspace operations - - ide-service: Manages IDE configurations and resolves workspace IDE requirements - - registry-facade: Modifies container images by adding layers for workspaces - - image-builder-mk3: Builds custom workspace images from user-defined Dockerfiles - - server: Main backend service handling API requests and user management - - proxy: Main entry point for all HTTP and WebSocket traffic - - ws-proxy: Handles routing and proxying of traffic to workspaces - - gitpod-cli: Command-line interface for interacting with Gitpod workspaces - - gitpod-db: Database layer for the Gitpod platform - - gitpod-protocol: Core type definitions and shared protocol library - - ide: Packages and manages IDEs available in Gitpod workspaces - - ide-proxy: Serves static IDE-related assets and proxies requests - - ws-manager-bridge: Bridges between workspace managers and the rest of the platform - - ide-metrics: Collects and processes metrics and error reports from IDE components - - local-app: Provides tools for interacting with Gitpod workspaces from local machine - - public-api-server: Provides a stable, versioned API for programmatic access to Gitpod - - usage: Tracks, calculates, and manages workspace usage and billing - - common-go: Foundational Go library providing shared utilities across services - - workspacekit: Manages container setup and namespace isolation for workspaces - - spicedb: Provides authorization and permission management - - scrubber: Removes or masks sensitive information from data - - service-waiter: Waits for services to become available - - docker-up: Sets up and manages Docker within workspace containers - - image-builder-bob: Builds and pushes workspace images during workspace startup - - node-labeler: Manages node labels and annotations for workspace scheduling - - openvsx-proxy: Caching proxy service for the OpenVSX registry - - scheduler-extender: Extends Kubernetes scheduling capabilities for workspaces - - ipfs: Provides distributed content-addressable storage for container images -- Created documentation for API components: - - content-service-api: Interfaces for managing workspace content, blobs, logs, and IDE plugins - - ide-metrics-api: Interfaces for collecting metrics and error reports from IDE components - - ide-service-api: Interfaces for managing IDE configurations and resolving workspace IDE requirements - - image-builder-api: Interfaces for building Docker images for workspaces - - local-app-api: Interfaces for communication between local machines and remote workspaces - - registry-facade-api: Interfaces for dynamically assembling workspace container images - - supervisor-api: Interfaces for workspace management, terminal handling, and port forwarding - - usage-api: Interfaces for tracking, calculating, and managing workspace usage and billing - - ws-daemon-api: Interfaces for workspace content management and container operations - - ws-manager-api: Interfaces for managing the lifecycle of workspaces in Kubernetes clusters - - ws-manager-bridge-api: Interfaces for dynamic management of workspace clusters -- Enhanced API component documentation with code generation information: - - Added details on how to regenerate code from protobuf definitions - - Documented the implementation details of the generation process - - Included instructions for building components after code regeneration - - Updated .clinerules to standardize API documentation with code generation sections - -As work progresses, this section will continue to be updated to reflect: -- Additional component documentation -- Code changes implemented -- Bug fixes -- Feature additions -- Refactoring efforts +- Established memory bank structure with core files +- Created standardized PRD workflow for feature development +- Documented 33 service components and 11 API components +- Enhanced API component documentation with code generation information +- Implemented server readiness probe with database, SpiceDB, and Redis connectivity checks ## Next Steps -The immediate next steps are: - -1. **Explore Component Interactions**: Understand how components interact with each other -2. **Set Up Development Environment**: Configure a local development environment for effective testing -3. **Explore Build System**: Gain hands-on experience with both in-tree and Leeway builds -4. **Test Component Builds**: Practice building and testing different component types -5. **Identify Initial Tasks**: Determine specific tasks or improvements to focus on -6. **Establish Testing Approach**: Define how changes will be tested and validated -7. **Update Memory Bank**: Continue to refine and expand the memory bank as new information is discovered - -## Active Decisions and Considerations - -### Architecture Decisions - -- **Component Boundaries**: Understanding and respecting the boundaries between different microservices -- **API Contracts**: Maintaining compatibility with existing API contracts -- **Performance Considerations**: Ensuring changes maintain or improve performance characteristics +1. **Component Interactions**: Understand inter-component communication +2. **Development Environment**: Configure local development setup +3. **Build System**: Gain experience with in-tree and Leeway builds +4. **Component Builds**: Practice building different component types +5. **Initial Tasks**: Identify specific improvement areas +6. **Testing Approach**: Define validation methodology +7. **Memory Bank Updates**: Continue documentation expansion -### Development Approach +## Active Decisions -- **Testing Strategy**: Determining appropriate testing approaches for different types of changes -- **Documentation Standards**: Establishing standards for code documentation and memory bank updates -- **Collaboration Model**: Defining how to effectively collaborate with the team +| Category | Considerations | +|----------|----------------| +| **Architecture** | • Component boundaries
• API contracts
• Performance impacts | +| **Development** | • Testing strategies
• Documentation standards
• Collaboration model | +| **Technical** | • Backward compatibility
• Security implications
• Scalability impacts | -### Technical Considerations +## Current Questions -- **Backward Compatibility**: Ensuring changes maintain compatibility with existing clients and integrations -- **Security Implications**: Evaluating security implications of any changes -- **Scalability**: Considering how changes impact system scalability - -## Current Questions and Uncertainties - -As we begin working with the Gitpod codebase, several questions and uncertainties exist: - -1. **Component Interactions**: How do the various components interact in specific scenarios? -2. **Performance Bottlenecks**: What are the current performance bottlenecks in the system? -3. **Testing Approach**: What is the most effective way to test changes to different components? -4. **Deployment Process**: What is the process for deploying changes to production? -5. **Feature Priorities**: What features or improvements are currently prioritized? - -These questions will be addressed as we continue to explore the codebase and work with the system. +1. How do components interact in specific scenarios? +2. What are the current performance bottlenecks? +3. What's the most effective testing approach for different components? +4. What is the process for production deployment? +5. Which features/improvements are currently prioritized? ## Active Experiments - -No active experiments are currently in progress. This section will be updated as experiments are designed and conducted to test hypotheses about the system or potential improvements. +No active experiments currently in progress. ## Recent Learnings -Initial exploration of the Gitpod codebase has revealed: - -- **Microservices Architecture**: Gitpod is built as a collection of loosely coupled services, each with specific responsibilities -- **Go and TypeScript**: Backend services are primarily written in Go, while frontend components use TypeScript/React -- **Kubernetes Native**: Many components are designed as Kubernetes controllers or operators -- **gRPC Communication**: Services communicate using gRPC for efficient, typed communication -- **Component Patterns**: Components follow consistent patterns: - - Go services typically have a cmd/ directory with command implementations - - TypeScript services use React and modern frontend practices - - Most components have a clear separation between API definitions and implementations -- **Build System Approaches**: Gitpod uses two primary approaches for building components: - - **In-tree builds**: Using language-specific tools (yarn, go) directly in the workspace - - Primary method for local development - - TypeScript components use commands defined in package.json (yarn build, yarn test, etc.) - - Go components use standard Go tools (go build, go test, etc.) - - **Out-of-tree builds**: Using Leeway, a custom build tool - - Primary method for CI to generate build artifacts - - Works by copying relevant sources into a separate file tree - - Can also be run from inside the workspace - - Manages complex dependencies between components -- **Server Health Checks**: The Gitpod server uses two distinct health check mechanisms: - - **Liveness Probe**: Checks the event loop lag to determine if the server is functioning properly - - **Readiness Probe**: Checks database, SpiceDB, and Redis connectivity to ensure the server is ready to handle requests - - Controlled by a ConfigCat feature flag `server_readiness_probe` (default: true) that can bypass the actual checks -- **Critical Dependencies**: The server has critical external dependencies that must be operational: - - **Database (TypeORM)**: Used for persistent storage - - **SpiceDB**: Used for authorization and permission management - - **Redis**: Used for caching, pub/sub messaging, and distributed locking -- **Server Architecture Patterns**: - - The server uses dependency injection (Inversify) for component management - - Components are registered in `container-module.ts` and injected where needed - - The server exposes HTTP endpoints for health checks and other functionality - - Routes are registered in the `registerRoutes` method in `server.ts` - - New functionality typically requires: - 1. Creating a new controller/service class - 2. Registering it in the container module - 3. Injecting it where needed - 4. Updating any relevant configuration files - -This section will be continuously updated as new insights are gained through working with the system. +| Area | Insights | +|------|----------| +| **Architecture** | • Microservices with loosely coupled services
• Go (backend) and TypeScript/React (frontend)
• Kubernetes-native components
• gRPC for service communication | +| **Component Patterns** | • Go services with cmd/ directory structure
• TypeScript with Reac
• Clear API/implementation separation | +| **Build System** | • In-tree: Language tools (yarn, go) for local dev
• Out-of-tree: Leeway for CI/CD and dependency management | +| **Server Architecture** | • Dependency injection (Inversify)
• Component registration in container-module.ts
• HTTP endpoints in registerRoutes method
• Health checks: liveness (event loop) and readiness (dependencies) | +| **Critical Dependencies** | • Database (TypeORM)
• SpiceDB (authorization)
• Redis (caching, pub/sub, locking) | diff --git a/memory-bank/components.md b/memory-bank/components.md index 0b6aabcb321dd3..c7b8f1db53aeeb 100644 --- a/memory-bank/components.md +++ b/memory-bank/components.md @@ -1,49 +1,25 @@ # Gitpod Components Index -This file serves as index for the per-component documentation in individual md files in the `memory-bank/components` directory. +This file serves as an index for the per-component documentation in individual md files in the `memory-bank/components` directory. -- [server](components/server.md) -- [content-service](components/content-service.md) -- [ws-manager-mk2](components/ws-manager-mk2.md) -- [ws-daemon](components/ws-daemon.md) -- [image-builder-mk3](components/image-builder-mk3.md) -- [content-service-api](components/content-service-api.md) -- [ws-manager-api](components/ws-manager-api.md) -- [ws-daemon-api](components/ws-daemon-api.md) -- [image-builder-api](components/image-builder-api.md) -- [supervisor-api](components/supervisor-api.md) -- [registry-facade-api](components/registry-facade-api.md) -- [ide-service-api](components/ide-service-api.md) -- [ide-metrics-api](components/ide-metrics-api.md) -- [ws-manager-bridge-api](components/ws-manager-bridge-api.md) -- [public-api](components/public-api.md) -- [usage-api](components/usage-api.md) -- [local-app-api](components/local-app-api.md) -- [dashboard](components/dashboard.md) -- [ide](components/ide.md) -- [proxy](components/proxy.md) -- [registry-facade](components/registry-facade.md) -- [blobserve](components/blobserve.md) -- [ipfs](components/ipfs.md) -- [openvsx-proxy](components/openvsx-proxy.md) -- [scheduler-extender](components/scheduler-extender.md) -- [gitpod-db](components/gitpod-db.md) -- [gitpod-protocol](components/gitpod-protocol.md) -- [supervisor](components/supervisor.md) -- [workspacekit](components/workspacekit.md) -- [ws-proxy](components/ws-proxy.md) -- [ide-proxy](components/ide-proxy.md) -- [ide-service](components/ide-service.md) -- [ide-metrics](components/ide-metrics.md) -- [docker-up](components/docker-up.md) -- [common-go](components/common-go.md) -- [service-waiter](components/service-waiter.md) -- [node-labeler](components/node-labeler.md) -- [scrubber](components/scrubber.md) -- [spicedb](components/spicedb.md) -- [public-api-server](components/public-api-server.md) -- [gitpod-cli](components/gitpod-cli.md) -- [local-app](components/local-app.md) -- [ws-manager-bridge](components/ws-manager-bridge.md) -- [image-builder-bob](components/image-builder-bob.md) -- [usage](components/usage.md) +## Core Services + +| Category | Components | +|----------|------------| +| **Workspace Management** | [ws-manager-mk2](components/ws-manager-mk2.md), [ws-daemon](components/ws-daemon.md), [supervisor](components/supervisor.md), [workspacekit](components/workspacekit.md), [ws-proxy](components/ws-proxy.md), [ws-manager-bridge](components/ws-manager-bridge.md) | +| **Image Management** | [image-builder-mk3](components/image-builder-mk3.md), [registry-facade](components/registry-facade.md), [blobserve](components/blobserve.md), [ipfs](components/ipfs.md), [image-builder-bob](components/image-builder-bob.md) | +| **Content Management** | [content-service](components/content-service.md) | +| **User Interface** | [dashboard](components/dashboard.md), [gitpod-cli](components/gitpod-cli.md), [local-app](components/local-app.md) | +| **IDE Integration** | [ide](components/ide.md), [ide-service](components/ide-service.md), [ide-proxy](components/ide-proxy.md), [ide-metrics](components/ide-metrics.md), [openvsx-proxy](components/openvsx-proxy.md) | +| **Core Infrastructure** | [server](components/server.md), [proxy](components/proxy.md), [gitpod-db](components/gitpod-db.md), [gitpod-protocol](components/gitpod-protocol.md), [public-api-server](components/public-api-server.md) | +| **Support Services** | [docker-up](components/docker-up.md), [common-go](components/common-go.md), [service-waiter](components/service-waiter.md), [node-labeler](components/node-labeler.md), [scrubber](components/scrubber.md), [spicedb](components/spicedb.md), [scheduler-extender](components/scheduler-extender.md), [usage](components/usage.md) | + +## API Components + +| Category | Components | +|----------|------------| +| **Workspace APIs** | [ws-manager-api](components/ws-manager-api.md), [ws-daemon-api](components/ws-daemon-api.md), [ws-manager-bridge-api](components/ws-manager-bridge-api.md), [supervisor-api](components/supervisor-api.md) | +| **Content APIs** | [content-service-api](components/content-service-api.md) | +| **Image APIs** | [image-builder-api](components/image-builder-api.md), [registry-facade-api](components/registry-facade-api.md) | +| **IDE APIs** | [ide-service-api](components/ide-service-api.md), [ide-metrics-api](components/ide-metrics-api.md) | +| **Platform APIs** | [public-api](components/public-api.md), [usage-api](components/usage-api.md), [local-app-api](components/local-app-api.md) | diff --git a/memory-bank/productContext.md b/memory-bank/productContext.md index 46d893870d44d6..f654d48466e136 100644 --- a/memory-bank/productContext.md +++ b/memory-bank/productContext.md @@ -1,68 +1,50 @@ # Product Context: Gitpod ## Why Gitpod Exists -Gitpod was created to solve the fundamental problem of development environment inconsistency and setup time. Traditional development workflows require developers to manually set up and maintain their development environments, leading to: +Gitpod addresses development environment inconsistency and setup time challenges through ephemeral, automated cloud environments. -1. **"Works on my machine" problems**: Inconsistent environments across team members -2. **Onboarding friction**: New team members spending days setting up their environment -3. **Context switching costs**: Significant time lost when switching between projects -4. **Resource limitations**: Local machines struggling with resource-intensive development tasks -5. **Configuration drift**: Environments becoming inconsistent over time +### Key Challenges Solved +| Challenge | Description | +|-----------|-------------| +| Environment Inconsistency | Eliminates "works on my machine" problems | +| Onboarding Friction | Reduces new team member setup from days to seconds | +| Context Switching | Minimizes project switching overhead | +| Resource Limitations | Overcomes local machine constraints | +| Configuration Drift | Prevents environments from becoming inconsistent | -Gitpod addresses these challenges by providing ephemeral, automated, and consistent development environments in the cloud. +## Value Proposition by Stakeholder -## Problems Gitpod Solves - -### For Individual Developers -- Eliminates environment setup and maintenance time -- Provides consistent, reproducible environments -- Enables development from any device with a browser -- Reduces local resource constraints (CPU, memory, storage) -- Simplifies context switching between projects - -### For Teams -- Ensures all team members work in identical environments -- Dramatically reduces onboarding time for new developers -- Improves collaboration through shared workspaces -- Streamlines code review processes -- Reduces infrastructure management overhead - -### For Organizations -- Enhances security through containerized environments -- Improves productivity by eliminating environment-related issues -- Enables standardization of development tools and practices -- Provides flexibility for remote and distributed teams -- Reduces costs associated with powerful local development machines +| Stakeholder | Benefits | +|-------------|----------| +| **Individual Developers** | • Zero environment setup/maintenance
• Consistent environments
• Device-agnostic development
• Unlimited resources
• Effortless project switching | +| **Teams** | • Identical environments
• Rapid onboarding
• Enhanced collaboration
• Streamlined code reviews
• Reduced infrastructure overhead | +| **Organizations** | • Improved security
• Increased productivity
• Standardized tooling
• Remote work enablement
• Hardware cost reduction | ## How Gitpod Works - -1. **Environment Definition**: Developers define their environment in a `.gitpod.yml` file, specifying dependencies, tools, and configurations. - -2. **Prebuilding**: Gitpod continuously prebuilds environments for branches, similar to CI systems, preparing environments before they're needed. - -3. **Instant Start**: When a developer starts a workspace, Gitpod spins up a container with the prebuild results, providing a ready-to-code environment in seconds. - -4. **Development**: Developers work in a familiar IDE interface (VS Code, JetBrains) with full access to all necessary tools and resources. - -5. **Collaboration**: Developers can share workspaces, create snapshots, and collaborate in real-time. - -6. **Integration**: Gitpod integrates with GitHub, GitLab, Bitbucket, and Azure DevOps, providing seamless workflow integration. +1. **Define**: Environment specified in `.gitpod.yml` +2. **Prebuild**: Environments prepared in advance (CI-like) +3. **Start**: Container spins up with prebuild results +4. **Develop**: Work in familiar IDE (VS Code, JetBrains) +5. **Collaborate**: Share workspaces and snapshots +6. **Integrate**: Seamless Git platform workflow ## User Experience Goals - -- **Instant Readiness**: Environments should be ready to code within seconds -- **Familiarity**: Provide the same experience as local development -- **Seamless Integration**: Work naturally with existing tools and workflows -- **Minimal Configuration**: Make it easy to define and maintain environment configurations -- **Reliability**: Ensure consistent, reproducible environments every time -- **Performance**: Deliver responsive, high-performance development experiences -- **Accessibility**: Enable development from any device with a browser +| Goal | Description | +|------|-------------| +| Instant Readiness | Ready to code within seconds | +| Familiarity | Same experience as local development | +| Seamless Integration | Works with existing tools and workflows | +| Minimal Configuration | Easy environment definition | +| Reliability | Consistent, reproducible environments | +| Performance | Responsive development experience | +| Accessibility | Development from any device | ## Key Differentiators - -- **Ephemeral Environments**: Fresh environment for each task, eliminating configuration drift -- **Prebuilds**: Environments prepared before they're needed, eliminating wait time -- **Git Integration**: Deep integration with git platforms for seamless workflow -- **Open Source**: Core platform is open source, providing transparency and extensibility -- **Enterprise Ready**: Supports self-hosted deployments with enterprise-grade security and compliance features -- **IDE Flexibility**: Supports multiple IDEs including VS Code and JetBrains products +| Differentiator | Advantage | +|----------------|-----------| +| Ephemeral Environments | Fresh environment per task, eliminating drift | +| Prebuilds | Zero wait time with prepared environments | +| Git Integration | Seamless workflow with major platforms | +| Open Source | Transparency and extensibility | +| Enterprise Ready | Self-hosted with security and compliance | +| IDE Flexibility | Supports VS Code and JetBrains products | diff --git a/memory-bank/progress.md b/memory-bank/progress.md index d8176fb89e5cd7..d82d858648b6b3 100644 --- a/memory-bank/progress.md +++ b/memory-bank/progress.md @@ -2,88 +2,41 @@ ## Current Status -We are in the early stages of our work with the Gitpod codebase. The current status is: - -- **Memory Bank**: Initial setup complete with core files and component documentation -- **Codebase Understanding**: Basic overview obtained, with detailed understanding of key components -- **Component Documentation**: Documentation created for 33 key components -- **Development Environment**: Not yet configured -- **Task Identification**: Not yet started +| Area | Status | +|------|--------| +| Memory Bank | Initial setup complete with core files and component documentation | +| Codebase Understanding | Basic overview with detailed understanding of key components | +| Component Documentation | 33 service components and 11 API components documented | +| Development Environment | Not yet configured | +| Task Identification | Not yet started | ## What Works -Our current contributions: -- Memory bank structure established -- Component documentation for: - - blobserve - - content-service - - dashboard - - ws-manager-mk2 - - supervisor - - ws-daemon - - ide-service - - registry-facade - - image-builder-mk3 - - server - - proxy - - ws-proxy - - gitpod-cli - - gitpod-db - - gitpod-protocol - - ide - - ide-proxy - - ws-manager-bridge - - ide-metrics - - local-app - - public-api-server - - usage - - common-go - - workspacekit - - spicedb - - scrubber - - service-waiter - - docker-up - - image-builder-bob - - node-labeler - - openvsx-proxy - - scheduler-extender - - ipfs - -The existing functionality of the Gitpod platform: - -### Core Functionality -- Workspace creation from Git repositories -- IDE integration (VS Code, JetBrains) -- Prebuild system for faster startup -- Git platform integrations (GitHub, GitLab, Bitbucket, Azure DevOps) -- Containerized workspaces with Docker support -- Collaborative features - -### Development Infrastructure -- Leeway build system for component management -- Kubernetes-based deployment -- Microservices architecture -- Testing frameworks +### Our Contributions +- Memory bank structure with comprehensive documentation +- Documentation for 33 service components and 11 API components +- Server readiness probe implementation with database, SpiceDB, and Redis checks +- Standardized development workflows (PRD process) + +### Existing Platform Functionality +| Category | Features | +|----------|----------| +| **Core** | • Git-based workspaces
• IDE integration (VS Code, JetBrains)
• Prebuild system
• Git platform integrations
• Containerized workspaces
• Collaboration features | +| **Infrastructure** | • Leeway build system
• Kubernetes deployment
• Microservices architecture
• Testing frameworks | ## What's Left to Build ### Immediate Tasks - Explore component interactions - Set up local development environment -- Explore build system approaches (in-tree and Leeway) -- Test component builds for different component types -- Identify specific components for deeper exploration +- Explore build system approaches +- Test component builds +- Identify components for deeper exploration - Establish testing methodology - Create initial contribution plan -### Future Improvements -This section will be populated as we identify specific improvements or features to implement. - ## Known Issues - -As we begin working with the codebase, we have not yet identified specific issues. This section will be updated as issues are discovered during development and testing. - -### Potential Areas of Concern +No specific issues identified yet. Potential areas of concern: - Component coupling and dependencies - Build system complexity - Testing coverage @@ -91,147 +44,51 @@ As we begin working with the codebase, we have not yet identified specific issue ## Milestones -### Completed Milestones -- Memory bank initialization -- Component documentation structure -- Documentation of first set of key components (blobserve, content-service, dashboard, ws-manager-mk2) -- Documentation of second set of key components (supervisor, ws-daemon) -- Documentation of third set of key components (ide-service, registry-facade, image-builder-mk3) -- Documentation of fourth set of key components (server, proxy, ws-proxy) -- Documentation of fifth set of key components (gitpod-cli, gitpod-db) -- Documentation of sixth set of key components (gitpod-protocol) -- Documentation of seventh set of key components (ide, ide-proxy, ws-manager-bridge) -- Documentation of eighth set of key components (ide-metrics, local-app) -- Documentation of ninth set of key components (public-api-server, usage) -- Documentation of tenth set of key components (common-go, workspacekit) -- Documentation of eleventh set of key components (spicedb, scrubber, service-waiter) -- Documentation of twelfth set of key components (docker-up, image-builder-bob, node-labeler) -- Documentation of thirteenth set of key components (openvsx-proxy, scheduler-extender, ipfs) -- Documentation of first set of API components (content-service-api, ide-metrics-api, ide-service-api) -- Documentation of second set of API components (image-builder-api, local-app-api, registry-facade-api) -- Documentation of third set of API components (supervisor-api, usage-api, ws-daemon-api) -- Documentation of fourth set of API components (ws-manager-api, ws-manager-bridge-api) -- First feature implementation: Server readiness probe for database and SpiceDB connectivity - -### Upcoming Milestones +### Recently Completed +- Documentation of all service components (33) +- Documentation of all API components (11) +- Server readiness probe implementation +- Standardized development workflows + +### Upcoming - Development environment setup - First component deep dive - Initial contribution - First feature implementation ## Metrics and Progress Indicators - -As we progress, we'll track various metrics to gauge our progress and effectiveness: - -- **Code Understanding**: Percentage of components with detailed documentation -- **Test Coverage**: Percentage of code covered by tests -- **Issue Resolution**: Number of issues identified and resolved -- **Feature Implementation**: Number of features successfully implemented -- **Documentation Quality**: Completeness and accuracy of memory bank +- Code Understanding: % of components with detailed documentation +- Test Coverage: % of code covered by tests +- Issue Resolution: Number of issues identified/resolved +- Feature Implementation: Number of features implemented +- Documentation Quality: Completeness and accuracy ## Blockers and Dependencies - -No specific blockers or dependencies have been identified yet. This section will be updated as we encounter any obstacles that impede progress. +No specific blockers identified yet. ## Recent Progress -- **2/26/2025**: - - Memory bank initialized with core documentation files - - Created components subdirectory in memory bank - - Documented blobserve component - - Documented content-service component - - Documented dashboard component - - Documented ws-manager-mk2 component - - Documented supervisor component - - Documented ws-daemon component - - Documented ide-service component - - Documented registry-facade component - - Documented image-builder-mk3 component - - Documented server component - - Documented proxy component - - Documented ws-proxy component - - Documented gitpod-cli component - - Documented gitpod-db component - - Documented gitpod-protocol component - - Documented ide component - - Documented ide-proxy component - - Documented ws-manager-bridge component - - Documented ide-metrics component - - Documented local-app component - - Documented public-api-server component - - Documented usage component - - Documented common-go component - - Documented workspacekit component - - Documented spicedb component - - Documented scrubber component - - Documented service-waiter component - - Documented docker-up component - - Documented image-builder-bob component - - Documented node-labeler component - - Documented openvsx-proxy component - - Documented scheduler-extender component - - Documented ipfs component - -- **2/27/2025**: - - Documented build system approaches: - - In-tree builds using language-specific tools (yarn, go) - - Out-of-tree builds using Leeway - - Updated techContext.md with detailed build process information - - Updated .clinerules with build patterns and commands - - Added build information tracking to memory bank maintenance procedures - - Updated documentation approach to include API components: - - Removed exclusion of "*-api" components from documentation - - Identified 11 API components that need documentation - - Updated .clinerules to reflect new documentation approach - - Updated activeContext.md and progress.md with new documentation tasks - - Documented first set of API components: - - content-service-api: Interfaces for managing workspace content, blobs, logs, and IDE plugins - - ide-metrics-api: Interfaces for collecting metrics and error reports from IDE components - - ide-service-api: Interfaces for managing IDE configurations and resolving workspace IDE requirements - - Enhanced API component documentation with code generation information: - - Added details on how to regenerate code from protobuf definitions - - Documented the implementation details of the generation process - - Included instructions for building components after code regeneration - - Updated .clinerules to standardize API documentation with code generation sections - - Documented second set of API components: - - image-builder-api: Interfaces for building Docker images for workspaces - - local-app-api: Interfaces for communication between local machines and remote workspaces - - registry-facade-api: Interfaces for dynamically assembling workspace container images - - Documented third set of API components: - - supervisor-api: Interfaces for workspace management, terminal handling, and port forwarding - - usage-api: Interfaces for tracking, calculating, and managing workspace usage and billing - - ws-daemon-api: Interfaces for workspace content management and container operations - - Documented fourth set of API components: - - ws-manager-api: Interfaces for managing the lifecycle of workspaces in Kubernetes clusters - - ws-manager-bridge-api: Interfaces for dynamic management of workspace clusters - -- **3/17/2025**: - - Implemented server readiness probe: - - Created ReadinessController to check database and SpiceDB connectivity - - Updated server container module and route configuration - - Created PRD document for the readiness probe implementation - - Updated Kubernetes deployment configuration to add the readiness probe - - Added ConfigCat feature flag `server_readiness_probe` to control readiness checks - - Enhanced server readiness probe with Redis health check: - - Added Redis connectivity check to the ReadinessController - - Updated PRD document to include Redis in the list of checked dependencies - - Ensured the feature flag continues to work with the new Redis check - - Updated memory bank with new learnings: - - Added information about server health checks and critical dependencies - - Documented server architecture patterns and dependency injection - - Added information about Kubernetes deployment configuration - - Documented feature flag implementation for readiness probe - - Added Redis as a critical dependency for the server - - Established standardized development workflows: - - Created `workflows.md` to document standardized development processes - - Documented the Product Requirements Document (PRD) workflow - - Updated `activeContext.md` to reference the new workflows +### 3/17/2025 +- Implemented server readiness probe with database, SpiceDB, and Redis checks +- Created PRD document for the implementation +- Updated Kubernetes deployment configuration +- Added ConfigCat feature flag for readiness checks +- Updated memory bank with server architecture patterns +- Established standardized development workflows + +### 2/27/2025 +- Documented build system approaches (in-tree and Leeway) +- Updated documentation to include API components +- Documented all 11 API components +- Enhanced API documentation with code generation information + +### 2/26/2025 +- Initialized memory bank with core files +- Created component documentation structure +- Documented all 33 service components ## Next Evaluation Point - -The next evaluation of progress will occur after: -1. Setting up the development environment -2. Completing the first deep dive into a specific component -3. Identifying initial tasks for contribution - -At that point, this document will be updated to reflect new findings and progress. +Progress will be evaluated after: +1. Development environment setup +2. First component deep dive +3. Initial task identification diff --git a/memory-bank/projectbrief.md b/memory-bank/projectbrief.md index 5517a664d804ad..6835bbbdcec274 100644 --- a/memory-bank/projectbrief.md +++ b/memory-bank/projectbrief.md @@ -1,39 +1,38 @@ # Project Brief: Gitpod ## Overview -Gitpod is a developer platform that provides on-demand, pre-configured development environments that automatically integrate with any tool, library, or dependency required for creating software. It enables developers to spin up fresh, automated dev environments for each task, in the cloud, in seconds. +Gitpod provides on-demand, pre-configured development environments in the cloud. It enables developers to spin up fresh, automated dev environments for each task in seconds. ## Core Features -- **Dev Environments as Code**: Define editor extensions and dependencies in a declarative `.gitpod.yml` configuration -- **Prebuilt Dev Environments**: Continuously prebuilds all git branches similar to a CI server -- **Security**: Each workspace runs on a secured single-use container -- **Docker-based Workspaces**: Instantly starts a container in the cloud based on a Docker image -- **Git Platform Integration**: Seamlessly integrates with GitHub, GitLab, Bitbucket, and Azure DevOps -- **Integrated Code Reviews**: Native code reviews on any PR/MR -- **Collaboration**: Invite team members to your dev environment or share snapshots -- **Customizable Developer Experience**: Same capabilities as a Linux machine, pre-configured and optimized +| Feature | Description | +|---------|-------------| +| Dev Environments as Code | Declarative `.gitpod.yml` configuration | +| Prebuilt Environments | Continuous prebuilding of git branches | +| Security | Secured single-use containers | +| Docker-based | Cloud containers based on Docker images | +| Git Integration | GitHub, GitLab, Bitbucket, Azure DevOps | +| Code Reviews | Native reviews on PRs/MRs | +| Collaboration | Team invites and environment snapshots | +| Customizable | Linux capabilities, pre-configured | ## Project Goals -1. Provide instant, ready-to-code development environments -2. Eliminate environment setup and maintenance overhead -3. Enable consistent development experiences across teams -4. Improve collaboration and code review workflows -5. Enhance security through containerized environments -6. Support a wide range of development workflows and technologies +1. Instant, ready-to-code environments +2. Eliminate setup/maintenance overhead +3. Consistent team development experiences +4. Improved collaboration workflows +5. Enhanced containerized security +6. Support diverse development technologies ## Target Audience -- Software developers and development teams +- Software developers and teams - Open source contributors - Educational institutions -- Enterprise development organizations +- Enterprise organizations ## Project Scope -Gitpod encompasses: -- Cloud-based development environment infrastructure +- Cloud-based environment infrastructure - IDE integration (VS Code, JetBrains) - Git platform integrations - Prebuild and caching systems - Collaboration tools - Self-hosted and managed deployment options - -This project brief serves as the foundation for all other memory bank documents and defines the core requirements and goals of Gitpod. diff --git a/memory-bank/systemPatterns.md b/memory-bank/systemPatterns.md index 7c12b976674590..a76639d95b9394 100644 --- a/memory-bank/systemPatterns.md +++ b/memory-bank/systemPatterns.md @@ -2,271 +2,183 @@ ## System Architecture -Gitpod follows a microservices architecture composed of several key components that work together to provide the complete development environment platform. The system is designed to be: - -- **Scalable**: Handles many concurrent users and workspaces -- **Resilient**: Maintains availability despite component failures -- **Extensible**: Allows for adding new features and integrations -- **Secure**: Isolates workspaces and protects user data +Gitpod follows a microservices architecture designed to be **Scalable**, **Resilient**, **Extensible**, and **Secure**. ### High-Level Architecture ``` -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ Git Platforms │ │ User Browser │ │ IDE Clients │ -│ (GitHub, etc.) │ │ │ │ (VS Code, etc.)│ -└────────┬────────┘ └────────┬────────┘ └────────┬────────┘ - │ │ │ - ▼ ▼ ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ API Gateway │ -└─────────────────────────────────────────────────────────────────┘ - │ │ │ - ▼ ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ Auth Service │ │ Dashboard │ │ IDE Service │ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ │ │ - └───────────────┬───────┴───────────────┬───────┘ - │ │ - ▼ ▼ -┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐ -│ WS Manager │ │ Image Builder │ │ Content Service│ -└─────────────────┘ └─────────────────┘ └─────────────────┘ - │ │ │ - ▼ ▼ ▼ -┌─────────────────────────────────────────────────────────────────┐ -│ Kubernetes Infrastructure │ -└─────────────────────────────────────────────────────────────────┘ +┌─────────────┐ ┌─────────────┐ ┌─────────────┐ +│Git Platforms│ │User Browser │ │IDE Clients │ +└──────┬──────┘ └──────┬──────┘ └──────┬──────┘ + │ │ │ + └────────┬───────┴────────┬───────┘ + ▼ ▼ + ┌───────────────────────────────┐ + │ API Gateway │ + └───────────────────────────────┘ + │ │ + ┌────────┴───────┐ ┌─────┴────────┐ + │ Auth Service │ │ Dashboard │ + └────────────────┘ └──────────────┘ + │ │ + ┌────────┴───────┐ ┌─────┴────────┐ + │ WS Manager │ │Image Builder │ + └────────────────┘ └──────────────┘ + │ │ + ┌────────┴────────────────┴────────┐ + │ Kubernetes Infrastructure │ + └───────────────────────────────────┘ ``` ## Key Components ### Core Services -1. **Workspace Manager (ws-manager)**: Orchestrates workspace lifecycle, managing creation, starting, stopping, and deletion of workspaces. - -2. **Workspace Daemon (ws-daemon)**: Runs on each node, managing workspace resources, file system operations, and runtime aspects. - -3. **Image Builder**: Builds Docker images for workspaces based on configuration and caches them for quick startup. - -4. **Content Service**: Manages file content, including git operations, file synchronization, and backup. - -5. **IDE Service**: Manages the IDE instances (VS Code, JetBrains) that run in workspaces. - -6. **Dashboard**: Web UI for managing workspaces, projects, and user settings. - -7. **Auth Service**: Handles authentication and authorization across the platform. - -8. **Proxy**: Routes traffic to the appropriate services and workspaces. +| Component | Purpose | +|-----------|---------| +| **Workspace Manager** | Orchestrates workspace lifecycle (creation, starting, stopping, deletion) | +| **Workspace Daemon** | Manages workspace resources and file system operations on each node | +| **Image Builder** | Builds and caches Docker images for workspaces | +| **Content Service** | Manages file content, git operations, and synchronization | +| **IDE Service** | Manages IDE instances (VS Code, JetBrains) | +| **Dashboard** | Web UI for workspace and user management | +| **Auth Service** | Handles authentication and authorization | +| **Proxy** | Routes traffic to appropriate services and workspaces | ### Supporting Components -1. **Registry Facade**: Provides efficient access to container images. - -2. **Blobserve**: Serves static content from container images. - -3. **Supervisor**: Runs inside each workspace, managing the workspace's internal services. - -4. **Public API**: Provides programmatic access to Gitpod functionality. +| Component | Purpose | +|-----------|---------| +| **Registry Facade** | Efficient access to container images | +| **Blobserve** | Serves static content from container images | +| **Supervisor** | Manages workspace internal services | +| **Public API** | Programmatic access to Gitpod functionality | -## Project Structure Patterns +## Project Structure & Code Style -- **Component Organization**: The project is organized into components, each with its own directory in the `components/` folder. -- **API Definitions**: API definitions are typically in separate packages with `-api` suffix (e.g., `content-service-api`). -- **Protocol Buffers**: gRPC service definitions use Protocol Buffers (`.proto` files). -- **Build Configuration**: Each component has a `BUILD.yaml` file defining its build configuration. -- **Docker Configuration**: Components that run as containers have a `leeway.Dockerfile`. +### Project Organization +- Components in `components/` directory +- API definitions in `-api` suffixed packages +- gRPC services defined with Protocol Buffers (`.proto`) +- `BUILD.yaml` for build configuration +- `leeway.Dockerfile` for containerized components -## Code Style Preferences - -### Go Code -- Follow standard Go conventions (gofmt). -- Error handling with explicit checks. -- Context propagation for cancellation. -- Structured logging. - -### TypeScript Code -- Use TypeScript for type safety. -- React for UI components. -- Functional components with hooks. -- ESLint and Prettier for formatting. +### Code Style +| Language | Key Practices | +|----------|---------------| +| **Go** | Standard conventions (gofmt), explicit error handling, context propagation, structured logging | +| **TypeScript** | Type safety, React for UI, functional components with hooks, ESLint/Prettier | ## Design Patterns -### Microservices Pattern -Gitpod is built as a collection of loosely coupled services, each with a specific responsibility. This enables independent scaling, deployment, and maintenance of components. - -### Container Orchestration -Kubernetes is used to manage the deployment, scaling, and operation of application containers across clusters of hosts. - -### Event-Driven Architecture -Components communicate through events for asynchronous operations, improving scalability and resilience. - -### API Gateway Pattern -A central API gateway routes requests to appropriate services, handling cross-cutting concerns like authentication. - -### Immutable Infrastructure -Workspaces are treated as immutable, with changes to configuration resulting in new environments rather than modifications to existing ones. +| Pattern | Implementation | +|---------|----------------| +| **Microservices** | Loosely coupled services with specific responsibilities | +| **Container Orchestration** | Kubernetes for deployment, scaling, and operations | +| **Event-Driven Architecture** | Asynchronous communication for scalability and resilience | +| **API Gateway** | Central routing with cross-cutting concerns handling | +| **Immutable Infrastructure** | Configuration changes create new environments | ## Component Relationships -### Workspace Lifecycle - -1. User requests a workspace through the dashboard or Git integration. -2. Auth service validates the request. -3. Workspace Manager creates the workspace specification. -4. Image Builder ensures the required image is available. -5. Workspace Manager instructs Kubernetes to create the workspace pod. -6. Workspace Daemon initializes the workspace environment. -7. Supervisor starts within the workspace. -8. IDE Service connects the IDE to the workspace. -9. Proxy routes user traffic to the workspace. - -The key components involved in the workspace lifecycle include: - - workspace-manager-mk2 - - ws-daemon - - ws-manager-bridge - - server - - image-builder-mk3 +### Workspace Lifecycle Flow +1. User requests workspace → Auth validates → WS Manager creates spec +2. Image Builder ensures image availability → WS Manager creates K8s pod +3. WS Daemon initializes environment → Supervisor starts +4. IDE Service connects IDE → Proxy routes traffic -### Data Flow +**Key components**: ws-manager-mk2, ws-daemon, ws-manager-bridge, server, image-builder-mk3 -1. **User Code**: Managed by Content Service, synchronized between workspace and git repositories -2. **Configuration**: Stored in database, applied by Workspace Manager during workspace creation -3. **Build Artifacts**: Cached by Image Builder for reuse in future workspaces -4. **User Data**: Stored in database, accessed through Dashboard and API +### Data Flow +- **User Code**: Content Service manages synchronization +- **Configuration**: Database storage, applied by WS Manager +- **Build Artifacts**: Cached by Image Builder +- **User Data**: Database storage, accessed via Dashboard/API ## Key Technical Decisions -1. **Kubernetes-Based**: Leveraging Kubernetes for container orchestration provides scalability and standardized infrastructure management. - -2. **Multi-IDE Support**: Supporting multiple IDEs (VS Code, JetBrains) increases flexibility for users with different preferences. - -3. **Prebuild System**: Prebuilding environments before they're needed significantly reduces startup times. - -4. **Workspace Pods**: Each workspace runs in its own Kubernetes pod, providing isolation and resource management. - -5. **TypeScript and Go**: Core services are implemented in TypeScript (user-facing) and Go (system-level), balancing developer productivity and performance. - -6. **gRPC Communication**: Internal services communicate using gRPC for efficient, typed communication. - -7. **Leeway Build System**: Custom build system for managing the complex dependencies between components. - -8. **Kubernetes Deployment Configuration**: All code that defines Kubernetes objects for deployable components lives in `install/installer`. This centralized approach ensures consistent deployment patterns across all components. - -9. **Public API Architecture**: External programmatic access is provided via `gitpod.v1` gRPC services. These services are defined using Protocol Buffers (`.proto` files) located in the `components/public-api/gitpod/v1/` directory. The `server` component (TypeScript/Express.js based) directly implements and hosts these gRPC services. The `public-api-server` component (Go based) can act as an external-facing gRPC gateway. - -10. **Authorization with SpiceDB**: Fine-grained, relationship-based access control (ReBAC) is managed by SpiceDB. The authorization schema, defining resources, relationships, and permissions, is located in `components/spicedb/schema/schema.yaml`. Service implementations (e.g., within the `server` component) query SpiceDB to enforce these permissions. +| Decision | Rationale | +|----------|-----------| +| **Kubernetes-Based** | Scalability and standardized infrastructure | +| **Multi-IDE Support** | Flexibility for different user preferences | +| **Prebuild System** | Significantly reduced startup times | +| **Workspace Pods** | Isolation and resource management | +| **TypeScript & Go** | Balance of developer productivity and performance | +| **gRPC Communication** | Efficient, typed service communication | +| **Leeway Build System** | Management of complex component dependencies | +| **K8s Deployment Config** | Centralized in `install/installer` for consistency | +| **Public API Architecture** | gRPC services defined in Protocol Buffers | +| **SpiceDB Authorization** | Fine-grained relationship-based access control | ## Development Workflows -### Product Requirements Document (PRD) Workflow -Gitpod uses a structured PRD workflow for feature development to ensure proper planning, documentation, and implementation: - -1. **Requirements Gathering** (Plan Mode): Understand the problem, explore existing components, gather information. -2. **PRD Creation** (Plan Mode): Create a detailed document in `prd/` (e.g., `NNN-featurename.md`) with standardized sections: Overview, Background, Requirements, Implementation Details, Testing, Deployment Considerations, Future Improvements -3. **Implementation Planning** (Plan Mode): Identify files to modify and plan the approach -4. **Implementation** (Act Mode): Create/modify necessary files following the plan -5. **Documentation Update** (Act Mode): Update memory bank with new knowledge -6. **Verification**: Ensure implementation meets requirements and documentation is complete - -This workflow ensures thorough planning, clear documentation, and knowledge preservation for all feature development. +### PRD Workflow +1. **Requirements Gathering** (Plan): Understand problem, explore components +2. **PRD Creation** (Plan): Create document with standardized sections +3. **Implementation Planning** (Plan): Identify files to modify +4. **Implementation** (Act): Create/modify files +5. **Documentation Update** (Act): Update memory bank +6. **Verification**: Validate implementation meets requirements ### Build Approaches -- **In-tree builds** (for development in a Gitpod workspace): - - TypeScript components: Use `yarn` commands defined in `package.json`. - - `yarn build`: Compile the component. - - `yarn test`: Run tests. - - `yarn lint`: Check code style. - - `yarn watch`: Watch for changes and rebuild. - - Go components: Use standard Go tools. - - `go build ./...`: Build all packages. - - `go test ./...`: Test all packages. - - `go run main.go`: Build and run. -- **Out-of-tree builds** (Leeway, primarily used in CI): - - `leeway build components/component-name:app`: Build a specific component. - - `leeway build -D components/component-name:app`: Build with dependencies. - - `leeway exec --package components/component-name:app -- command`: Run a command for a package. - -### Testing - -- Unit tests alongside code. -- Integration tests in separate directories. -- End-to-end tests in the `test/` directory. -- Component-specific test commands in `package.json` (for TypeScript). -- Go tests use standard `go test` command. - -### Local Development - -- Use Gitpod workspaces for development (dogfooding). -- Components can be run individually for testing. -- Preview environments for testing changes. -- Use in-tree builds for rapid iteration during development. - -## Known Challenges - -- **Build System Complexity**: The Leeway build system has a learning curve. -- **Component Dependencies**: Understanding dependencies between components can be challenging. -- **Testing Environment**: Setting up proper testing environments for all components. - -## Tool Usage Patterns - -- **VS Code**: Primary IDE for TypeScript development. -- **GoLand/IntelliJ**: Often used for Go development. -- **Docker**: Used for containerized development and testing. -- **kubectl**: Used for interacting with Kubernetes clusters. -- **Werft**: CI/CD system for automated builds and tests. +- **In-tree builds** (for development in a Gitpod workspace): + - TypeScript components: Use `yarn` commands defined in `package.json`. + - `yarn build`: Compile the component. + - `yarn test`: Run tests. + - `yarn lint`: Check code style. + - `yarn watch`: Watch for changes and rebuild. + - Go components: Use standard Go tools. + - `go build ./...`: Build all packages. + - `go test ./...`: Test all packages. + - `go run main.go`: Build and run. +- **Out-of-tree builds** (Leeway, primarily used in CI): + - `leeway build components/component-name:app`: Build a specific component. + - `leeway build -D components/component-name:app`: Build with dependencies. + - `leeway exec --package components/component-name:app -- command`: Run a command for a package. + + +### Testing & Development +- Unit tests alongside code +- Integration tests in separate directories +- End-to-end tests in `test/` directory +- Gitpod workspaces for development (dogfooding) +- Preview environments for testing ## Documentation Patterns +- README.md for component purpose +- API docs from Protocol Buffer definitions +- Component docs in `memory-bank/components/` +- Standardized structure (Overview, Purpose, Architecture, etc.) +- API components include Code Generation section -- **README.md**: Each component should have a README explaining its purpose. -- **API Documentation**: Generated from Protocol Buffer definitions. -- **Memory Bank Documentation**: - - Component-specific documentation is stored in `memory-bank/components/` directory. - - Each component gets its own markdown file with detailed information about its purpose, architecture, and implementation. - - Component documentation should focus on both service components that implement business logic and API components that define interfaces. - - Documentation follows a consistent structure with sections for Overview, Purpose, Architecture, Key Features, etc. - - API component documentation should include a "Code Generation and Building" section that explains: - - How to regenerate code from protobuf definitions. - - The implementation details of the generation process. - - How to build components that depend on the API after regeneration. +## Known Challenges +- Build system complexity +- Component dependency understanding +- Testing environment setup ## Evolution of Project Decisions -This section captures how and why certain architectural and design decisions were made. - ### Memory Bank Organization - -- **Component Documentation**: The decision to create separate documentation files for each component in the `memory-bank/components/` directory was made to: - 1. Provide a clear, organized structure for component documentation. - 2. Allow for detailed documentation of each component's purpose, architecture, and implementation. - 3. Make it easier to find information about specific components. - 4. Enable incremental updates to component documentation without affecting other files. -- **API Component Documentation**: The decision to include "*-api" components in separate documentation was made because: - 1. API components define critical interfaces between services. - 2. Understanding these interfaces is essential for developers working across components. - 3. API documentation helps clarify system boundaries and communication patterns. - 4. Separate documentation makes it easier to track API changes and versioning. - -## User Preferences - -This section will be updated as specific user preferences for working with the codebase are identified. +- Component documentation in separate files for: + - Clear organization + - Detailed documentation + - Easy information finding + - Incremental updates +- API component documentation separated to: + - Define critical interfaces + - Clarify system boundaries + - Track API changes and versioning ## Build and Test Information - -When working with components, it is crucial to document the following: - -- **Build Commands**: Document any new or component-specific build commands encountered. -- **Test Commands**: Document how to run tests for each component. -- **Dependencies**: Note any special dependencies required for building or testing. -- **Common Issues**: Document common build or test issues and their solutions. -- **Performance Considerations**: Note any performance considerations for builds. - -Whenever new build patterns or commands are encountered, the following should be updated: -1. The relevant component documentation in `memory-bank/components/`. -2. This `systemPatterns.md` file (previously the learning journal) with general patterns. -3. The `techContext.md` file if it represents a significant pattern. - -This information is critical for effective development work, as being able to build and test components is fundamental to making changes and verifying their correctness. +Document for each component: +- Build commands +- Test commands +- Dependencies +- Common issues +- Performance considerations + +Update documentation in: +1. Component docs in `memory-bank/components/` +2. This file for general patterns +3. `techContext.md` for significant patterns diff --git a/memory-bank/techContext.md b/memory-bank/techContext.md index fb5cb54aad0fba..164b256fe5ff87 100644 --- a/memory-bank/techContext.md +++ b/memory-bank/techContext.md @@ -2,185 +2,83 @@ ## Technologies Used -### Programming Languages -- **Go**: Used for system-level services, performance-critical components, and Kubernetes integration -- **TypeScript**: Used for user-facing services, dashboard, and IDE integration -- **Java**: Used for JetBrains IDE integration -- **Shell Scripts**: Used for automation, build scripts, and development tools - -### Frameworks & Libraries -- **React**: Frontend dashboard UI -- **gRPC**: Service-to-service communication -- **Protocol Buffers**: Data serialization -- **Kubernetes API**: Workspace orchestration -- **Docker API**: Container management -- **Leeway**: Custom build system for managing component dependencies - -### Infrastructure -- **Kubernetes**: Container orchestration platform -- **Docker**: Containerization technology -- **MySQL**: Primary database for persistent data. - - TypeORM is used as the ORM -- **Redis**: Caching and ephemeral data storage -- **Helm**: Kubernetes package management -- **Terraform**: Infrastructure as code for deployment - -### Development Tools -- **VS Code**: Primary IDE for TypeScript/JavaScript development -- **GoLand/IntelliJ**: IDE for Go development -- **pre-commit**: Git hooks for code quality -- **ESLint/Prettier**: Code formatting and linting -- **Werft**: CI/CD system +| Category | Technologies | +|----------|--------------| +| **Programming Languages** | • Go: System-level services, K8s integration
• TypeScript: User-facing services, dashboard
• Java: JetBrains IDE integration
• Shell: Automation, build scripts | +| **Frameworks & Libraries** | • React: Frontend UI
• gRPC: Service communication
• Protocol Buffers: Data serialization
• Kubernetes API: Workspace orchestration
• Docker API: Container management
• Leeway: Custom build system | +| **Infrastructure** | • Kubernetes: Container orchestration
• Docker: Containerization
• MySQL/TypeORM: Persistent storage
• Redis: Caching, ephemeral storage
• Helm: K8s package management
• Terraform: Infrastructure as code | +| **Development Tools** | • VS Code: TypeScript/JS development
• GoLand/IntelliJ: Go development
• pre-commit: Git hooks
• ESLint/Prettier: Code formatting
• Werft: CI/CD system | ## Development Setup ### Local Development -The project uses a Gitpod-based development workflow (dogfooding), with the following key aspects: - -1. **Gitpod Workspace**: Development occurs in Gitpod workspaces defined by `.gitpod.yml` -2. **Component-Based**: Each component can be developed and tested independently -3. **Leeway Build System**: Manages dependencies between components -4. **Dev Containers**: Development occurs in containers that mirror production +- Gitpod-based workflow (dogfooding) +- Component-based independent development +- Leeway build system for dependency management +- Containerized development mirroring production ### Build Process -Gitpod uses two primary approaches for building components: - -#### 1. In-tree Builds (Primary for Local Development) -Components are built directly in the workspace using language-specific tools: - -- **TypeScript/JavaScript Components**: - - Commands are defined in each component's `package.json` - - Common commands: - - `yarn build`: Compiles the component - - `yarn test`: Runs unit tests - - `yarn lint`: Checks code style - - `yarn watch`: Watches for changes and rebuilds automatically - - Example (from server component): - ```bash - cd components/server - yarn build # Build the component - yarn test:unit # Run unit tests - yarn test:db # Run database tests - ``` - -- **Go Components**: - - Use standard Go tools - - Common commands: - - `go build`: Compiles the component - - `go test`: Runs tests - - `go run`: Builds and runs the component - - Example: - ```bash - cd components/ws-daemon - go build ./... # Build all packages - go test ./... # Test all packages - ``` - -#### 2. Leeway Builds (Out-of-tree, Primary for CI) -Leeway is a custom build tool that: -- Copies relevant sources into a separate file tree -- Manages complex dependencies between components -- Generates build artifacts for CI/CD pipelines -- Can also be run from inside the workspace - -Common Leeway commands: -```bash -leeway build components/server:app # Build a specific component -leeway build -D components/server:app # Build with dependencies -leeway exec --package components/server:app -- yarn test # Run a command for a package -``` - -#### 3. Component Packaging -- **Docker Images**: Components are packaged as Docker images using `leeway.Dockerfile` files -- **Helm Charts**: Deployment configurations are managed as Helm charts for Kubernetes deployment +| Approach | Description | Common Commands | +|----------|-------------|----------------| +| **In-tree** (Local Dev) | • Direct in workspace
• Language-specific tools | **TypeScript**:
`yarn build` - Compile
`yarn test` - Run tests
`yarn lint` - Check style
`yarn watch` - Auto-rebuild

**Go**:
`go build ./...` - Build packages
`go test ./...` - Run tests
`go run main.go` - Build and run | +| **Leeway** (CI/CD) | • Out-of-tree builds
• Manages dependencies
• Generates artifacts | `leeway build components/name:app`
`leeway build -D components/name:app`
`leeway exec --package components/name:app -- cmd` | +| **Packaging** | • Docker images via `leeway.Dockerfile`
• Helm charts for K8s deployment | | ### Testing Strategy -1. **Unit Tests**: Component-level tests for individual functions and classes -2. **Integration Tests**: Tests for interactions between components -3. **End-to-End Tests**: Tests for complete user workflows -4. **Preview Environments**: Dedicated test environments for feature validation +- Unit tests: Component-level function/class testing +- Integration tests: Cross-component interaction testing +- End-to-end tests: Complete workflow validation +- Preview environments: Dedicated feature testing ## Technical Constraints -### Performance Requirements -- **Workspace Startup Time**: Workspaces should start in under 10 seconds when prebuilt -- **IDE Responsiveness**: IDE should maintain low latency even over internet connections -- **Resource Efficiency**: Efficient use of cluster resources to maximize density - -### Security Constraints -- **Workspace Isolation**: Strong isolation between user workspaces -- **Least Privilege**: Components operate with minimal required permissions -- **Data Protection**: Secure handling of user code and credentials -- **Network Security**: Controlled network access between components - -### Scalability Requirements -- **Horizontal Scaling**: All components must support horizontal scaling -- **Multi-Cluster Support**: Support for distributing workspaces across multiple clusters -- **Resource Limits**: Enforced limits on workspace resource consumption - -### Compliance Requirements -- **GDPR Compliance**: Proper handling of user data -- **SOC2 Compliance**: Security and availability controls -- **Data Residency**: Support for region-specific data storage +| Category | Requirements | +|----------|-------------| +| **Performance** | • Workspace startup < 10s when prebuilt
• Low-latency IDE over internet
• Efficient resource utilization | +| **Security** | • Strong workspace isolation
• Least privilege component operation
• Secure user code/credential handling
• Controlled network access | +| **Scalability** | • Horizontal scaling for all components
• Multi-cluster workspace distribution
• Enforced workspace resource limits | +| **Compliance** | • GDPR: User data handling
• SOC2: Security and availability
• Data residency: Region-specific storage | ## Dependencies ### External Dependencies -- **Git Providers**: GitHub, GitLab, Bitbucket, Azure DevOps -- **Container Registries**: Docker Hub, GCR, ECR -- **Cloud Providers**: GCP, AWS, Azure (for self-hosted deployments) -- **IDE Platforms**: VS Code, JetBrains IDEs +| Category | Dependencies | +|----------|-------------| +| **Git Providers** | GitHub, GitLab, Bitbucket, Azure DevOps | +| **Container Registries** | Docker Hub, GCR, ECR | +| **Cloud Providers** | GCP, AWS, Azure | +| **IDE Platforms** | VS Code, JetBrains IDEs | ### Internal Dependencies -The system has several internal dependencies between components: - -1. **Workspace Manager** depends on: - - Kubernetes API - - Image Builder - - Content Service - -2. **Dashboard** depends on: - - Auth Service - - Workspace Manager - - Content Service - -3. **IDE Service** depends on: - - Workspace Manager - - Content Service - - Supervisor - -4. **Content Service** depends on: - - Git providers - - Storage systems - -5. **Supervisor** depends on: - - IDE Service - - Content Service - -### Third-Party Libraries -Key third-party libraries and their purposes: - -1. **Kubernetes Client Libraries**: Interaction with Kubernetes API -2. **Docker Client Libraries**: Container management -3. **Database Drivers**: MySQL, Redis connectivity -4. **gRPC/Protocol Buffers**: Service communication -5. **React and UI Libraries**: Dashboard frontend -6. **Authentication Libraries**: OAuth, JWT handling +| Component | Depends On | +|-----------|------------| +| **Workspace Manager** | Kubernetes API, Image Builder, Content Service | +| **Dashboard** | Auth Service, Workspace Manager, Content Service | +| **IDE Service** | Workspace Manager, Content Service, Supervisor | +| **Content Service** | Git providers, Storage systems | +| **Supervisor** | IDE Service, Content Service | + +### Key Third-Party Libraries +- Kubernetes/Docker client libraries +- Database drivers (MySQL, Redis) +- gRPC/Protocol Buffers +- React and UI libraries +- Authentication libraries (OAuth, JWT) ## Development Workflow - -1. **Feature Planning**: Features are planned and designed -2. **Implementation**: Code is written and tested locally -3. **Code Review**: Changes are reviewed through pull requests +1. **Feature Planning**: Requirements and design +2. **Implementation**: Local development and testing +3. **Code Review**: Pull request review process 4. **CI/CD**: Automated testing and deployment -5. **Preview**: Features are tested in preview environments -6. **Release**: Changes are deployed to production +5. **Preview**: Feature validation in test environments +6. **Release**: Production deployment ## Technical Debt and Challenges - -1. **Component Coupling**: Some components have tight coupling that could be improved -2. **Test Coverage**: Certain areas lack comprehensive test coverage -3. **Documentation**: Some internal APIs lack detailed documentation -4. **Legacy Components**: Some older components need modernization -5. **Build System Complexity**: The custom build system has a learning curve +| Challenge | Description | +|-----------|-------------| +| **Component Coupling** | Some tight coupling between components | +| **Test Coverage** | Incomplete coverage in certain areas | +| **Documentation** | Some internal APIs lack detailed docs | +| **Legacy Components** | Older components needing modernization | +| **Build System** | Learning curve for custom build system |