Billing controller excludes WorkspaceInstances which are already included in previous invoices

[easyCZ]

The billing controller will need to be given the full set of WorkspaceInstances (which the UsageController fetched) to perform it's run. This is because in a given billing period (May 1 - May 31), a user can have potentially a large number of subscriptions (not in parallel) - create then cancel and repeat. For the cancelled subscriptions, they will of course need to settle it but we must not charge them for any usage from a cancelled subscription (but within that month). And the Billing Controller is the only component which should have access to Stripe to retrieve this data.
In practice, the controller will need to exclude all WorkspaceInstances which match previously already billed WorkspaceInstances.

Architecture

System architecture:

Interactions with Stripe and the content-service:

Tasks

Done

• Set up a placeholder webhook endpoint on the public API ([public-api] Add a placeholder Stripe webhook #11806)
• Configure ingress to the webhook in preview and live environments. ([usage] Configure ingress for Stripe webhook #11866)
• Ensure that the billing controller can exclude instances from Stripe invoices based on start date.
  • [usage] Add billInstancesAfter config setting #11882
  • [installer] Support setting billInstancesAfter #11883

Webhook registration

• Register webhook in Stripe (Stripe docs)
• Add webhook signing secret as K8S secret and mount it into the public-api pod.
• Implement Stripe signature verification on the webhook. (Stripe docs)

Implement FinalizeInvoice RPC

• Define FinalizeInvoice RPC on usage that the public api will invoke from the webhook handler. ([UBP] Add placeholder implementation of FinalizeInvoice #11949)
• Connect webhook so that it invokes FinalizeInvoice RPC when an invoice.finalized event is received. ([UBP] Connect Stripe webhook to FinalizeInvoice RPC #11987)
• Retrieve Stripe invoice for the requested invoice id
• Download usage report from content-service (report to download is indicated by Stripe metadata)
• Find all sessions in the report for the teamID and mark them as billed in the database.
• Create d_b_billed_sessions in the server and related object models in the BillingService #11867
• Implement method setBilledSession (for Chargebee) in the Billing Service #11884

Other

• Add report ids to invoice metadata
  • So that when the invoice finalizes we know which report to retrieve from the content service.

Notes

• To add the webhook signing secret to the cluster we can follow the same pattern as was done for the Stripe API keys - create the secret in the GCP secret dashboard (for preview, staging and production)
  • Add secrets to staging and prod clusters via terraform, eg these PRs for preview [1] and staging/production [1,2].

[easyCZ]

Related (internal) conversation

[easyCZ]

Manually configured staging environment to send invoice.finalized webhooks

[easyCZ]

Stripe signing secret is now loaded into staging/production with https://github.com/gitpod-io/ops/pull/3974

[easyCZ]

A WIP for finalizing invoices. Currently, the finalization event doens't tell us which AttributionID it is for. It only gives us the Invoice ID.

We'll need to link the InvoiceID to the attribution ID used for the invoice. Likely, this will need to come from the subscription when it is made originally. This currently contains the teamId field which could be used.

[easyCZ]

We won't be going with this approach. Instead we'll approach it with #12741