- Release Signoff Checklist
- Introduction
- Motivation
- User Stories
- Goals
- COSI architecture
- COSI API
- Bucket Creation
- Generating Access Credentials for Buckets
- Attaching Bucket Information to Pods
- Sharing Buckets
- Accessing existing Buckets
- Bucket deletion
- Usability
- Object Lifecycle
- COSI API Reference
- COSI Driver
- Test Plan
- Graduation Criteria
- Alternatives Considered
- (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
- (R) KEP approvers have approved the KEP status as
implementable - (R) Design details are appropriately documented
- (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input
- (R) Graduation criteria is in place
- (R) Production readiness review completed
- Production readiness review approved
- "Implementation History" section is up-to-date for milestone
- User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
- Supporting documentation e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
This proposal introduces the Container Object Storage Interface (COSI), a standard for provisioning and consuming object storage in Kubernetes.
File and block storage are treated as first class citizens in the Kubernetes ecosystem via CSI. Workloads using CSI volumes enjoy the benefits of portability across vendors and across Kubernetes clusters without the need to change application manifests. An equivalent standard does not exist for Object storage.
Object storage has been rising in popularity in the recent years as an alternative form of storage to filesystems and block devices. Object storage paradigm promotes disaggregation of compute and storage. This is done by making data available over the network, rather than locally. Disaggregated architectures allow compute workloads to be stateless, which consequently makes them easier to manage, scale and automate.
We define 3 kinds of stakeholders:
- Admins - Members with escalated privileges and authority to manage site wide/cluster wide policies to control access, assign storage quotas and other concerns related to hardware or application resources.
- Application Developer/Operator (User) - Members with privileges to run workloads, request storage for them, and manage resources inside a specific namespace.
- Object Storage Provider (OSP) - Vendors offering Object Storage capabilities
- Establish cluster/site wide policies for data redundancy, durability and other data related parameters
- Establish cluster/site wide policies for access control over buckets
- Enable/restrict users who can create, delete and reuse buckets
- Request a bucket to be created and/or provisioned for workloads
- Request access to be created/deleted for a specific bucket
- Request deletion of a created bucket
- Integrate with the official standard for orchestrating Object Storage resources in Kubernetes
- Support automated Bucket Creation
- Support automated Bucket Deletion
- Support automated Access Credential Generation
- Support automated Access Credential Revokation
- Support automated Bucket Provisioning for workloads (enabling pods to access buckets)
- Support Bucket Reuse (use existing buckets via COSI)
- Support Automated Bucket Sharing across namespaces
- Support workload portability across clusters
- Achieve the above goals in a vendor neutral manner
- Standardize mechanism for third-party vendors to integrate easily with COSI
- Allow users (non-admins) to create and utilize buckets (self-service)
- Establish best-in-class Access Control practices for bucket creation and sharing
- Data Plane API standardization will not be addressed by this project
- Bucket Mutation will not be supported as of now
Since this is an entirely new feature, it is possible to implement this completely out of tree. The following components are proposed for this architecture:
- COSI ControllerManager
- COSI Sidecar
- COSI Driver
- The COSI ControllerManager is the central controller that validates, authorizes and binds COSI created buckets to BucketClaims. Only one active instance of ControllerManager should be present.
- The COSI Sidecar is the point of integration between COSI and drivers. All operations that require communication with the OSP is triggered by the Sidecar using gRPC calls to the driver. One active instance of Sidecar should be present for each driver.
- The COSI driver communicates with the OSP to conduct Bucket related operations.
More information about COSI driver is here
COSI defines 5 new API types
Detailed information about these API types are provided inline with user stories.
Here is a TL;DR version:
- BucketClaims/Bucket are similar to PVC/PV.
- BucketClaim is used to request generation of new buckets.
- Buckets represent the actual Bucket.
- BucketClass is similar to StorageClass. It is meant for admins to define and control policies for Bucket Creation
- BucketAccess is required before a bucket can be "attached" to a pod.
- BucketAccess both represents the attachment status and holds a pointer to the access credentials secret.
- BucketAccessClass is meant for admins to control authz/authn for users requesting access to buckets.
The following stakeholders are involved in the lifecycle of bucket creation:
- Users - request buckets to be created for their workload
- Admins - setup COSI drivers in the cluster, and specify their configurations
Here are the steps for creating a Bucket:
The BucketClass represents a set of common properties shared by multiple buckets. It is used to specify the driver for creating Buckets, and also for configuring driver-specific parameters. More information about BucketClass is here
The BucketClaim is a claim to create a new Bucket. This resource can be used to specify the bucketClassName, which in-turn includes configuration pertinent to bucket creation. More information about BucketClaim is available here
BucketClaim - bcl1 BucketClass - bc1
|------------------------------| |--------------------------------|
| metadata: | | deletionPolicy: delete |
| namespace: ns1 | | driverName: s3.amazonaws.com |
| spec: | | parameters: |
| bucketClassName: bc1 | | key: value |
| protocols: | |--------------------------------|
| - s3 |
|------------------------------|
The ControllerManager creates a Bucket object by copying fields over from both the BucketClaim and BucketClass. This step can only proceed if the BucketClass exists. This Bucket object is an intermediate object, with its status condition BucketReady to false. This indicates that the Bucket is yet to be created in the OSP.
More information about Bucket is here
Bucket - bcl-$uuid
|--------------------------------------|
| name: bcl-$uuid |
| spec: |
| bucketClassName: bc1 |
| protocols: |
| - s3 |
| parameters: |
| key: value |
| driverName: s3.amazonaws.com |
| status: |
| conditions: |
| - type: BucketReady |
| status: False |
|--------------------------------------|
The COSI sidecar, which runs alongside each of the drivers, listens for Bucket events. All but the sidecar with the specified driver will ignore the Bucket object. Only the appropriate sidecar will make a gRPC request to create a Bucket in the OSP.
More information about COSI gRPC API is here
COSI Driver (s3.amazonaws.com)
|------------------------------------------|
| grpc DriverCreateBucket({ |
| "name": "bcl-$uuid", |
| "protocols": ["s3"], |
| "parameters": { |
| "key": "value" |
| } |
| }) |
|------------------------------------------|
Once the Bucket is successfully created in the OSP, and a successful status is retrieved from the gRPC request, sidecar sets BucketReadycondition to True. At this point, the Bucket is ready to be utilized by workloads.
The following stakeholders are involved in the lifecycle of access credential generation:
- Users - request access to buckets
- Admins - establish cluster wide access policies
Access credentials are represented by BucketAccess objects. The separation of BucketClaim and BucketAccess is a reflection of the usage pattern of Object Storage, where buckets are always accessed over the network, and all access is subject to authentication and authorization i.e. lifecycle of a bucket and its access are not tightly coupled.
Example: for the same bucket, one might need a BucketAccess with a "read-only" policy and another to with a "write" policy
Here are the steps for creating a BucketAccess:
The BucketAccessClass represents a set of common properties shared by multiple BucketAccesses. It is used to specify policies for creating access credentials, and also for configuring driver-specific access parameters. More information about BucketAccessClass is here
The BucketAccess is used to request access to a bucket. It contains fields for choosing the Bucket for which the credentials will be generated, and also includes a bucketAccessClassName field, which in-turn contains configuration for authorizing users to access buckets. More information about BucketAccess is here
BucketAccessClass can be used to specify a authorization mechanism. It can be one of
- KEY (default)
- IAM
The KEY based mechanism is where access and secret keys are generated to be provided to pods. IAM style is where pods are implicitly granted access to buckets by means of a metadata service. IAM style access provides greater control for the infra/cluster administrator to rotate secret tokens, revoke access, change authorizations etc., which makes it more secure.
BucketAccess - ba1 BucketAccessClass - bac1
|---------------------------------------| |----------------------------------|
| metadata: | | driverName: s3.amazonaws.com |
| namespace: ns1 | | parameters: |
| spec: | | key: value |
| bucketAccessClassName: bac1 | | authenticationType: KEY |
| bucketClaimName: bcl1 | |----------------------------------|
| credentialsSecretName: bucketcreds1 |
| protocol: s3 |
| status: |
| conditions: |
| - name: AccessGranted |
| value: False |
|---------------------------------------|
The credentialsSecretName is the name of the secret that COSI will generate containing credentials to access the bucket. The same secret name has to be set in the podSpec as well as the projected secret volumes.
In case of IAM style authentication, along with the credentialsSecretName, serviceAccountName field must also be specified. This will map the specified serviceaccount to the appropriate service account in the OSP.
BucketAccess - ba1 BucketAccessClass - bac2
|---------------------------------------| |----------------------------------|
| metadata: | | driverName: s3.amazonaws.com |
| namespace: ns1 | | parameters: |
| spec: | | key: value |
| bucketAccessClassName: bac2 | | authenticationType: IAM |
| bucketClaimName: bcl1 | |----------------------------------|
| credentialsSecretName: bucketcreds1 |
| serviceAccountName: svacc1 |
| protocol: s3 |
| status: |
| conditions: |
| - name: AccessGranted |
| value: False |
|---------------------------------------|
All but the sidecar for the specified driver will ignore the BucketClaim object. Only the appropriate sidecar will make a gRPC request to its driver to generate credentials/map service accounts.
This step will only proceed if the Bucket already exist. The Bucket's BucketReady condition should be true. Until access been granted, the AccessGranted condition in BucketAccess will be false.
COSI Driver (s3.amazonaws.com)
|------------------------------------------|
| grpc DriverGrantBucketAccess({ |
| "name": "ba-$uuid", |
| "bucketID": "br-$uuid", |
| "parameters": { |
| "key": "value" |
| } |
| }) |
|------------------------------------------|
Each BucketAccess is meant to map to a unique service account in the OSP. Once the requested privileges have been granted, a secret by the name specified in credentialsSecretName in the BucketClaim is created. The secret will reside in the namespace of the BucketClaim. The secret will contain either access keys or service account tokens based on the chosen authentication type. The format of this secret can be found here
If this call returns successfully, the sidecar sets AccessGranted condition to True in the BucketAccess.
NOTE:
- The secret will not be created until the credentials are generated/service account mappings are complete.
- Within a namespace, one BucketAccess and secret pair is generally sufficient, but cases which may want to control this more granularly can use multiple.
- The secret will be created with a finalizer that prevents it from being deleted until the associated bucketAccess is deleted.
The following stakeholders are involved in the lifecycle of attaching bucket information into pods:
- Users - specify bucket in the pod definition
Once a valid BucketAccess is available (AccessGranted is True), pods can use them.
The secret mentioned in the credentialsSecretName field of the BucketAccess should be set as a projected volume in the PodSpec. If either the Bucket provisioning is incomplete, or the access generation is incomplete - the pod will not run. It will wait indefinitely for those two conditions to be true.
PodSpec - pod1
|-------------------------------------------------|
| spec: |
| containers: |
| - volumeMounts: |
| - name: cosi-bucket1 |
| mountPath: /cosi/bucket1 |
| - name: cosi-bucket2 |
| mountPath: /cosi/bucket2 |
| volumes: |
| - name: cosi-bucket1 |
| projected: |
| sources: |
| - secret: |
| name: bucketcreds1 |
| - name: cosi-bucket2 |
| projected: |
| sources: |
| - secret: |
| name: bucketcreds2 |
|-------------------------------------------------|
If IAM style authentication was specified, then the serviceAccountName specified in BucketAccess must be specified as the serviceAccountName in the podSpec.
PodSpec - pod1
|-------------------------------------------------|
| spec: |
| serviceAccountName: svacc1 |
| containers: |
| - volumeMounts: |
| name: cosi-bucket |
| mountPath: /cosi/bucket1 |
| volumes: |
| - name: cosi-bucket |
| projected: |
| sources: |
| - secret: |
| name: bucketcreds1 |
|-------------------------------------------------|
The volume mountPath will be the directory where bucket credentials and other related information will be served.
NOTE: the contents of the files served in mountPath will be a COSI generated file containing credentials and other information required for accessing the bucket. This is NOT intended to specify a mountpoint to expose the bucket as a filesystem.
NOTE: the secret containing bucketInfo can be provided to the pod using any other {secret -> pod} provisioning mechanism, including environment variables. In case of environment variables, the secrets will be exposed to other processes in the same host, as environment variables are not inherently secure.
The above volume definition will prompt kubernetes to retrieve the secret and place it in the volumeMount path defined above. The contents of the secret will be of the format shown below:
bucket_info.json
|-----------------------------------------------|
| { |
| apiVersion: "v1alpha1", |
| kind: "BucketInfo", |
| metadata: { |
| name: "bc-$uuid" |
| }, |
| spec: { |
| bucketName: "bc-$uuid", |
| authenticationType: "KEY", |
| endpoint: "https://s3.amazonaws.com", |
| accessKeyID: "AKIAIOSFODNN7EXAMPLE", |
| accessSecretKey: "wJalrXUtnFEMI/K...", |
| region: "us-west-1", |
| protocols: [ |
| "s3" |
| ] |
| } |
| } |
|-----------------------------------------------|
In case IAM style authentication was specified, then workloadIdentityToken will be provided.
bucket_info.json
|-------------------------------------------------|
| { |
| apiVersion: "v1alpha1", |
| kind: "BucketInfo", |
| metadata: { |
| name: "bc-$uuid" |
| }, |
| spec: { |
| bucketName: "bc-$uuid", |
| authenticationType: "IAM", |
| workloadIdentityToken: "ZXlKaGJHY2...", |
| endpoint: "https://s3.amazonaws.com", |
| region: "us-west-1", |
| protocols: [ |
| "s3" |
| ] |
| } |
| } |
|-------------------------------------------------|
Workloads are expected to read the definitions in this file to access a bucket. The BucketInfo API will not be a CRD in the cluster, however, it follows the same conventions as the rest of the COSI APIs. More details can be found here
This section describes the current design for sharing buckets with other namespaces. As of the current milestone (alpha) of COSI, any bucket created in one namespace cannot be accessed in another namespace. i.e. no bucket sharing is possible. In future versions, a namespace level access control will be enforced, and buckets will be constrained to particular namespaces using selectors. Admins will be able to control which namespaces can access which buckets using namespace selectors.
The benefits of COSI can also be brought to existing buckets/ones created outside of COSI. This user story explains the steps to import a bucket:
When a Bucket object is manually created, and has its bucketID set, then COSI assumes that this Bucket has already been created.
The admin must ensure that this bucket binds only to a specific BucketClaim by specifying the BucketClaim.
Bucket - br-$uuid
|-------------------------------------------------|
| name: bucketName123 |
| spec: |
| bucketID: bucketname123 |
| bucketClaim: |
| name: bucketClaim123 |
| namespace: ns1 |
| protocols: |
| - s3 |
| parameters: |
| key: value |
| driverName: s3.amazonaws.com |
|-------------------------------------------------|
Only the specified BucketClaim will be able to bind to the Bucket.
BucketClaim - bucketClaim123
|------------------------------------------|
| name: bucketClaim123 |
| spec: |
| existingBucketName: bucketName123 |
| |
|------------------------------------------|
Similar to the BucketAccess for COSI created bucket, this BucketAccess should reference the BucketClaim that refers to this bucket.
BucketAccess - bac2
|-------------------------------|
| spec: |
| bucketClaim: bucketClaim123 |
| |
|-------------------------------|
- A Bucket created by COSI as a result of a BucketClaim can deleted by deleting the BucketClaim
- A Bucket created outside of COSI, once bound, can be deleted by deleting the BucketClaim to which it is bound
- A Bucket created outside of COSI, unless it is bound to a particular BucketClaim, cannot be deleted by users from any particular namespace. Privileged users can however delete the Bucket object at their discretion.
Once a delete has been issued to a bucket, no new BucketAccesses can be created for it. Buckets having valid BucketAccesses (Buckets in use) will not be deleted until all the BucketAccesses are cleaned up.
Buckets can be created with one of two deletion policies:
- Retain
- Delete
When the deletion policy is Retain, then the underlying bucket is not cleaned up when the Bucket object is deleted. When the deletion policy is Delete, then the underlying bucket is cleaned up when the Bucket object is deleted.
Only when all accessors (BucketAccesses) of the Bucket are deleted, is the Bucket itself cleaned up. There is a finalizer on the Bucket that prevents it from being deleted until all the accessors are done using it.
When a user deletes a BucketAccess, the corresponding secret/serviceaccount are also deleted. If a pod has that secret mounted when delete is called, then a finalizer on the secret will prevent it from being deleted. Instead, the deletionTimestamp will be set on the secret. In this way, access to a Bucket is preserved until the application pod dies.
When an admin deletes any of the class objects, it does not affect existing Buckets as fields from the class objects are copied into the Buckets during creation.
If a Bucket is manually deleted by an admin, then a finalizer on the Bucket prevents it from being deleted until the binding BucketClaim is deleted.
Self service is easily possible with the current design as both the BucketRequest and BucketClaim resources are namespace scoped, and users need not have admin privileges to create, modify and delete them.
The only admin steps are creation of class objects(BucketClass, BucketAccessClass) and Bucket imports. The creation of a class object is no different from requiring a StorageClass for provisioning PVCs. It is a well-understood pattern among Kubernetes users. Importing a Bucket requires special permissions because its lifecycle is not managed by COSI, and special care needs to be taken to prevent clones, accidental deletions and other mishaps (for instance, setting the deletion policy to Delete).
As of the current design of COSI, mutating bucket properties is not supported. However, the current design does not prevent us from supporting it in the future. Mutable properties include encryption, object lifecycle configuration, replication configuration etc. These properties will be supported in future versions along with the capability to mutate them.
These properties will be specified in the BucketRequest and follow the same pattern of events as Bucket creation. i.e. Bucket API object will be updated to reflect the properties set in BucketRequest, and then a controller will pick-up these changes and call the appropriate APIs to reflect them in the backend.
The following resources are managed by admins
- Bucket in case of brownfield buckets
- BucketClass
- BucketAccessClass
The following resources are managed by a user. These are created with a reference to their corresponding class objects:
- BucketClaim -> BucketClass
- BucketAccess -> Bucket, BucketClaim, BucketAccessClass
The COSI controller responds by creating the following objects
- BucketRequest <-> Bucket (1:1)
- BucketClaim -> Bucket
Notes:
- There are NO cycles in the relationship graph of the above mentioned API objects.
- Mutations are not supported in the API.
Resource to represent a Bucket in OSP. Buckets are cluster-scoped.
Bucket {
TypeMeta
ObjectMeta
Spec BucketSpec {
// DriverName is the name of driver associated with this bucket
DriverName string
// DeletionPolicy is used to specify how COSI should handle deletion of this
// bucket. There are 3 possible values:
// - Retain: Indicates that the bucket should not be deleted from the OSP (default)
// - Delete: Indicates that the bucket should be deleted from the OSP
// once all the workloads accessing this bucket are done
// +optional
DeletionPolicy DeletionPolicy
// Name of the BucketClass specified in the BucketRequest
BucketClassName string
// Name of the BucketClaim that resulted in the creation of this Bucket
// In case the Bucket object was created manually, then this should refer
// to the BucketClaim with which this Bucket should be bound
BucketClaim corev1.ObjectReference
// Protocols are the set of data APIs this bucket is expected to support.
// The possible values for protocol are:
// - S3: Indicates Amazon S3 protocol
// - Azure: Indicates Microsoft Azure BlobStore protocol
// - GCS: Indicates Google Cloud Storage protocol
Protocols []Protocol
// Parameters is an opaque map for passing in configuration to a driver
// for creating the bucket
// +optional
Parameters map[string]string
// ExistingBucketID is the unique id of the bucket in the OSP. This field should be
// used to specify a bucket that has been created outside of COSI.
// This field will be empty when the Bucket is dynamically provisioned by COSI.
// +optional
ExistingBucketID string
}
Status BucketStatus {
// BucketReady is a boolean condition to reflect the successful creation
// of a bucket.
BucketReady bool
// BucketID is the unique id of the bucket in the OSP. This field will be
// populated by COSI.
// +optional
BucketID string
}
}A claim to create Bucket. BucketClaim is namespace-scoped
BucketClaim {
TypeMeta
ObjectMeta
Spec BucketClaimSpec {
// Name of the BucketClass
BucketClassName string
// Protocols are the set of data API this bucket is required to support.
// The possible values for protocol are:
// - S3: Indicates Amazon S3 protocol
// - Azure: Indicates Microsoft Azure BlobStore protocol
// - GCS: Indicates Google Cloud Storage protocol
Protocols []Protocol
// Name of a bucket object that was manually
// created to import a bucket created outside of COSI
// If unspecified, then a new Bucket will be dynamically provisioned
// +optional
ExistingBucketName string
}
Status BucketClaimStatus {
// BucketReady indicates that the bucket is ready for consumpotion
// by workloads
BucketReady bool
// BucketName is the name of the provisioned Bucket in response
// to this BucketClaim. It is generated and set by the COSI controller
// before making the creation request to the OSP backend.
// +optional
BucketName string
}Resouce for configuring common properties for multiple Buckets. BucketClass is cluster-scoped.
BucketClass {
TypeMeta
ObjectMeta
// DriverName is the name of driver associated with this bucket
DriverName string
// DeletionPolicy is used to specify how COSI should handle deletion of this
// bucket. There are 3 possible values:
// - Retain: Indicates that the bucket should not be deleted from the OSP
// - Delete: Indicates that the bucket should be deleted from the OSP
// once all the workloads accessing this bucket are done
DeletionPolicy DeletionPolicy
// Parameters is an opaque map for passing in configuration to a driver
// for creating the bucket
// +optional
Parameters map[string]string
}A resource to access a Bucket. BucketAccess is namespace-scoped
BucketAccess {
TypeMeta
ObjectMeta
Spec BucketAccessSpec {
// BucketClaimName is the name of the BucketClaim.
BucketClaimName string
// Protocol is the name of the Protocol
// that this access credential is supposed to support
// If left empty, it will choose the protocol supported
// by the bucket. If the bucket supports multiple protocols,
// the end protocol is determined by the driver.
// +optional
Protocol Protocol
// BucketAccessClassName is the name of the BucketAccessClass
BucketAccessClassName string
// CredentialsSecretName is the name of the secret that COSI should populate
// with the credentials. If a secret by this name already exists, then it is
// assumed that credentials have already been generated. It is not overridden.
// This secret is deleted when the BucketAccess is delted.
CredentialsSecretName string
// ServiceAccountName is the name of the serviceAccount that COSI will map
// to the OSP service account when IAM styled authentication is specified
// +optional
ServiceAccountName string
}
Status BucketAccessStatus {
// AccessGranted indicates the successful grant of privileges to access the bucket
AccessGranted bool
// AccountID is the unique ID for the account in the OSP. It will be populated
// by the COSI sidecar once access has been successfully granted.
// +optional
AccountID string
}Resource for configuring common properties for multiple BucketClaims. BucketAccessClass is a clustered resource
BucketAccessClass {
TypeMeta
ObjectMeta
// DriverName is the name of driver associated with
// this BucketAccess
DriverName string
// AuthenticationType denotes the style of authentication
// It can be one of
// KEY - access, secret tokens based authentication
// IAM - implicit authentication of pods to the OSP based on service account mappings
AuthenticationType AuthenticationType
// Parameters is an opaque map for passing in configuration to a driver
// for granting access to a bucket
// +optional
Parameters map[string]string
}Resource mounted into pods containing information for applications to gain access to buckets.
BucketInfo {
TypeMeta
ObjectMeta
Spec BucketInfoSpec {
// BucketName is the name of the Bucket
BucketName string
// AuthenticationType denotes the style of authentication
// It can be one of
// KEY - access, secret tokens based authentication
// IAM - implicit authentication of pods to the OSP based on service account mappings
AuthenticationType AuthenticationType
// Endpoint is the URL at which the bucket can be accessed
Endpoint string
// Region is the vendor-defined region where the bucket "resides"
Region string
// Protocols are the set of data APIs this bucket is expected to support.
// The possible values for protocol are:
// - S3: Indicates Amazon S3 protocol
// - Azure: Indicates Microsoft Azure BlobStore protocol
// - GCS: Indicates Google Cloud Storage protocol
Protocols []Protocol
}
}A component that runs alongside COSI Sidecar and satisfies the COSI gRPC API specification. Sidecar and driver work together to orchestrate changes in the OSP. The driver acts as a gRPC server to the COSI Sidecar. Each COSI driver is identified by a unique id.
The sidecar uses the unique id to direct requests to the appropriate driver. Multiple instances of drivers with the same id will be added into a group and only one of them will act as the leader at any given time.
This gRPC call responds with the name of the driver. The name is used to identify the appropriate driver for a given BucketRequest or BucketClaim.
DriverGetInfo
|------------------------------------------| |---------------------------------------|
| grpc DriverGetInfoRequest{} | ===> | DriverGetInfoResponse{ |
|------------------------------------------| | "name": "s3.amazonaws.com" |
| } |
|---------------------------------------|
This gRPC call creates a bucket in the OSP, and returns information about the new bucket. This api must be idempotent. The input to this call is the name of the bucket and an opaque parameters field.
The returned bucketID should be a unique identifier for the bucket in the OSP. This value could be the name of the bucket too. This value will be used by COSI to make all subsequent calls related to this bucket.
DriverCreateBucket
|------------------------------------------| |-----------------------------------------------|
| grpc DriverCreateBucketRequest{ | ===> | DriverCreateBucketResponse{ |
| "name": "br-$uuid", | | "bucketID": "br-$uuid", |
| "parameters": { | | "bucketInfo": { |
| "key": "value" | | "s3": { |
| } | | "bucketName": "br-$uuid", |
| } | | "region": "us-west1", |
| -----------------------------------------| | "endpoint": "s3.amazonaws.com" |
| } |
| } |
| } |
|-----------------------------------------------|
This gRPC call creates a set of access credentials for a bucket. This api must be idempotent. The input to this call is the id of the bucket, a set of opaque parameters and name of the account. This accountName field is the concatenation of the characters ba (short for BucketAccess) and its UID. It is used as the idempotency key for requests to the drivers regarding a particular BA.
The returned accountID should be a unique identifier for the account in the OSP. This value could be the name of the account too. This value will be included in all subsequent calls to the driver for changes to the BucketAccess.
DriverGrantBucketAccess
|---------------------------------------------| |-----------------------------------------------|
| grpc DriverGrantBucketAccessRequest{ | ===> | DriverGrantBucketAccessResponse{ |
| "bucketID": "br-$uuid", | | "accountID": "bar-$uuid", |
| "accountName": "bar-$uuid" | | "credentials": { |
| "authenticationType": "KEY" | | "s3": { |
| "parameters": { | | "accessKeyID": "AKIAODNN7EXAMPLE", |
| "key": "value", | | "accessSecretKey": "wJaUtnFEMI/K..." |
| } | | } |
| } | | } |
|---------------------------------------------| | } |
|-----------------------------------------------|
This gRPC call deletes a bucket in the OSP.
DriverDeleteBucket
|---------------------------------------------| |-----------------------------------------------|
| grpc DriverDeleteBucketRequest{ | ===> | DriverDeleteBucketResponse{} |
| "bucketID": "br-$uuid" | |-----------------------------------------------|
| } |
|---------------------------------------------|
This gRPC call revokes access granted to a particular account.
DriverDeleteBucket
|---------------------------------------------| |-----------------------------------------------|
| grpc DriverRevokeBucketAccessRequest{ | ===> | DriverRevokeBucketAccessResponse{} |
| "bucketID": "br-$uuid", | |-----------------------------------------------|
| "accountID": "bar-$uuid" |
| } |
|---------------------------------------------|
- Unit tests will cover the functionality of the controllers.
- Unit tests will cover the new APIs.
- An end-to-end test suite will cover testing all the components together.
- Component tests will cover testing each controller in a blackbox fashion.
- Tests need to cover both correctly and incorrectly configured cases.
- API is reviewed and accepted
- Design COSI APIs to support Greenfield, Green/Brown Field, Brownfield and Static Driverless provisioning
- Design COSI APIs to support authentication using access/secret keys, and IAM.
- Evaluate gaps, update KEP and conduct reviews for all design changes
- Develop unit test cases to demonstrate that the above mentioned use cases work correctly
- Consider using a typed configuration for Bucket properties (parameter fields in Bucket, BucketClass, BucketAccess, BucketAccessClass)
- Implement all COSI components to support agreed design.
- Design and implement support for sharing buckets across namespaces.
- Design and implement quotas/restrictions for Buckets and BucketAccess.
- Basic unit and e2e tests as outlined in the test plan.
- Metrics for bucket create and delete, and granting and revoking bucket access.
- Metrics in provisioner for bucket create and delete, and granting and revoking bucket access.
- Stress tests to iron out possible race conditions in the controllers.
- Users deployed in production and have gone through at least one K8s upgrade.
This KEP has had a long journey and many revisions. Here we capture the main alternatives and the reasons why we decided on a different solution.
-
To improve workload portability user namespace resources should not reference non-deterministic generated names. If a
BucketAccessRequest(BAR) references aBucketinstance's name, and that name is pseudo random (eg. a UID added to the name) then the BAR, and hence the workload deployment, is not portable to another cluser. -
If the
Bucketinstance name is in the BAC instead of the BAR then the user is not burdened with knowledge ofBucketnames, and there is some centralized admin control over brownfield bucket access.
-
The greenfield -> brownfield workflow is very awkward with this approach. The user creates a
BucketRequest(BR) to provision a new bucket which they then want to access. The user creates a BAR pointing to a BAC which must contain the name of this newly created `Bucketinstance. Since theBucket's name is non-deterministic the admin cannot create the BAC in advance. Instead, the user must ask the admin to find the newBucketinstance and add its name to new (or maybe existing) BAC. -
App portability is still a concern but we believe that deterministic, unique
BucketandBucketAccessnames can be generated and referenced in BRs and BARs.
No changes are required on upgrade to maintain previous behaviour.
COSI is out-of-tree, so version skew strategy is N/A
- Feature gate (also fill in values in
kep.yaml)- Feature gate name:
- Components depending on the feature gate:
- Other
- Describe the mechanism: Create Deployment and DaemonSet resources (along with supporting secrets, configmaps etc.) for the three controllers that COSI requires
- Will enabling / disabling the feature require downtime of the control plane? No
- Will enabling / disabling the feature require downtime or reprovisioning of a node? No
No
Yes. Delete the resources created when installing COSI
N/A since we are only targeting alpha for this Kubernetes release
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
No
The operator can query Bucket* objects to find if their workloads are associated with them.
- Events
- Event Reason:
Bucket provisioning 'bucket-name' failed
- Event Reason:
- API .status
- Condition
PodReady=False "Error: secrets 'bucket-secret' not found") - Other field:
- Condition
- Other (treat as last resort)
- Details:
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
- Metrics
- Metric name:
- [Optional] Aggregation method:
- Components exposing the metric:
- Other (treat as last resort)
- Details:
Are there any missing metrics that would be useful to have to improve observability of this feature?
No
Existing components will not make any new API calls.
The API load of COSI components will be a factor of the number of buckets being managed and the number of bucket-accessors for these buckets. Essentially O(num-buckets * num-bucket-access). There is no per-node or per-namespace load by COSI.
Yes, the following cluster scoped resources
- Bucket
- BucketClass
- BucketAccessClass
and the following namespaced scoped resources
- BucketAccess
Not by the framework itself. Calls to external systems will be made by vendor drivers for COSI.
No
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
Yes. Containers requesting Buckets will not start until Buckets have been provisioned. This is similar to dynamic volume provisioning
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
Not likely to increase resource consumption in a significant manner
We need Linux VMs for e2e testing in CI.