Skip to content

Commit 0a0e4ad

Browse files
kaizencckaizencc
authored
feat(cli): garbage collect s3 assets (under --unstable flag) (#31611)
## S3 Asset Garbage Collection This PR introduces a new CLI command under the new `--unstable` flag. This flag ensures that users understand and opt-in to experimental or incomplete CLI features. `cdk gc` will garbage collect unused assets in your bootstrapped S3 bucket. It goes through each object in the bucket, checks to see if the asset hash shows up in a cloudformation stack, and if not, tags the object as unused and/or deletes the object (depending on your configuration). ## **THIS COMMAND WILL DELETE OBJECTS IN YOUR BOOTSTRAPPED S3 BUCKET** basic garbage collection (immediately deletes objects that are unused): ```bash cdk gc aws://0123456789012/us-east-1 \ --unstable='gc' \ --type='s3' ``` garbage collection with a buffer (deletes unused objects > # of days specified): ```bash cdk gc aws://0123456789012/us-east-1 \ --unstable='gc' \ --type='s3' \ --rollback-buffer-days=30 ``` garbage collection with a created at buffer (deletes unused objects only if they have lived longer than this many days): ```bash cdk gc aws://0123456789012/us-east-1 \ --unstable='gc' \ --type='s3' \ --created-buffer-days=5 ``` garbage collect a specific bootstrap stack: ```bash cdk gc aws://0123456789012/us-east-1 \ --unstable='gc' \ --type='s3' \ --bootstrap-stack-name=cdktest-0lc2i3vebi7-bootstrap-stack ``` before actually deleting your assets, you will be prompted one last time: ```bash Found 1 objects to delete based off of the following criteria: - objects have been isolated for > 0 days - objects were created > 0 days ago Delete this batch (yes/no/delete-all)? ``` To disable this, specify the `--skip-delete-prompt` option. ## Todo in another PR - [ ] ECR asset collection ---- *By submitting this pull request, I confirm that my contribution is made under the terms of the Apache-2.0 license*
1 parent be4154b commit 0a0e4ad

File tree

11 files changed

+1940
-17
lines changed

11 files changed

+1940
-17
lines changed

‎packages/@aws-cdk-testing/cli-integ/lib/with-cdk-app.ts

+45
Original file line numberDiff line numberDiff line change
@@ -331,6 +331,30 @@ export interface CdkModernBootstrapCommandOptions extends CommonCdkBootstrapComm
331331
readonly usePreviousParameters?: boolean;
332332
}
333333

334+
export interface CdkGarbageCollectionCommandOptions {
335+
/**
336+
* The amount of days an asset should stay isolated before deletion, to
337+
* guard against some pipeline rollback scenarios
338+
*
339+
* @default 0
340+
*/
341+
readonly rollbackBufferDays?: number;
342+
343+
/**
344+
* The type of asset that is getting garbage collected.
345+
*
346+
* @default 'all'
347+
*/
348+
readonly type?: 'ecr' | 's3' | 'all';
349+
350+
/**
351+
* The name of the bootstrap stack
352+
*
353+
* @default 'CdkToolkit'
354+
*/
355+
readonly bootstrapStackName?: string;
356+
}
357+
334358
export class TestFixture extends ShellHelper {
335359
public readonly qualifier = this.randomString.slice(0, 10);
336360
private readonly bucketsToDelete = new Array<string>();
@@ -464,6 +488,26 @@ export class TestFixture extends ShellHelper {
464488
});
465489
}
466490

491+
public async cdkGarbageCollect(options: CdkGarbageCollectionCommandOptions): Promise<string> {
492+
const args = [
493+
'gc',
494+
'--unstable=gc', // TODO: remove when stabilizing
495+
'--confirm=false',
496+
'--created-buffer-days=0', // Otherwise all assets created during integ tests are too young
497+
];
498+
if (options.rollbackBufferDays) {
499+
args.push('--rollback-buffer-days', String(options.rollbackBufferDays));
500+
}
501+
if (options.type) {
502+
args.push('--type', options.type);
503+
}
504+
if (options.bootstrapStackName) {
505+
args.push('--bootstrapStackName', options.bootstrapStackName);
506+
}
507+
508+
return this.cdk(args);
509+
}
510+
467511
public async cdkMigrate(language: string, stackName: string, inputPath?: string, options?: CdkCliOptions) {
468512
return this.cdk([
469513
'migrate',
@@ -634,6 +678,7 @@ async function ensureBootstrapped(fixture: TestFixture) {
634678
CDK_NEW_BOOTSTRAP: '1',
635679
},
636680
});
681+
637682
ALREADY_BOOTSTRAPPED_IN_THIS_RUN.add(envSpecifier);
638683
}
639684

‎packages/@aws-cdk-testing/cli-integ/tests/cli-integ-tests/garbage-collection.integtest.ts

+202
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,202 @@
1+
import { GetObjectTaggingCommand, ListObjectsV2Command, PutObjectTaggingCommand } from '@aws-sdk/client-s3';
2+
import { integTest, randomString, withoutBootstrap } from '../../lib';
3+
4+
jest.setTimeout(2 * 60 * 60_000); // Includes the time to acquire locks, worst-case single-threaded runtime
5+
6+
integTest(
7+
'Garbage Collection deletes unused assets',
8+
withoutBootstrap(async (fixture) => {
9+
const toolkitStackName = fixture.bootstrapStackName;
10+
const bootstrapBucketName = `aws-cdk-garbage-collect-integ-test-bckt-${randomString()}`;
11+
fixture.rememberToDeleteBucket(bootstrapBucketName); // just in case
12+
13+
await fixture.cdkBootstrapModern({
14+
toolkitStackName,
15+
bootstrapBucketName,
16+
});
17+
18+
await fixture.cdkDeploy('lambda', {
19+
options: [
20+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
21+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
22+
'--toolkit-stack-name', toolkitStackName,
23+
'--force',
24+
],
25+
});
26+
fixture.log('Setup complete!');
27+
28+
await fixture.cdkDestroy('lambda', {
29+
options: [
30+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
31+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
32+
'--toolkit-stack-name', toolkitStackName,
33+
'--force',
34+
],
35+
});
36+
37+
await fixture.cdkGarbageCollect({
38+
rollbackBufferDays: 0,
39+
type: 's3',
40+
bootstrapStackName: toolkitStackName,
41+
});
42+
fixture.log('Garbage collection complete!');
43+
44+
// assert that the bootstrap bucket is empty
45+
await fixture.aws.s3.send(new ListObjectsV2Command({ Bucket: bootstrapBucketName }))
46+
.then((result) => {
47+
expect(result.Contents).toBeUndefined();
48+
});
49+
}),
50+
);
51+
52+
integTest(
53+
'Garbage Collection keeps in use assets',
54+
withoutBootstrap(async (fixture) => {
55+
const toolkitStackName = fixture.bootstrapStackName;
56+
const bootstrapBucketName = `aws-cdk-garbage-collect-integ-test-bckt-${randomString()}`;
57+
fixture.rememberToDeleteBucket(bootstrapBucketName); // just in case
58+
59+
await fixture.cdkBootstrapModern({
60+
toolkitStackName,
61+
bootstrapBucketName,
62+
});
63+
64+
await fixture.cdkDeploy('lambda', {
65+
options: [
66+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
67+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
68+
'--toolkit-stack-name', toolkitStackName,
69+
'--force',
70+
],
71+
});
72+
fixture.log('Setup complete!');
73+
74+
await fixture.cdkGarbageCollect({
75+
rollbackBufferDays: 0,
76+
type: 's3',
77+
bootstrapStackName: toolkitStackName,
78+
});
79+
fixture.log('Garbage collection complete!');
80+
81+
// assert that the bootstrap bucket has the object
82+
await fixture.aws.s3.send(new ListObjectsV2Command({ Bucket: bootstrapBucketName }))
83+
.then((result) => {
84+
expect(result.Contents).toHaveLength(1);
85+
});
86+
87+
await fixture.cdkDestroy('lambda', {
88+
options: [
89+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
90+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
91+
'--toolkit-stack-name', toolkitStackName,
92+
'--force',
93+
],
94+
});
95+
fixture.log('Teardown complete!');
96+
}),
97+
);
98+
99+
integTest(
100+
'Garbage Collection tags unused assets',
101+
withoutBootstrap(async (fixture) => {
102+
const toolkitStackName = fixture.bootstrapStackName;
103+
const bootstrapBucketName = `aws-cdk-garbage-collect-integ-test-bckt-${randomString()}`;
104+
fixture.rememberToDeleteBucket(bootstrapBucketName); // just in case
105+
106+
await fixture.cdkBootstrapModern({
107+
toolkitStackName,
108+
bootstrapBucketName,
109+
});
110+
111+
await fixture.cdkDeploy('lambda', {
112+
options: [
113+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
114+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
115+
'--toolkit-stack-name', toolkitStackName,
116+
'--force',
117+
],
118+
});
119+
fixture.log('Setup complete!');
120+
121+
await fixture.cdkDestroy('lambda', {
122+
options: [
123+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
124+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
125+
'--toolkit-stack-name', toolkitStackName,
126+
'--force',
127+
],
128+
});
129+
130+
await fixture.cdkGarbageCollect({
131+
rollbackBufferDays: 100, // this will ensure that we do not delete assets immediately (and just tag them)
132+
type: 's3',
133+
bootstrapStackName: toolkitStackName,
134+
});
135+
fixture.log('Garbage collection complete!');
136+
137+
// assert that the bootstrap bucket has the object and is tagged
138+
await fixture.aws.s3.send(new ListObjectsV2Command({ Bucket: bootstrapBucketName }))
139+
.then(async (result) => {
140+
expect(result.Contents).toHaveLength(2); // also the CFN template
141+
const key = result.Contents![0].Key;
142+
const tags = await fixture.aws.s3.send(new GetObjectTaggingCommand({ Bucket: bootstrapBucketName, Key: key }));
143+
expect(tags.TagSet).toHaveLength(1);
144+
});
145+
}),
146+
);
147+
148+
integTest(
149+
'Garbage Collection untags in-use assets',
150+
withoutBootstrap(async (fixture) => {
151+
const toolkitStackName = fixture.bootstrapStackName;
152+
const bootstrapBucketName = `aws-cdk-garbage-collect-integ-test-bckt-${randomString()}`;
153+
fixture.rememberToDeleteBucket(bootstrapBucketName); // just in case
154+
155+
await fixture.cdkBootstrapModern({
156+
toolkitStackName,
157+
bootstrapBucketName,
158+
});
159+
160+
await fixture.cdkDeploy('lambda', {
161+
options: [
162+
'--context', `bootstrapBucket=${bootstrapBucketName}`,
163+
'--context', `@aws-cdk/core:bootstrapQualifier=${fixture.qualifier}`,
164+
'--toolkit-stack-name', toolkitStackName,
165+
'--force',
166+
],
167+
});
168+
fixture.log('Setup complete!');
169+
170+
// Artificially add tagging to the asset in the bootstrap bucket
171+
const result = await fixture.aws.s3.send(new ListObjectsV2Command({ Bucket: bootstrapBucketName }));
172+
const key = result.Contents!.filter((c) => c.Key?.split('.')[1] == 'zip')[0].Key; // fancy footwork to make sure we have the asset key
173+
await fixture.aws.s3.send(new PutObjectTaggingCommand({
174+
Bucket: bootstrapBucketName,
175+
Key: key,
176+
Tagging: {
177+
TagSet: [{
178+
Key: 'aws-cdk:isolated',
179+
Value: '12345',
180+
}, {
181+
Key: 'bogus',
182+
Value: 'val',
183+
}],
184+
},
185+
}));
186+
187+
await fixture.cdkGarbageCollect({
188+
rollbackBufferDays: 100, // this will ensure that we do not delete assets immediately (and just tag them)
189+
type: 's3',
190+
bootstrapStackName: toolkitStackName,
191+
});
192+
fixture.log('Garbage collection complete!');
193+
194+
// assert that the isolated object tag is removed while the other tag remains
195+
const newTags = await fixture.aws.s3.send(new GetObjectTaggingCommand({ Bucket: bootstrapBucketName, Key: key }));
196+
197+
expect(newTags.TagSet).toEqual([{
198+
Key: 'bogus',
199+
Value: 'val',
200+
}]);
201+
}),
202+
);

‎packages/aws-cdk/README.md

+73
Original file line numberDiff line numberDiff line change
@@ -25,6 +25,7 @@ The AWS CDK Toolkit provides the `cdk` command-line interface that can be used t
2525
| [`cdk watch`](#cdk-watch) | Watches a CDK app for deployable and hotswappable changes |
2626
| [`cdk destroy`](#cdk-destroy) | Deletes a stack from an AWS account |
2727
| [`cdk bootstrap`](#cdk-bootstrap) | Deploy a toolkit stack to support deploying large stacks & artifacts |
28+
| [`cdk gc`](#cdk-gc) | Garbage collect assets associated with the bootstrapped stack |
2829
| [`cdk doctor`](#cdk-doctor) | Inspect the environment and produce information useful for troubleshooting |
2930
| [`cdk acknowledge`](#cdk-acknowledge) | Acknowledge (and hide) a notice by issue number |
3031
| [`cdk notices`](#cdk-notices) | List all relevant notices for the application |
@@ -876,6 +877,78 @@ In order to remove that permissions boundary you have to specify the
876877
cdk bootstrap --no-previous-parameters
877878
```
878879

880+
### `cdk gc`
881+
882+
CDK Garbage Collection.
883+
884+
> [!CAUTION]
885+
> CDK Garbage Collection is under development and therefore must be opted in via the `--unstable` flag: `cdk gc --unstable=gc`.
886+
>
887+
> [!WARNING]
888+
> `cdk gc` currently only supports garbage collecting S3 Assets. You must specify `cdk gc --unstable=gc --type=s3` as ECR asset garbage collection has not yet been implemented.
889+
890+
`cdk gc` garbage collects unused S3 assets from your bootstrap bucket via the following mechanism:
891+
892+
- for each object in the bootstrap S3 Bucket, check to see if it is referenced in any existing CloudFormation templates
893+
- if not, it is treated as unused and gc will either tag it or delete it, depending on your configuration.
894+
895+
The most basic usage looks like this:
896+
897+
```console
898+
cdk gc --unstable=gc --type=s3
899+
```
900+
901+
This will garbage collect S3 assets from the current bootstrapped environment(s) and immediately delete them. Note that, since the default bootstrap S3 Bucket is versioned, object deletion will be handled by the lifecycle
902+
policy on the bucket.
903+
904+
Before we begin to delete your assets, you will be prompted:
905+
906+
```console
907+
cdk gc --unstable=gc --type=s3
908+
909+
Found X objects to delete based off of the following criteria:
910+
- objects have been isolated for > 0 days
911+
- objects were created > 1 days ago
912+
913+
Delete this batch (yes/no/delete-all)?
914+
```
915+
916+
Since it's quite possible that the bootstrap bucket has many objects, we work in batches of 1000 objects. To skip the
917+
prompt either reply with `delete-all`, or use the `--confirm=false` option.
918+
919+
```console
920+
cdk gc --unstable=gc --type=s3 --confirm=false
921+
```
922+
923+
If you are concerned about deleting assets too aggressively, there are multiple levers you can configure:
924+
925+
- rollback-buffer-days: this is the amount of days an asset has to be marked as isolated before it is elligible for deletion.
926+
- created-buffer-days: this is the amount of days an asset must live before it is elligible for deletion.
927+
928+
When using `rollback-buffer-days`, instead of deleting unused objects, `cdk gc` will tag them with
929+
today's date instead. It will also check if any objects have been tagged by previous runs of `cdk gc`
930+
and delete them if they have been tagged for longer than the buffer days.
931+
932+
When using `created-buffer-days`, we simply filter out any assets that have not persisted that number
933+
of days.
934+
935+
```console
936+
cdk gc --unstable=gc --type=s3 --rollback-buffer-days=30 --created-buffer-days=1
937+
```
938+
939+
You can also configure the scope that `cdk gc` performs via the `--action` option. By default, all actions
940+
are performed, but you can specify `print`, `tag`, or `delete-tagged`.
941+
942+
- `print` performs no changes to your AWS account, but finds and prints the number of unused assets.
943+
- `tag` tags any newly unused assets, but does not delete any unused assets.
944+
- `delete-tagged` deletes assets that have been tagged for longer than the buffer days, but does not tag newly unused assets.
945+
946+
```console
947+
cdk gc --unstable=gc --type=s3 --action=delete-tagged --rollback-buffer-days=30
948+
```
949+
950+
This will delete assets that have been unused for >30 days, but will not tag additional assets.
951+
879952
### `cdk doctor`
880953

881954
Inspect the current command-line environment and configurations, and collect information that can be useful for

0 commit comments

Comments
 (0)