Skip to content

Added required validations in lifecycle rules #8914

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Apr 14, 2025

Conversation

achouhan09
Copy link
Member

@achouhan09 achouhan09 commented Mar 27, 2025

Describe the Problem

The following validations are missing from PutBucketLifecycleConfiguration:

  • Expiration Date Must Be Midnight UTC
    • Date must be exactly 00:00:00 UTC, or AWS rejects it.
  • AbortIncompleteMultipartUpload Cannot Be Used with Size/Tags
    • AWS does not allow it with Tags or ObjectSizeGreaterThan/LessThan.
  • Expiration Rule Conflict
    • Days/Date cannot be used with ExpiredObjectDeleteMarker.
  • Multiple Filter Actions Require And
    • AWS requires multiple Filters to be inside an And block.

Explain the Changes

  1. Added validations for expiration date, AbortIncompleteMultipartUpload, expiration rule conflicts, and multiple filters requiring And.

Issues: Fixed #xxx / Gap #xxx

  1. Fix: lifecycle | PutBucketLifecycleConfiguration - missing validations #8870

Testing Instructions:

  1. run lifecycle tests: make run-single-test-postgres testname=lifecycle_index.js
  2. removed the faulty test from ceph s3 tests and created ceph s3 tests bug
  • Tests added

@achouhan09 achouhan09 requested review from nadavMiz, romayalon, a team and nbecker-cibot and removed request for a team March 27, 2025 14:42
@achouhan09 achouhan09 removed the request for review from nbecker-cibot March 27, 2025 14:46
@pull-request-size pull-request-size bot added size/M and removed size/S labels Apr 1, 2025
@pull-request-size pull-request-size bot added size/L and removed size/M labels Apr 1, 2025
@achouhan09 achouhan09 requested a review from nadavMiz April 1, 2025 11:07
@achouhan09 achouhan09 force-pushed the lifecycle-val branch 6 times, most recently from a7bd587 to 182699d Compare April 4, 2025 13:16
@achouhan09
Copy link
Member Author

s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_tags1

same test is failing for both the CI due to filter validation we have added in the PR. We have put a validation so that multiple fields in filter cannot be passed without And but in the ceph tests they are passing prefix and tags without And in filter.
@romayalon @nadavMiz What should I do here?

@romayalon
Copy link
Contributor

romayalon commented Apr 6, 2025

@achouhan09 we are S3 compatible, therefore we should do what AWS is allowing. Please run the same check on AWS and let us know if the bug is on ceph tests or on our side

s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_tags1

same test is failing for both the CI due to filter validation we have added in the PR. We have put a validation so that multiple fields in filter cannot be passed without And but in the ceph tests they are passing prefix and tags without And in filter. @romayalon @nadavMiz What should I do here?

@achouhan09
Copy link
Member Author

@achouhan09 we are S3 compatible, therefore we should do what AWS is allowing. Please run the same check on AWS and let us know if the bug is on ceph tests or on our side

s3tests_boto3/functional/test_s3.py::test_lifecycle_expiration_tags1
same test is failing for both the CI due to filter validation we have added in the PR. We have put a validation so that multiple fields in filter cannot be passed without And but in the ceph tests they are passing prefix and tags without And in filter. @romayalon @nadavMiz What should I do here?

here in ceph-s3 tests they are applying a lifecycle rule without And operator for multiple predicates (prefix and tag), which is not allowed in aws.
Please check the below screenshot, the aws command is failing when I try to apply the same rule:
Screenshot from 2025-04-07 13-09-18

@achouhan09 achouhan09 requested a review from romayalon April 7, 2025 07:47
@romayalon
Copy link
Contributor

@achouhan09 and when adding the And it works?
if the answer is yes, please open a bug to ceph if it's not already opened

@achouhan09
Copy link
Member Author

@achouhan09 and when adding the And it works? if the answer is yes, please open a bug to ceph if it's not already opened

@romayalon yes its working with And (reference aws doc), I have opened a new ceph-s3 tests issue here.

I cannot merge this PR without the ceph-s3 test fixes or we need to make temporary changes in our code and remove that test. wdyt?

…on and AboutIncompleteMultipartUpload

2. Added tests for the above validations

Signed-off-by: Aayush Chouhan <[email protected]>
@achouhan09 achouhan09 merged commit bbebeac into noobaa:master Apr 14, 2025
11 checks passed
@achouhan09 achouhan09 deleted the lifecycle-val branch April 14, 2025 16:37
@romayalon romayalon mentioned this pull request May 15, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

lifecycle | PutBucketLifecycleConfiguration - missing validations
4 participants