Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

USHIFT-5461: Implement S3 caching for container images in layer1 #4718

Open
wants to merge 5 commits into
base: main
Choose a base branch
from

Conversation

ggiguash
Copy link
Contributor

@ggiguash ggiguash commented Mar 21, 2025

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

The time saving seems to be around 5m for the image build step as that's the time it takes to build layer1 when ISO images are skipped.

Notes:

  • The times are approximate and they may vary slightly.
  • To focus on the effect of the current change, only the infra-iso-build times are presented
  • The PR test history can be viewed here.
  • Build step is the "same" for pre-submit and periodic tests
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 42m 37m
Presubmit arm 35m 27m
Periodic x86 42m 37m
Periodic arm 35m 27m

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 21, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Mar 21, 2025
@openshift-ci openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 21, 2025
Copy link
Contributor

openshift-ci bot commented Mar 21, 2025

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copy link
Contributor

openshift-ci bot commented Mar 21, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: ggiguash

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 21, 2025
@ggiguash
Copy link
Contributor Author

/test ?

Copy link
Contributor

openshift-ci bot commented Mar 21, 2025

@ggiguash: The following commands are available to trigger required jobs:

/test e2e-aws-ai-model-serving
/test e2e-aws-footprint-and-performance
/test e2e-aws-tests
/test e2e-aws-tests-arm
/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm
/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm
/test e2e-aws-tests-periodic
/test e2e-aws-tests-periodic-arm
/test images
/test ocp-full-conformance-rhel-eus
/test ocp-full-conformance-serial-rhel-eus
/test test-rpm
/test test-unit
/test verify

The following commands are available to trigger optional jobs:

/test security
/test test-rebase

Use /test all to run the following jobs that were automatically triggered:

pull-ci-openshift-microshift-main-e2e-aws-tests
pull-ci-openshift-microshift-main-e2e-aws-tests-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-periodic
pull-ci-openshift-microshift-main-e2e-aws-tests-bootc-periodic-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-cache
pull-ci-openshift-microshift-main-e2e-aws-tests-cache-arm
pull-ci-openshift-microshift-main-e2e-aws-tests-periodic
pull-ci-openshift-microshift-main-e2e-aws-tests-periodic-arm
pull-ci-openshift-microshift-main-images
pull-ci-openshift-microshift-main-security
pull-ci-openshift-microshift-main-test-unit
pull-ci-openshift-microshift-main-verify

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

1 similar comment
@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Note: the times are approximate and they may vary slightly

Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 xxx
Presubmit arm
Periodic x86 2:50h xxx
Periodic arm 2:50h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Note: the times are approximate and they may vary slightly.
The PR test history can be viewed here.

Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 xxx
Presubmit arm
Periodic x86 2:50h xxx
Periodic arm 2:50h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Note: the times are approximate and they may vary slightly.
The PR test history can be viewed here.

Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 1:30h xxx
Presubmit arm 1:25h
Periodic x86 2:50h xxx
Periodic arm 2:50h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Note: the times are approximate and they may vary slightly.
The PR test history can be viewed here.

Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 1:30h xxx
Presubmit arm 1:25h
Periodic x86 1:30h xxx
Periodic arm 1:25h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Notes:

  • The times are approximate and they may vary slightly.
  • The PR test history can be viewed here.
  • The periodic jobs that matter for this change are the ones that run as part of pre-submits, not nightly.
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 1:30h xxx
Presubmit arm 1:25h
Periodic x86 1:30h xxx
Periodic arm 1:25h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Time savings TBD and updated in the description

Notes:

  • The times are approximate and they may vary slightly.
  • The PR test history can be viewed here.
  • The periodic jobs that matter the most for this change are the ones that run as part of pre-submits, not nightly.
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 1:30h xxx
Presubmit arm 1:25h
Periodic x86 1:30h xxx
Periodic arm 1:25h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 22, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

TODO: Potential time savings to be updated in the description

Notes:

  • The times are approximate and they may vary slightly.
  • The PR test history can be viewed here.
  • The periodic jobs that matter the most for this change are the ones that run as part of pre-submits, not nightly.
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 1:30h xxx
Presubmit arm 1:25h
Periodic x86 1:30h xxx
Periodic arm 1:25h

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@ggiguash ggiguash force-pushed the registry-cache-over-vpn branch from a50ff29 to 27c8efe Compare March 22, 2025 10:27
@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

3 similar comments
@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 24, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

Notes:

  • The times are approximate and they may vary slightly.
  • To focus on the effect of the current change, only the infra-iso-build times are presented
  • The PR test history can be viewed here.
  • Build step is the "same" for pre-submit and periodic tests
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 40m 37m
Presubmit arm 35m 27m
Periodic x86 40m 37m
Periodic arm 35m 27m

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 24, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

Notes:

  • The times are approximate and they may vary slightly.
  • To focus on the effect of the current change, only the infra-iso-build times are presented
  • The PR test history can be viewed here.
  • Build step is the "same" for pre-submit and periodic tests
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 42m 37m
Presubmit arm 35m 27m
Periodic x86 42m 37m
Periodic arm 35m 27m

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

openshift-ci-robot commented Mar 24, 2025

@ggiguash: This pull request references USHIFT-5461 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.19.0" version, but no target version was set.

In response to this:

Attempt copying container images from the local mirror registry to container storage and skip the builds if the copy succeeds. The --force-rebuild flag overrides this behavior by always rebuilding the images.

The time saving seems to be around 5m for the image build step as that's the time it takes to build layer1 when ISO images are skipped.

Notes:

  • The times are approximate and they may vary slightly.
  • To focus on the effect of the current change, only the infra-iso-build times are presented
  • The PR test history can be viewed here.
  • Build step is the "same" for pre-submit and periodic tests
Job Type Duration Before the Change Duration After the Change
Cache x86 1:35h 1:35h
Cache arm 1:30h 1:30h
Presubmit x86 42m 37m
Presubmit arm 35m 27m
Periodic x86 42m 37m
Periodic arm 35m 27m

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ggiguash ggiguash force-pushed the registry-cache-over-vpn branch from eabc874 to 216c2c1 Compare March 24, 2025 10:05
@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-bootc
/test e2e-aws-tests-bootc-arm
/test e2e-aws-tests-bootc-periodic
/test e2e-aws-tests-bootc-periodic-arm

@ggiguash ggiguash marked this pull request as ready for review March 24, 2025 18:10
@openshift-ci openshift-ci bot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 24, 2025
@openshift-ci openshift-ci bot requested review from agullon and vanhalenar March 24, 2025 18:11
@ggiguash
Copy link
Contributor Author

/test e2e-aws-tests-cache
/test e2e-aws-tests-cache-arm

Copy link
Contributor

openshift-ci bot commented Mar 25, 2025

@ggiguash: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Comment on lines +82 to +85
location = "localhost"
[[registry.mirror]]
location = "${MIRROR_REGISTRY_URL}"
insecure = true
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this mean that for localhost/IMAGE it will always attempt to first use mirror instead of local storage?

# See https://docs.projectquay.io/deploy_quay.html#poc-configuring-database
if [ ! -d "${MIRROR_REGISTRY_DIR}/postgres" ] ; then
mkdir -p "${MIRROR_REGISTRY_DIR}/postgres"
setfacl -m u:26:-wx "${MIRROR_REGISTRY_DIR}/postgres"
# setfacl -m u:26:-wx "${MIRROR_REGISTRY_DIR}/postgres"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we remove it? It can be always pulled from the history if needed.

setfacl -m u:1001:-wx "${MIRROR_REGISTRY_DIR}/storage"
if [ ! -d "${QUAY_STORAGE_DIR}" ] ; then
mkdir -p "${QUAY_STORAGE_DIR}"
setfacl -m u:1001:-wx "${QUAY_STORAGE_DIR}"
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we run quay container as "$(id -u):$(id -g)"?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants