Skip to content

Use Shallow Submodule Init to Reduce Clone Times #1603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed

Conversation

TShapinsky
Copy link

@TShapinsky TShapinsky commented Oct 31, 2024

Added --depth 1 to the git submodule update command in Makefile.in.

To test I setup a docker image which built the repository with ./configure --prefix=/opt/riscv --with-arch=rv32gc --with-abi=ilp32d. The image with this change was 14.77 GB where as the version without this change was 16.93 GB. That is a savings of 2.16 GB. This won't be a big deal for people developing in a stable environment where they only init the submodules once, but for cases where the build is happening in a container this is a big gain.

A different approach would be to populate the shallow property in .gitmodules. However, this approach only saves about 100MB, perhaps because the shallow property is not applied recursively.

Copy link
Collaborator

@cmuellner cmuellner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cmuellner
Copy link
Collaborator

Thanks for the PR!

Another side-effect of this is that we download less and speed up the git clone steps.

@cmuellner
Copy link
Collaborator

Looks like we need to adjust the .gitmodules file as well:

fatal: Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.

Copy link
Collaborator

@cmuellner cmuellner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CI/CD shows the following error:

fatal: Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.

@TShapinsky
Copy link
Author

CI/CD shows the following error:

fatal: Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.

Thanks. I'll see if I can reproduce that locally

@TShapinsky TShapinsky force-pushed the shallow-submodule-update branch from 20538fe to fe52c30 Compare November 1, 2024 02:12
@TShapinsky TShapinsky force-pushed the shallow-submodule-update branch from 4a64780 to 925a5c7 Compare November 1, 2024 03:09
@TShapinsky
Copy link
Author

Closing while I test so I don't generate too many actions issues

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants