Skip to content

git submodule update fails #1669

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
nfraly opened this issue Feb 13, 2025 · 17 comments
Closed

git submodule update fails #1669

nfraly opened this issue Feb 13, 2025 · 17 comments

Comments

@nfraly
Copy link

nfraly commented Feb 13, 2025

This might be a re-opening of an old issue, or it may be new, not sure. Failing to grab submodules during make, and submodule update is failing as well. I have a local copy from last night that worked to use, but just wanted to bring this issue forward so it can get some attention (first time posting an issue to a big repo like this, sorry for any faux pas).

Edit to add: GCC and binutils have both thrown errors, but the script from issue 1654 shows only spike and pk are not advertised

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Feb 13, 2025

See here:

This works fine for me so try not doing any recursive clone manually:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669
cd issue-1669
./configure ...
make 

Edit: oh - that seemed to work for me yesterday but not today:

make
cd /home/user/issue-1666 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1666/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/user/issue-1666/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/user/issue-1666/gcc'...
remote: Enumerating objects: 146147, done.
remote: Counting objects: 100% (146147/146147), done.
remote: Compressing objects: 100% (112411/112411), done.
remote: Total 146147 (delta 37701), reused 91058 (delta 31710), pack-reused 0
Receiving objects: 100% (146147/146147), 175.47 MiB | 9.37 MiB/s, done.
Resolving deltas: 100% (37701/37701), done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
fatal: Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/user/issue-1666/gcc/.git] Error 128

For the moment you may need to disable the shallow cloning:

  1. Edit .gitmodules to comment out/remove all shallow = true lines
  2. Edit Makefile.in to remove depth 1
  3. Run configure and make

@nfraly
Copy link
Author

nfraly commented Feb 13, 2025

Disabling the shallow cloning appears to have worked. Thank you!!

@jleidel
Copy link

jleidel commented Feb 13, 2025

Also confirming that this persists. Following @TommyMurphyTM1234's instructions from above does work temporarily.

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Feb 13, 2025

Thanks for the updates. As I mentioned, @mickflemm explains the issue here

but, unfortunately, at the moment I don't really know what specific changes are needed in order to prepare a PR that fixes the problem.

However @mickflemm did say that he hopes to be able to have a look again later this month.

I presume that the CI builds will also fail next time that they run to create a release...

@hpretl
Copy link

hpretl commented Feb 13, 2025

Here the sed's for the above WA:

sed -i '/shallow = true/d' .gitmodules
sed -i 's/--depth 1//g' Makefile.in

@TVS2004Chaitanya
Copy link

See here:

This works fine for me so try not doing any recursive clone manually:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669
cd issue-1669
./configure ...
make 

Edit: oh - that seemed to work for me yesterday but not today:

make
cd /home/user/issue-1666 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1666/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/user/issue-1666/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/user/issue-1666/gcc'...
remote: Enumerating objects: 146147, done.
remote: Counting objects: 100% (146147/146147), done.
remote: Compressing objects: 100% (112411/112411), done.
remote: Total 146147 (delta 37701), reused 91058 (delta 31710), pack-reused 0
Receiving objects: 100% (146147/146147), 175.47 MiB | 9.37 MiB/s, done.
Resolving deltas: 100% (37701/37701), done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
fatal: Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/user/issue-1666/gcc/.git] Error 128

For the moment you may need to disable the shallow cloning:

  1. Edit .gitmodules to comment out/remove all shallow = true lines
  2. Edit Makefile.in to remove depth 1
  3. Run configure and make

But I did not get any .gitmodule file when I cloned the github repository, should I just make the changes in the Makefile.in to overcome the failing of submodules?

@TommyMurphyTM1234
Copy link
Collaborator

But I did not get any .gitmodule file when I cloned the github repository, should I just make the changes in the Makefile.in to overcome the failing of submodules?

No. You need to ensure that the .gitmodules file has been fetched by the clone operation and then edit that and the Makefile.in before running configure. This post outlines how to use sed to make the necessary changes to circumvent the shallow clone problem for now:

Note that files whose names start with a dot/period/'.' are often not visible by default to perhaps it's there but you're just not seeing it? Try ls -a in a terminal shell to see if it's actually there. If it's not then you'll have to try cloning afresh.

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Feb 24, 2025

To clarify...

How it fails at the moment due to shallow clone issues:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669
Cloning into 'issue-1669'...
remote: Enumerating objects: 9537, done.
remote: Counting objects: 100% (430/430), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 9537 (delta 369), reused 324 (delta 324), pack-reused 9107 (from 3)
Receiving objects: 100% (9537/9537), 6.50 MiB | 7.22 MiB/s, done.
Resolving deltas: 100% (4753/4753), done.

cd issue-1669

./configure --prefix=`pwd`/installed-tools
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/bash
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... no
configure: creating ./config.status
mconfig.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed

make
cd /home/user/issue-1669 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1669/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/user/issue-1669/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/user/issue-1669/gcc'...
remote: Enumerating objects: 146240, done.
remote: Counting objects: 100% (146240/146240), done.
remote: Compressing objects: 100% (112584/112584), done.
Receiving objects: 100% (146240/146240), 175.63 MiB | 1.16 MiB/s, done.
remote: Total 146240 (delta 37682), reused 90996 (delta 31629), pack-reused 0
Resolving deltas: 100% (37682/37682), done.
remote: Enumerating objects: 18, done.
remote: Counting objects: 100% (18/18), done.
remote: Compressing objects: 100% (10/10), done.
remote: Total 10 (delta 8), reused 0 (delta 0), pack-reused 0
Unpacking objects: 100% (10/10), 1.40 KiB | 62.00 KiB/s, done.
From https://gcc.gnu.org/git/gcc
 + 580627961...f3d4208e7 master     -> origin/master  (forced update)
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
fatal: Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/user/issue-1669/gcc/.git] Error 128

Try again but this time apply the workaround:

cd ..
rm -rf issue-1669

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669
Cloning into 'issue-1669'...
remote: Enumerating objects: 9537, done.
remote: Counting objects: 100% (428/428), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 9537 (delta 367), reused 322 (delta 322), pack-reused 9109 (from 3)
Receiving objects: 100% (9537/9537), 6.52 MiB | 8.35 MiB/s, done.
Resolving deltas: 100% (4747/4747), done.

cd issue-1669

ls -a
.   .git     .gitignore   LICENSE      README.md  configure     contrib  gcc  glibc          llvm  newlib  qemu        scripts  test
..  .github  .gitmodules  Makefile.in  binutils   configure.ac  dejagnu  gdb  linux-headers  musl  pk      regression  spike    uclibc-ng

sed -i '/shallow = true/d' .gitmodules
sed -i 's/--depth 1//g' Makefile.in

./configure --prefix=`pwd`/installed-tools
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables...
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/bash
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... no
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed

make
cd /home/user/issue-1669 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1669/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress  /home/user/issue-1669/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/user/issue-1669/gcc'...
remote: Enumerating objects: 250386, done.
remote: Counting objects: 100% (250386/250386), done.
remote: Compressing objects: 100% (13844/13844), done.
Receiving objects: 100% (3170552/3170552), 1.20 GiB | 8.73 MiB/s, done.
remote: Total 3170552 (delta 246349), reused 236620 (delta 236530), pack-reused 2920166
Resolving deltas: 100% (2613919/2613919), done.
Submodule path 'gcc': checked out '04696df09633baf97cdbbdd6e9929b9d472161d3'
cd /home/user/issue-1669 && \
flock `git rev-parse --git-dir`/config git submodule init /home/user/issue-1669/newlib/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress  /home/user/issue-1669/newlib/
Submodule 'newlib' (https://sourceware.org/git/newlib-cygwin.git) registered for path 'newlib'
Cloning into '/home/user/issue-1669/newlib'...
remote: Enumerating objects: 2628, done.
remote: Counting objects: 100% (2628/2628), done.
remote: Compressing objects: 100% (2548/2548), done.
remote: Total 218854 (delta 1941), reused 92 (delta 72), pack-reused 216226
Receiving objects: 100% (218854/218854), 130.57 MiB | 8.52 MiB/s, done.
Resolving deltas: 100% (176216/176216), done.
Submodule path 'newlib': checked out '26f7004bf73c421c3fd5e5a6ccf470d05337b435'
...

@aWZHY0yQH81uOYvH
Copy link

aWZHY0yQH81uOYvH commented Feb 25, 2025

The workaround (removing shallow = true and --depth 1) does not work for me. Even a standard git submodule update --init fails with

error: Server does not allow request for unadvertised object beb2cdbcda911764b2bed5e57921fe90493260bd
fatal: Fetched in submodule path 'binutils', but it did not contain beb2cdbcda911764b2bed5e57921fe90493260bd. Direct fetching of that commit failed.

@Awab-Ahsan
Copy link

Awab-Ahsan commented Feb 25, 2025

I am trying to setup RISCV GCC toolchain on redhat. I have succesffuly set up it in ubuntu along with spike and proxyt kernel. But now with redhat I am facing issues with respect to cloning of the git.
I have followed both traditional way and the ways mentioned above both are not working for me.
Traditional

git clone https://github.com/riscv/riscv-gnu-toolchain
Cloning into 'riscv-gnu-toolchain'...
remote: Enumerating objects: 9537, done.
remote: Counting objects: 100% (428/428), done.
remote: Compressing objects: 100% (106/106), done.
remote: Total 9537 (delta 367), reused 322 (delta 322), pack-reused 9109 (from 3)
Receiving objects: 100% (9537/9537), 6.52 MiB | 1.10 MiB/s, done.
Resolving deltas: 100% (4747/4747), done.

../configure --prefix=$RISC_V --with-arch=rv32gv --with-abi=ilp32
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/sh
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... /usr/bin/ftp
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed

make
cd /home/advancedresearch/Downloads/riscv-gnu-toolchain/build/.. && \
flock `git rev-parse --git-dir`/config git submodule init /home/advancedresearch/Downloads/riscv-gnu-toolchain/build/../gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/advancedresearch/Downloads/riscv-gnu-toolchain/build/../gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/advancedresearch/Downloads/riscv-gnu-toolchain/gcc'...
remote: Enumerating objects: 146242, done.        
remote: Counting objects: 100% (146242/146242), done.        
remote: Compressing objects: 100% (112586/112586), done.        
remote: Total 146242 (delta 37682), reused 90993 (delta 31629), pack-reused 0        
Receiving objects: 100% (146242/146242), 175.63 MiB | 808.00 KiB/s, done.
Resolving deltas: 100% (37682/37682), done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
fatal: Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/advancedresearch/Downloads/riscv-gnu-toolchain/build/../gcc/.git] Error 128

Now if i follow the way told above it still fails

[advancedresearch@advancedresearch Downloads]$ git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669
Cloning into 'issue-1669'...
remote: Enumerating objects: 9537, done.
remote: Counting objects: 100% (427/427), done.
remote: Compressing objects: 100% (105/105), done.
remote: Total 9537 (delta 366), reused 322 (delta 322), pack-reused 9110 (from 3)
Receiving objects: 100% (9537/9537), 6.50 MiB | 1.09 MiB/s, done.
Resolving deltas: 100% (4747/4747), done.

[advancedresearch@advancedresearch Downloads]$ cd issue-1669/
[advancedresearch@advancedresearch issue-1669]$ ./configure --prefix=$RISC_V --with-arch=rv32gv --with-abi=ilp32
checking for gcc... gcc
checking whether the C compiler works... yes
checking for C compiler default output file name... a.out
checking for suffix of executables... 
checking whether we are cross compiling... no
checking for suffix of object files... o
checking whether the compiler supports GNU C... yes
checking whether gcc accepts -g... yes
checking for gcc option to enable C11 features... none needed
checking for grep that handles long lines and -e... /usr/bin/grep
checking for fgrep... /usr/bin/grep -F
checking for grep that handles long lines and -e... (cached) /usr/bin/grep
checking for bash... /bin/sh
checking for __gmpz_init in -lgmp... yes
checking for mpfr_init in -lmpfr... yes
checking for mpc_init2 in -lmpc... yes
checking for curl... /usr/bin/curl
checking for wget... /usr/bin/wget
checking for ftp... /usr/bin/ftp
configure: creating ./config.status
config.status: creating Makefile
config.status: creating scripts/wrapper/awk/awk
config.status: creating scripts/wrapper/sed/sed

[advancedresearch@advancedresearch issue-1669]$ make
cd /home/advancedresearch/Downloads/issue-1669 && \
flock `git rev-parse --git-dir`/config git submodule init /home/advancedresearch/Downloads/issue-1669/gcc/ && \
flock `git rev-parse --git-dir`/config git submodule update --progress --depth 1 /home/advancedresearch/Downloads/issue-1669/gcc/
Submodule 'gcc' (https://gcc.gnu.org/git/gcc.git) registered for path 'gcc'
Cloning into '/home/advancedresearch/Downloads/issue-1669/gcc'...
remote: Enumerating objects: 146242, done.        
remote: Counting objects: 100% (146242/146242), done.        
remote: Compressing objects: 100% (112586/112586), done.        
remote: Total 146242 (delta 37681), reused 90993 (delta 31629), pack-reused 0        
Receiving objects: 100% (146242/146242), 175.63 MiB | 499.00 KiB/s, done.
Resolving deltas: 100% (37681/37681), done.
remote: Total 0 (delta 0), reused 0 (delta 0), pack-reused 0
error: Server does not allow request for unadvertised object 04696df09633baf97cdbbdd6e9929b9d472161d3
fatal: Fetched in submodule path 'gcc', but it did not contain 04696df09633baf97cdbbdd6e9929b9d472161d3. Direct fetching of that commit failed.
make: *** [Makefile:348: /home/advancedresearch/Downloads/issue-1669/gcc/.git] Error 128

Please suggest me what to do

@TommyMurphyTM1234
Copy link
Collaborator

TommyMurphyTM1234 commented Feb 25, 2025

Now if i follow the way told above it still fails

You haven't done anything different in the second attempt that I can see so the results/failure are the same.

Please suggest me what to do

Please read my earlier post:

In summary - to workaround the "shallow clone" problem for now do this:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669

cd issue-1669

# Work around for "shallow clone" problem thanks to @hpretl 
# https://github.com/riscv-collab/riscv-gnu-toolchain/issues/1669#issuecomment-2657214585
sed -i '/shallow = true/d' .gitmodules
sed -i 's/--depth 1//g' Makefile.in

./configure ...

make

@Awab-Ahsan
Copy link

Now if i follow the way told above it still fails

You haven't done anything different in the second attempt that I can see so the results/failure are the same.

Please suggest me what to do

Please read my earlier post:

In summary - to workaround the "shallow clone" problem for now do this:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669

cd issue-1669

# Work around for "shallow clone" problem thanks to @hpretl 
# https://github.com/riscv-collab/riscv-gnu-toolchain/issues/1669#issuecomment-2657214585
sed -i '/shallow = true/d' .gitmodules
sed -i 's/--depth 1//g' Makefile.in

./configure ...

make

gcc is started building by using this method, it is taking more time than usual but so far going great. Will update if it resolves the issue. Thanks alot

@riscv-collab riscv-collab deleted a comment from TVS2004Chaitanya Feb 25, 2025
@riscv-collab riscv-collab deleted a comment from Awab-Ahsan Feb 25, 2025
@TommyMurphyTM1234
Copy link
Collaborator

@TVS2004Chaitanya - please do not post off topic additional issues about vector support in this thread. If necessary open a new issue. I have deleted the off topic posts.

@TommyMurphyTM1234
Copy link
Collaborator

The workaround (removing shallow = true and --depth 1) does not work for me. Even a standard git submodule update --init fails with ...

Please read my earlier post:

In summary - to workaround the "shallow clone" problem for now do this:

git clone https://github.com/riscv-collab/riscv-gnu-toolchain issue-1669

cd issue-1669

# Work around for "shallow clone" problem thanks to @hpretl 
# https://github.com/riscv-collab/riscv-gnu-toolchain/issues/1669#issuecomment-2657214585
sed -i '/shallow = true/d' .gitmodules
sed -i 's/--depth 1//g' Makefile.in

./configure ...

make

@aWZHY0yQH81uOYvH
Copy link

The workaround (removing shallow = true and --depth 1) does not work for me. Even a standard git submodule update --init fails with ...

Please read my earlier post:

It's worth noting that it seems like you have to delete and re-clone the entire repo. Yesterday I was following these same steps but just deleted the broken submodules which did not allow the workaround to work.

@TommyMurphyTM1234
Copy link
Collaborator

It's worth noting that it seems like you have to delete and re-clone the entire repo.

Yes, that's why my instructions above show a clean/new clone of the repo rather than trying to use the workaround on the results of a previously failed clone.

@TommyMurphyTM1234
Copy link
Collaborator

Fixed by this PR:

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

7 participants