helpSavannah Administration - Support: sr #110553, Allow fetch of specific commit...

 
 

sr #110553: Allow fetch of specific commit hash from git://git.sv.gnu.org

Submitted by:  g10
Submitted on:  Thu 21 Oct 2021 06:33:22 AM UTC
 
Category:  None Priority:  5 - Normal
Severity:  3 - Normal Status:  Done
Assigned to:  ineiev Operating System:  None
Open/Closed:  Closed
* Mandatory Fields

Add a New Comment (Rich Markup)
   

Comment Type & Canned Response:
       

 

( Jump to the original submission )

Fri 05 Nov 2021 07:25:12 PM UTC, comment #7: 

comment #5:

> AFAIK https://git.savannah.gnu.org/git lists all Git repositories on Savannah.  This feature is managed separately for each repository.  I didn't turn it all for all Savannah repositories because I wasn't sure that would be the right thing to do.


Ok, I don't really need it on any particular repo on https://git.savannah.gnu.org/git right now, so its fine. I was thinking it might be useful for GRUB, for which I'm working on some automated testing scripts. But now I'm thinking it won't be needed.

As I understand it, the trade off for the git server in enabling this is some extra computation for less bandwidth usage. Since no one is complaining about it, I doubt the load would increase much with this option turned on for all repos because it wouldn't be used much.

I'll be back if it turns out that this feature would be useful for me on a particular repo.

g10
  Spam posted by _279005
Thu 28 Oct 2021 07:21:25 AM UTC, comment #5: 

AFAIK https://git.savannah.gnu.org/git lists all Git repositories on Savannah.  This feature is managed separately for each repository.  I didn't turn it all for all Savannah repositories because I wasn't sure that would be the right thing to do.

Ineiev <ineiev>
Site Administrator
Wed 27 Oct 2021 04:31:13 PM UTC, comment #4: 

I've also noticed that this issue exists for the git server at https://git.savannah.gnu.org/git as well. I assumed they were the same git server. Would it be possible to add the configuration change to this git server as well?

g10
Fri 22 Oct 2021 05:43:53 PM UTC, comment #3: 

Yes, much faster than cloning. I've confirmed that the changes do work. The checkout is now under 1 minute.

Thank you!

root@debian:~/gnulib-test# git -C gnulib remote add origin git://git.sv.gnu.org/gnulib
root@debian:~/gnulib-test# time git -C gnulib fetch --depth=2 origin d271f868a8df9bbec29049d01e056481b7a1a263
remote: Counting objects: 9705, done.
remote: Compressing objects: 100% (8668/8668), done.
remote: Total 9705 (delta 5051), reused 2198 (delta 1014)
Receiving objects: 100% (9705/9705), 8.25 MiB | 850.00 KiB/s, done.
Resolving deltas: 100% (5051/5051), done.
From git://git.sv.gnu.org/gnulib

  • branch            d271f868a8df9bbec29049d01e056481b7a1a263 -> FETCH_HEAD

real    0m29.022s
user    0m7.136s
sys     0m2.133s
root@debian:~/gnulib-test# time git -C gnulib reset --hard FETCH_HEAD
Updating files: 100% (9636/9636), done.
HEAD is now at d271f86 lock: Fix link error with --enable-threads=pth.

real    0m7.662s
user    0m2.395s
sys     0m4.957s

g10
Fri 22 Oct 2021 08:58:07 AM UTC, comment #2: 

I've set uploadpack.allowReachableSHA1InWant to true in gnulib.git config. git-config(1) says that the option may be computationally expensive, but probably for gnulib repository it's faster than cloning.

Please check if it works.

Ineiev <ineiev>
Site Administrator
Thu 21 Oct 2021 07:59:48 AM UTC, comment #1: 

For some numbers, not in a VM, but on a laptop, the clone takes about 5 minutes, but the fetch shallow fetch takes about 1 minute.

On the qemu VM (with 2G of memory) its 42 minutes(!!) for the clone and 20 seconds total for the shallow fetch. The VM was on a different network with a lot more bandwidth so receiving the objects was much faster than the laptop, but the resolving deltas phase was much longer. It seems like the resolving deltas duration increases non-linearly with the number of deltas.

# time git -C gnulib fetch --depth=2 origin d271f868a8d
f9bbec29049d01e056481b7a1a263
remote: Enumerating objects: 9705, done.
remote: Counting objects: 100% (9705/9705), done.
remote: Compressing objects: 100% (8661/8661), done.
remote: Total 9705 (delta 5116), reused 1992 (delta 1021), pack-reused 0
Receiving objects: 100% (9705/9705), 8.21 MiB | 1.19 MiB/s, done.
Resolving deltas: 100% (5116/5116), done.
From https://github.com/coreutils/gnulib

  • branch            d271f868a8df9bbec29049d01e056481b7a1a263 -> FETCH_HEAD

real    0m13.167s
user    0m8.278s
sys     0m2.569s

# time git -C gnulib reset --hard FETCH_HEAD
Updating files: 100% (9636/9636), done.
HEAD is now at d271f86 lock: Fix link error with --enable-threads=pth.

real    0m7.529s
user    0m2.353s
sys     0m4.876s

=========================================================

# time git clone git://git.sv.gnu.org/gnulib gnulib-official.git
Cloning into 'gnulib-official.git'...
remote: Counting objects: 244240, done.
remote: Compressing objects: 100% (29927/29927), done.
remote: Total 244240 (delta 214262), reused 244208 (delta 214241)
Receiving objects: 100% (244240/244240), 51.13 MiB | 1.50 MiB/s, done.
Resolving deltas: 100% (214262/214262), done.
Updating files: 100% (10502/10502), done.

real    42m23.480s
user    41m20.532s
sys     0m25.876s

g10
Thu 21 Oct 2021 06:33:22 AM UTC, original submission:  

GRUB2 uses the gnulib project at a specific commit to build. It uses the bootstrap script from the gnulib project to clone the gnulib git repo. Since the GRUB2 bootstrap script configured to use gnulib at a particular commit, it can not use clone (clone doesn't support commit hashes, that I know of), so it must download the whole repository and then checkout that commit.

This is problematic on slower systems (eg. a VM) where the clone can take more than 15 minutes (gnulib has a lot of commits). After reading this SO comment[1], I see there's a much faster way to checkout just a commit. However, when I try to fetch using the commit hash the server returns an error:

$ git fetch origin d271f868a8df9bbec29049d01e056481b7a1a263
error: Server does not allow request for unadvertised object d271f868a8df9bbec29049d01e056481b7a1a263

I've tried this with the github mirror of gnulib, which does support this and it is indeed much faster.

I believe there is a server side configuration that git://git.sv.gnu.org maintainers could turn on to enable this feature. Are there any objections to having this enabled? I'd like to get a patch submitted to gnulib to enable this in the bootstrap script so this can be of use to other projects using bootstrap.

Thanks

[1] https://stackoverflow.com/a/3489576/2108011

g10

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by _279005 (Posted a comment)
  • -email is unavailable- added by ineiev (Posted a comment)
  • -email is unavailable- added by g10 (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

     

     

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2021-11-08 ineiev StatusNeed Info => Done
        Open/ClosedOpen => Closed
    2021-10-28 ineiev StatusDone => Need Info
        Open/ClosedClosed => Open
    2021-10-25 ineiev StatusReady For Test => Done
        Open/ClosedOpen => Closed
    2021-10-22 ineiev StatusNone => Ready For Test
        Assigned toNone => ineiev

    Back to the top


    Powered by Savane 3.9