mainSavannah Administration - Support: sr #110591, errors during git push

 
 

sr #110591: errors during git push

Submitter:  Bruno Haible <haible>
Submitted:  Fri 31 Dec 2021 11:53:00 PM UTC
   
 
Category:  Source code repositories - developer access Priority:  5 - Normal
Severity:  4 - Important Status:  In Progress
Privacy:  Public Assigned to:  rwp
Operating System:  GNU/Linux Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 03 Jan 2022 09:06:15 PM UTC, comment #9: 

I have finished a first pass scrape across the entire collection of
repositories.  As one might imagine that over the years of spread
between the creation of them, and over the years of different
sysadmins making changes, there are variations.  All of the ones that
are the majority cases have been fixed to comply with the needs of the
current git version and do not have disabled hook scripts in place.
They were all moved to name.sample or name.datestamp type names to
avoid the spurious hint message.

That leaves a reduced list number of 139 that I need to review and
deal with next.

Bob Proulx <rwp>
Site Administrator
Sun 02 Jan 2022 01:16:57 AM UTC, comment #8: 

Background: Earlier git created sample hook scripts in place and included this comment.

# To enable this hook, make this file executable.

Which is to say that there will be a lot of these that are just original files from when the repository was created.  In addition to the custom modifications such as the git_multimail.py commit diff email notifications that we have added, and sometimes removed, from there.

I think there are two main options to resolve the current issue globally.  1) Update the config files, all 4643 of them, to disable this "hint:" from being emitted.  And 2) Rename the sample hook scripts from the in-place naming of the earlier versions of git to the new git practice of naming them hook-name.sample.

Somewhat ironically there is no version history for the git repository config and hook scripts.  Therefore having file timestamps showing when the file was last modified is the only clue left and extremely useful.  I would like to preserve that tiny clue.  That pushes me to the second option of renaming hooks to .sample like git creates now for new repositories.  That avoids the need to touch any dates anywhere.  And the renamed files will of course maintain their previous timestamps too.  Since none of these have version history this leaves as much debug trail as can exist for them.

It's the same number of directories as config file.  But one doesn't do this by hand so it's just machine time either way.  I'll script something up to fix these en masse.

Bob Proulx <rwp>
Site Administrator
Sun 02 Jan 2022 12:53:31 AM UTC, comment #7: 

In this case I think it makes the most sense to delete the hook.

It had at one time supported two things.  One was a CI build that I was maintaining but that went away in 2007.  The other was mail to a target Karl maintained.

I think both of those have expired.  So I am going to clean up and unconfigure the old cruft that was left behind that tripped things up here.

Bob Proulx <rwp>
Site Administrator
Sat 01 Jan 2022 08:24:01 PM UTC, comment #6: 

I had run chmod a+x /usr/local/bin/git-post-receive-hook, thinking that needed to be executable, but that just reenabled something that had not run for years and that no longer works.

I've just restored to prior state via chmod a-x ...

We'll have to address the underlying problem (reported by Paul about hook files not being executable) some other way, possibly by deleting them.

Jim Meyering <meyering>
Site Administrator
Sat 01 Jan 2022 06:51:45 PM UTC, comment #5: 

Today, I get another message from "remote":

remote: Resolving deltas: 100% (352/352), completed with 323 local objects.
remote: Traceback (most recent call last):
remote:   File "hooks/post-receive", line 6, in <module>
remote:     import git_multimail
remote: ImportError: No module named git_multimail


It seems that a Python module 'git_multimail' is being referenced.

Bruno Haible <haible>
Sat 01 Jan 2022 05:57:17 AM UTC, comment #4: 


> It's now running weekly across all of the repositories.


I see. Indeed, there was a number of unusually large commits in gnulib.git over the last few days; maybe that was the trigger, i.e. maybe that was why this week in particular the frequency of 1/7d was not enough.

Thanks for your help!!

Bruno Haible <haible>
Sat 01 Jan 2022 02:33:41 AM UTC, comment #3: 

The git hook problem was a git hook failure exactly as the error message said. It was trying to run a post-receive hook which was a symlink pointing to a location that did not exist on the new system.  I copied that from the old system forward.  It is installed now.  Please try the push again and let me know if it is successful now or not.

If there is one there will be more.  I will run a 'find' looking for symlinks and see if I can spot any additional now dangling symlinks.

For the "suboptimal pack - out of memory" I have added some swap space to increase the total available virtual memory on the system available to processes. 

I should have tried this before adding the swap but I did not.  After adding the swap I ran gc manually.  Note that this is run automatically by cron.  It used to be run once a month on all repositories but Emacs started timing out taking longer than 60s and so frequency of this was increased to four times a month.  It's now running weekly across all of the repositories.

root@vcs2:/net/vcs/git/gnulib.git# /usr/bin/time git gc
Counting objects: 247848, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (31652/31652), done.
Writing objects: 100% (247848/247848), done.
Total 247848 (delta 217482), reused 246201 (delta 216122)
Checking connectivity: 247848, done.

25.34user 1.57system 0:23.57elapsed 114%CPU (0avgtext+0avgdata 811432maxresident)k
128816inputs+134440outputs (8major+259025minor)pagefaults 0swaps

Of course this git-gc was run on top of the previous git-gc and therefore won't be identical.  And it might not even be close to similar, due to the previous action.  It will incrementally work forward each time.  But it does not seem to have needed an excessive amount of memory.

I looked for "gc.log" files and found none present.

I am going to close the ticket but don't think that I have gone away.  I am looking very closely at the system.  Please keep reporting anything you find that is out of the ordinary.

Bob Proulx <rwp>
Site Administrator
Sat 01 Jan 2022 01:54:39 AM UTC, comment #2: 

Thank you for reporting these from-the-client-side events.  Otherwise we would not know about them.  And this is important information to know about.

The first error seems like an actual hook script something though.  I'll look at it and see.  Things are on the new updated server now and there have been some unexpected dependencies that were missing.  Can't discount that until it is examined.

The git-gc packing though does look like a memory resource problem being out-of-memory.  However let me know that the new server has 50% more RAM then the old server had.  At the same time the new server has updated versions of both git and glibc and the Linux kernel.  We know that those things don't get less resource intense over time but always need more resources over time.  But I would be unhappy to hear that it needs more than 50% more RAM with those newer versions though.

Plus previously git shared the system with svn with viewvc, hg, and bzr with loggerhead.  But I haven't migrated those over yet.  So on the new server with 50% more RAM it is only git and nothing but git there.

As I look at the running system it does not look to me like it is in any memory stress at all.  It looks to be running around 1 GB of RAM usage as an average as I am looking at top and such.  That doesn't really mean anything.  But out of the 6 GB available it looks to have plenty available to use.

I will increase the virtual memory avaible with some file system swap as an initial response to the reports that things are out-of-memory.  Maybe there are impulse spikes that are well above what can be handled now in which case increasing swap would help with those spikes.  It's definitely far from experiencing any swap thrashing.

Bob Proulx <rwp>
Site Administrator
Fri 31 Dec 2021 11:53:48 PM UTC, comment #1: 

Probably git.savannah.gnu.org is running low on available memory.

Bruno Haible <haible>
Fri 31 Dec 2021 11:53:00 PM UTC, original submission:  

I can't push to the gnulib git repository any more; it still worked an hour ago.

Once I get an error:

remote: fatal: unable to generate checkdiff for ChangeLog
remote: error: hook declined to update refs/heads/master
To ssh://git.sv.gnu.org/srv/git/gnulib
 ! [remote rejected]     master -> master (hook declined)
error: failed to push some refs to 'ssh://haible@git.sv.gnu.org/srv/git/gnulib'


Another time I get a message that says "error" but is not fatal:

remote: error: The last gc run reported the following. Please correct the root cause
remote: and remove gc.log.
remote: Automatic cleanup will not be performed until the file is removed.
remote:
remote: fatal: Out of memory, malloc failed (tried to allocate 373121 bytes)
remote: error: failed to run repack
remote:
To ssh://git.sv.gnu.org/srv/git/gnulib
   f698ea718..2671376bc  master -> master


Also, Ben Pfaff reported today:

remote: Counting objects: 19423, done.
remote: warning: suboptimal pack - out of memory
remote: Compressing objects: 100% (3819/3819), done.


Bruno Haible <haible>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by meyering (Posted a comment)
  • -email is unavailable- added by rwp (Posted a comment)
  • -email is unavailable- added by haible (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-01-01 rwp Open/ClosedOpen Closed
    2022-01-01 rwp StatusNone In Progress
        Assigned toNone rwp

    Back to the top

    Powered by Savane 3.13-02a9.
    Corresponding source code