helpSavannah Administration - Support: sr #109439, Commit notification hook...

 
 

sr #109439: Commit notification hook mishandles non-ASCII author names

Submitted by:  Ludovic Courtès <civodul>
Submitted on:  Mon 08 Jan 2018 08:31:59 AM UTC  
 
Category:  Source code repositories - developer access Priority:  5 - Normal
Severity:  3 - Normal Status:  None
Assigned to:  None Operating System:  None
Open/Closed:  Open

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

Tue 09 Jan 2018 07:22:10 PM UTC, comment #2: 

I spent some time looking into this problem and the issue is much too complicated to type into a web page text area.  I thought about dragging this conversation over to the mailing list but decided to give it a shot here anyway.  Glenn is correct about the Latin1 encoding being the problem.

There are many problems.  One is that Savannah's web interface is designed around Latin1 not UTF-8.  I don't know what needs to be done to fix the web UI to migrate it from Latin1 to UTF-8.  I didn't try it and am not sure but I am pretty sure that if I update the database to contain UTF-8 content instead of Latin1 content then the web page would be the reverse mangling.

https://savannah.gnu.org/users/civodul

Oh, and there is also a lot of content stored in the database in UTF-8 content too.  Even though the database character encoding is specified as Latin1.  Assaf has an entry describing this problem in the TODO list.  That mismatch is also a problem for other data in the other direction.

In any case here are some data factoids just as general information.  I will dump some data from the MySQL database.

vcs0:~# getent passwd civodul | awk -F: '{print$5}' | od -tx1 -c
0000000  4c  75  64  6f  76  69  63  20  43  6f  75  72  74  e8  73  0a
          L   u   d   o   v   i   c       C   o   u   r   t 350   s  \n

vcs0:~# getent passwd civodul | awk -F: '{print$5}' | iconv -f LATIN1 -t UTF-8 | od -tx1 -c
0000000  4c  75  64  6f  76  69  63  20  43  6f  75  72  74  c3  a8  73
          L   u   d   o   v   i   c       C   o   u   r   t 303 250   s
0000020  0a
         \n

This shows that indeed the content from the database is returned in a Latin1 encoding.  This is then used by git-multimail and onward.  If it were UTF-8 then from here onward through the email it should all work okay.

At the moment I think a reasonable workaround would be handling this in the git-multimail wrapper that we are already using with git-multimail.  It's all Python and I am a Perl guy so please forgive me if I don't know Python well enough to make the changes myself.  But if someone were to propose patches to the python then I think this could be fixed there.  Here is raw access to the git repository including config for git-multimail.  The file needing patching is post-receive.  Looking at that file should give a python person enough information on the process and they should be able to hack in a workaround.

https://git.savannah.gnu.org/git/guix.git/hooks/

If the fromaddr could be passed through "iconv -f LATIN1 -t UTF-8" then I think the result would work around the current Latin1 issues.  Patches solicited.

And one more thing.  We are using git-multimail from just after the 1.0.0 tag plus 3 with two local changes on top of that from 2014.  It's been working well so there hasn't been a need to update.  But if someone were offended that we aren't using the latest version of git-multimail and was willing to test out the new version then I'd be happy to work through the upgrade with them.

Bob Proulx <rwp>
Site Administrator
Tue 09 Jan 2018 04:26:15 AM UTC, comment #1: 

This was reported to git-multimail (which is still what Savannah uses, I think?) years ago, and closed wontfix:

https://github.com/git-multimail/git-multimail/issues/70

Presumably the Savannah user database is still Latin-1.

Glenn Morris <gm>
Mon 08 Jan 2018 08:31:59 AM UTC, original submission:  

Hello,

The email notification hook behind the Guix repositories incorrectly handles non-ASCII commit author names.

See for instance:

  https://lists.gnu.org/archive/html/guix-commits/2018-01/msg00197.html
  https://lists.gnu.org/archive/html/guix-commits/2018-01/msg00198.html
  https://lists.gnu.org/archive/html/guix-commits/2018-01/msg00199.html
  https://lists.gnu.org/archive/html/guix-commits/2018-01/msg00200.html
  https://lists.gnu.org/archive/html/guix-commits/2018-01/threads.html

It also leads to invalid headers like this:

Mail-Followup-To:  "guix-commits@gnu.org, Ludovic Court"@savannah.gnu.org,
"ès <ludo@gnu.org>"@savannah.gnu.org

Could it be that the notification hook is running in a non-UTF8 locale?

TIA,
Ludo'.

Ludovic Courtès <civodul>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rwp (Posted a comment)
  • -email is unavailable- added by gm (Posted a comment)
  • -email is unavailable- added by civodul (Submitted the item)
  • -email is unavailable- added by civodul
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follows 1 latest change.

    Date Changed by Updated Field Previous Value => Replaced by
    2018-01-08 civodul Carbon-Copy- => Added mhw@netris.org

    Back to the top


    Powered by Savane 3.5