bugMHonArc - Bugs: bug #20252, [gnu.org #336933] RFC2047 header...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #20252: [gnu.org #336933] RFC2047 header encoding bug

Submitter:  Joshua Ginsberg <jaginsberg>
Submitted:  Mon 25 Jun 2007 03:13:39 PM UTC
   
 
Category:  Character Sets Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Behavior
Status:  Works For Me Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Operating System:  All Perl Version:  5.8.3-19.5
Component Version:  0.7.3 Fixed Release:  N/A

Jump to the original submission

Sat 04 Aug 2007 06:44:47 PM UTC, comment #14: 

My guess the following bug (fixed in v2.6.11) is
the source of the problem:

https://savannah.nongnu.org/bugs/?11187

Since it appears the problem is fixed in a later
version of mhonarc, I'm going to close this item.

If the problem persists after GNU's software update,
we can reopen the issue.

Earl Hood <ehood>
Group administrator
Fri 03 Aug 2007 03:53:53 PM UTC, comment #13: 

Cool, nice to see this will be fixed.

Best regards

Robert Millan <robertmh>
Fri 03 Aug 2007 03:49:19 PM UTC, comment #12: 


We are still running MHonarc 2.6.10 on the GNU lists server. I've just tested the conversion of the message with

  ./mhonarc -single input > output.html

which generates the faulty encoding with 2.6.10, but does the right thing with 2.6.16.

Our lists server is due for a major software upgrade sometime this fall at which point we will regenerate the html archives, which will resolve this problem.

Thanks,
Ward.

Ward Vandewege <ward>
Tue 17 Jul 2007 11:44:56 AM UTC, comment #11: 

Isn't it the same as bug #11187? Sounds similar at least.
(That one was fixed in mhonarc 2.6.11 while gnu.org uses 2.6.10.)

Egmont Koblinger <egmont>
Thu 28 Jun 2007 08:54:50 PM UTC, comment #10: 


> The HTML generated at lists.gnu.org seems to be utf-8 but
> truncated at one byte. Is it possible that mhonarc handled
> conversion, or do we have another conversion tool in place
> here?


Only the gnu.org folks can answer that.

At this time, I cannot confirm that this problem is
due to mhonarc.

You may want submit a bug to the gnu folks directly on this.
I cannot provide much more help w/o knowing what resource
configuration they are using and if there is custom
processing that may introduce problems.

Earl Hood <ehood>
Group administrator
Thu 28 Jun 2007 06:50:56 AM UTC, comment #9: 


Tried that, and got the same result (with 0xe4).  However, this is latin1, and can't possibly work.  As Jeff said:

"Note the combination of Chinese, English, and umlauts; unicode is the only answer."

The HTML generated at lists.gnu.org seems to be utf-8 but truncated at one byte.  Is it possible that mhonarc handled this conversion, or do we have another conversion tool in place here?

Robert Millan <robertmh>
Wed 27 Jun 2007 11:17:19 PM UTC, comment #8: 

Just download mhonarc from www.mhonarc.org, install, and
run.  Mhonarc is independent of any MTA.

To convert a single message, you can do:

mhonarc -single message.822 > message.html

I ran the above on the sample attached message, and
the output looks correct.  The name in question
got translated to:

Vesa J&#xE4;&#xE4;skel&#xE4;inen

I'd need to see the resource settings used by the gnu.org
archives to see what may be wrong in the configuration.
My guess is they could be "flattening out" character handling
for lists, maybe for performance and/or security reasons.

Earl Hood <ehood>
Group administrator
Wed 27 Jun 2007 05:23:21 AM UTC, comment #7: 


I'm attaching a raw copy of the message that generated the bad html.

I don't have a mhonarc install to test it.  Is it possible to install and process a single message right-away without setting up MTA integration, etc?


(file #13182)

Robert Millan <robertmh>
Tue 26 Jun 2007 11:34:47 PM UTC, comment #6: 

For me to do an accurate analysis, I first need access to
the original raw mail message.

Then, it will help to know what resource settings are
being used for the archive in question since some resources
affect how mhonarc process character sets.

A quick dirty test is to run mhonarc (with default settings)
on just the message in question to see what happens.  If
the HTML created looks proper, then the problem is due
to some resource setting.  An example may be if resource
settings assume a single charset for all messages.

If the HTML looks bad, then one possibility is how the
email message is encoded.  I.e.  If the message is not
conforming to email standards, things may not turn out
right.

Since character set processing may leverage different
Perl modules depending on what the given perl installation
provides, it is possible some module may be introducing
errors.

Of course, there may be a bug in MHonArc, but I cannot
tell without testing.  Since Jeff states that the message
can be rendered properly, we at least know something
does work properly :)

Note, even though Jeff states that using TEXTENCODE to
UTF-8 everything can be done, and is generally a good idea,
it is dependent on the search engine that is being used
for the archives.  The gnu.org archives use Namazu, so
UTF-8 encoding is not an option in this case.

Earl Hood <ehood>
Group administrator
Tue 26 Jun 2007 06:31:06 PM UTC, comment #5: 

Erm, I meant to say From of course ;-)

Robert Millan <robertmh>
Tue 26 Jun 2007 06:30:23 PM UTC, comment #4: 


Hi, I'm the person who reported this problem to gnu.org sysadmins.

> Given that, what I'm imagining has happened is that in the first
> case, the UTF-8 characters are assumed to be iso-8859-1, an 8 bit
> character encoding, and are written as the first byte of the
> UTF-8 encoding; however in the second case, I'm supposing that it
> is properly transcoding from utf-8 to latin1.


Sounds plausible.  I suspect what makes it behave differently is that in the first case, subject is base64-encoded (in the second it isn't).

I'm afraid I can't really help.  I have zero knowledge about MHonArc internals (and am not very fluent in perl either).  Sorry.

Robert Millan <robertmh>
Tue 26 Jun 2007 02:59:21 PM UTC, comment #3: 

I understand what you're trying to say, but I'm not sure you're correct.

First, Apache is returning the pages UTF-8 encoded:

HEAD /archive/html/grub-devel/2007-06/msg00004.html HTTP/1.0
Host: lists.gnu.org

HTTP/1.1 200 OK
Date: Tue, 26 Jun 2007 14:49:29 GMT
Server: Apache/2.0.51 (Fedora)
Last-Modified: Fri, 01 Jun 2007 21:19:30 GMT
ETag: "e3105a-11a8-c5549c80"
Accept-Ranges: bytes
Content-Length: 4520
Connection: close
Content-Type: text/html; charset=UTF-8

Second, the encodings presented as entities between the two pages are different. In the first URL, msg00004.html, the special characters are written as &#xC3 whereas in the second URL, they are written as &#xE4.

The correct character has a unicode codepage of 0xe4, an iso-8859-1 encoding of 0xe4, and a utf-8 encoding of 0xc3a4. Given that, what I'm imagining has happened is that in the first case, the UTF-8 characters are assumed to be iso-8859-1, an 8 bit character encoding, and are written as the first byte of the UTF-8 encoding; however in the second case, I'm supposing that it is properly transcoding from utf-8 to latin1.

But I'm not very fluent in the internals of MHonArc. Thoughts?

-jag


Joshua Ginsberg <jaginsberg>
Mon 25 Jun 2007 07:07:05 PM UTC, comment #2: 

And here's the exact message. Note the combination of
Chinese, English, and umlauts; unicode is the only answer.

http://www.mail-archive.com/grub-devel@gnu.org/msg02923.html


Jeff Breidenbach <jab>
Mon 25 Jun 2007 07:03:59 PM UTC, comment #1: 

Not a mhonarc bug.

Almost certainly, mhonarc is converting the name to UTF-8,
but Apache is sending the web page out with an ISO-8859-1
header. Here's an example of mhonarc doing just fine with
a message from the same person.

http://www.mail-archive.com/grub-devel@gnu.org/msg00411.htm

Solution (going forward) is to have mhonarc produce UTF-8
for everything, and for the webserver to label it as such.


Jeff Breidenbach <jab>
Mon 25 Jun 2007 03:13:39 PM UTC, original submission:  

From a user report on lists.gnu.org:

"""
To provide a bit more info, I looked at the headers in my mailbox, and
matched them with HTMLised messages from lists.gnu.org.

It seems that only some forms of encoding are affected. I don't know which
encoding is the first example using, but it displays fine in my mail client
(mutt):

This (http://lists.gnu.org/archive/html/grub-devel/2007-06/msg00004.html):

From: =?UTF-8?B?VmVzYSBKw6TDpHNrZWzDpGluZW4=?= <chaac@nic.fi>

displays as:

From: Vesa JÃÃskelÃinen

this (http://lists.gnu.org/archive/html/grub-devel/2007-05/msg00155.html):

From: =?ISO-8859-1?Q?Vesa_J=E4=E4skel=E4inen?= <chaac@nic.fi>

displays as:

From: Vesa Jääskeläinen
"""

Thanks!

-jag

Joshua Ginsberg <jaginsberg>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #13182:  bad_message added by robertmh (4KiB - application/octet-stream - raw message that generated wrong html)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by ward (Posted a comment)
  • -email is unavailable- added by egmont (Posted a comment)
  • -email is unavailable- added by ehood (Posted a comment)
  • -email is unavailable- added by robertmh (Posted a comment)
  • -email is unavailable- added by jab (Posted a comment)
  • -email is unavailable- added by jaginsberg (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 5 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2007-08-04 ehood Open/ClosedOpen Closed
        Fixed Release N/A
    2007-06-28 ehood StatusNone Works For Me
    2007-06-27 robertmh Attached File- Added bad_message, #13182
    2007-06-25 jaginsberg Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code