bugMHonArc - Bugs: bug #11187, incorrectly parsing UTF-8 encoded...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #11187: incorrectly parsing UTF-8 encoded messages

Submitter:  Egmont Koblinger <egmont>
Submitted:  Thu 02 Dec 2004 12:04:40 AM UTC
   
 
Category:  Character Sets Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Behavior
Status:  Fixed Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Operating System:  Linux Perl Version:  5.8.5
Component Version:  2.6.10 Fixed Release:  2.6.11

Jump to the original submission

Fri 03 Dec 2004 08:41:25 PM UTC, comment #6: 

Fix checked into CVS.

Earl Hood <ehood>
Group administrator
Fri 03 Dec 2004 08:11:21 PM UTC, comment #5: 

Yes, your patch is definitely nicer than my one. I told you
I'm beginner in perl :-)
Thanks for the fix!

Anonymous
Fri 03 Dec 2004 06:42:28 PM UTC, comment #4: 

The sample patch provided is not applicable for 5.6.x since
the Encode module is only available for 5.8.x and later.

After some searching, it appears that adding the "U0"
specifier to unpack makes things work.  I do not know fully
understand why unpack requires this to get things to work,
but it appears to fix the problem.

I've attached a patch to this report.  It will be checked
into CVS after I can resolve some connectivity problems
with the CVS server.

Earl Hood <ehood>
Group administrator
Thu 02 Dec 2004 07:28:13 PM UTC, comment #3: 

Sample patch follows that fixes the problem. It's just
a case study to show what the problem is, depending on the
Encode module may not be nice and I have no idea whether
it's supported in older perls. (Note that I'm absolute
beginner in perl.)

The problem is that when unpack is executed in line 159
(according to the original 2.6.10 source) then its
parameter ($1) is just a sequence of bytes and perl has
no idea that it should be interpreted as utf8. Hence I
guess it interprets it according to latin1 and that's why
unpack doesn't do what we need. Before using unpack we
have to tell perl "hey that's an utf8 string".

Anonymous
Thu 02 Dec 2004 12:58:53 AM UTC, comment #2: 

I attach a test case. This doesn't only happen for one
particular message but rather for every message I write
with mutt using UTF-8 encoding so it's not a problem to
generate a publicly visible test case.
Both the subject and the body contain the following string:
"asdf" then "e acute" (both latin1 and 2) then "e grave"
(only latin1) then "o doubleacute" (only latin2) then an
euro sign (neither latin1 nor latin2) followed by "jkl;".

The input directory contains the message. The output-actual
directory was generated with mainstream mhonarc 2.6.10 using
"mhonarc -outdir output-actual input". Similarly
output-expected was generated with mhonarc patched as
described above. All this packed into a single tarball.

Egmont Koblinger <egmont>
Thu 02 Dec 2004 12:36:47 AM UTC, comment #1: 

Can submitter please zip up sample message and send it to the author's address for evaluation?  Or you can attach the bundle  to this bug report if it is okay that the email message is readable by the public.

Please also provide sample correct and incorrect conversion of the message.

Earl Hood <ehood>
Group administrator
Thu 02 Dec 2004 12:04:40 AM UTC, original submission:  

I use mhonarc without any configuration file, just simply
the command "mhonarc -outdir outdir indir" whereas "indir"
only contains one file with one single message encoded in
UTF-8. (Both the subject and the body contain UTF-8 encoded
accented letters, the subject uses quoted-printable, the
body's transfer encoding is 8-bit).

The output html files are quite strange. For each UTF-8
byte sequence only the first byte is taken into account
and it is converted to a html escape. For example, the
Euro sign (U+20AC, UTF-8: E2 82 AC) will appear in the html
output as "&#E2;" and then 82 and AC are skipped, processing
goes on with the next Unicode character.

In MHonarc/CharEnt.pm line 153 there's a switch to check
whether perl is new enough to support UTF-8. If it isn't,
then manual processing of UTF-8 character takes place.
Forcing the "non-UTF-8-aware perl" branch of the "if"
statement (that is, changing the "if ($] >= 5.006)" to
"if (0)" repairs the problem, in this case the output will
be the expected "&#20AC;".

I don't think it matters, but I have LANG=hu_HU (latin2
locale) and no other LC_* variables set. However, UTF-8
locales are also available on my system.

Egmont Koblinger <egmont>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #2139:  mhonarc-utf8-CharEnt.patch added by ehood (346B - application/octet-stream - UTF-8 to entity ref patch that works for Perl 5.6.x and 5.8.x)
file #2124:  mhonarc-utf8.patch added by None (516B - application/octet-stream - sample fix)
file #2122:  mhonarc-utf8.tar.gz added by egmont (3KiB - application/x-gzip - example)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 7 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-05-20 ehood Open/ClosedOpen Closed
    Fixed ReleaseCVS 2.6.11
2004-12-03 ehood StatusNone Fixed
    Fixed Release CVS
2004-12-03 ehood Attached File- Added mhonarc-utf8-CharEnt.patch, #1938
2004-12-02 None Attached File- Added mhonarc-utf8.patch, #1936
2004-12-02 egmont Attached File- Added mhonarc-utf8.tar.gz, #1933

Back to the top

Powered by Savane 3.13-f8d8.
Corresponding source code