bugMHonArc - Bugs: bug #11187, incorrectly parsing UTF-8 encoded...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #11187: incorrectly parsing UTF-8 encoded messages

Submitted by:  Egmont Koblinger <egmont>
Submitted on:  Thu 02 Dec 2004 12:04:40 AM UTC  
 
Category: Character SetsSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Incorrect Behavior
Status: FixedPrivacy: Public
Assigned to: NoneOpen/Closed: Closed
Operating System: LinuxPerl Version: 5.8.5
Component Version: 2.6.10Fixed Release: 2.6.11

(Jump to the original submission Jump to the original submission)

Fri 03 Dec 2004 08:41:25 PM UTC, comment #6:

Fix checked into CVS.

Earl Hood <ehood>
Project Administrator
Fri 03 Dec 2004 08:11:21 PM UTC, comment #5:

Yes, your patch is definitely nicer than my one. I told you
I'm beginner in perl :-)
Thanks for the fix!

Anonymous
Fri 03 Dec 2004 06:42:28 PM UTC, comment #4:

The sample patch provided is not applicable for 5.6.x since
the Encode module is only available for 5.8.x and later.

After some searching, it appears that adding the "U0"
specifier to unpack makes things work. I do not know fully
understand why unpack requires this to get things to work,
but it appears to fix the problem.

I've attached a patch to this report. It will be checked
into CVS after I can resolve some connectivity problems
with the CVS server.

Earl Hood <ehood>
Project Administrator
Thu 02 Dec 2004 07:28:13 PM UTC, comment #3:

Sample patch follows that fixes the problem. It's just
a case study to show what the problem is, depending on the
Encode module may not be nice and I have no idea whether
it's supported in older perls. (Note that I'm absolute
beginner in perl.)

The problem is that when unpack is executed in line 159
(according to the original 2.6.10 source) then its
parameter ($1) is just a sequence of bytes and perl has
no idea that it should be interpreted as utf8. Hence I
guess it interprets it according to latin1 and that's why
unpack doesn't do what we need. Before using unpack we
have to tell perl "hey that's an utf8 string".

Anonymous
Thu 02 Dec 2004 12:58:53 AM UTC, comment #2:

I attach a test case. This doesn't only happen for one
particular message but rather for every message I write
with mutt using UTF-8 encoding so it's not a problem to
generate a publicly visible test case.
Both the subject and the body contain the following string:
"asdf" then "e acute" (both latin1 and 2) then "e grave"
(only latin1) then "o doubleacute" (only latin2) then an
euro sign (neither latin1 nor latin2) followed by "jkl;".

The input directory contains the message. The output-actual
directory was generated with mainstream mhonarc 2.6.10 using
"mhonarc -outdir output-actual input". Similarly
output-expected was generated with mhonarc patched as
described above. All this packed into a single tarball.

Egmont Koblinger <egmont>
Thu 02 Dec 2004 12:36:47 AM UTC, comment #1:

Can submitter please zip up sample message and send it to the author's address for evaluation? Or you can attach the bundle to this bug report if it is okay that the email message is readable by the public.

Please also provide sample correct and incorrect conversion of the message.

Earl Hood <ehood>
Project Administrator
Thu 02 Dec 2004 12:04:40 AM UTC, original submission:

I use mhonarc without any configuration file, just simply
the command "mhonarc -outdir outdir indir" whereas "indir"
only contains one file with one single message encoded in
UTF-8. (Both the subject and the body contain UTF-8 encoded
accented letters, the subject uses quoted-printable, the
body's transfer encoding is 8-bit).

The output html files are quite strange. For each UTF-8
byte sequence only the first byte is taken into account
and it is converted to a html escape. For example, the
Euro sign (U+20AC, UTF-8: E2 82 AC) will appear in the html
output as "&#E2;" and then 82 and AC are skipped, processing
goes on with the next Unicode character.

In MHonarc/CharEnt.pm line 153 there's a switch to check
whether perl is new enough to support UTF-8. If it isn't,
then manual processing of UTF-8 character takes place.
Forcing the "non-UTF-8-aware perl" branch of the "if"
statement (that is, changing the "if ($] >= 5.006)" to
"if (0)" repairs the problem, in this case the output will
be the expected "&#20AC;".

I don't think it matters, but I have LANG=hu_HU (latin2
locale) and no other LC_* variables set. However, UTF-8
locales are also available on my system.

Egmont Koblinger <egmont>

 

Attached Files
file #2139:  mhonarc-utf8-CharEnt.patch added by ehood (346B - application/octet-stream - UTF-8 to entity ref patch that works for Perl 5.6.x and 5.8.x)
file #2124:  mhonarc-utf8.patch added by None (516B - application/octet-stream - sample fix)
file #2122:  mhonarc-utf8.tar.gz added by egmont (3KiB - application/x-gzip - example)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 7 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Fri 20 May 2005 04:09:58 PM UTCehoodOpen/ClosedOpen=>Closed
  Fixed ReleaseCVS=>2.6.11
Fri 03 Dec 2004 08:41:25 PM UTCehoodStatusNone=>Fixed
  Fixed Release=>CVS
Fri 03 Dec 2004 06:42:28 PM UTCehoodAttached File-=>Added mhonarc-utf8-CharEnt.patch, #1938
Thu 02 Dec 2004 07:28:13 PM UTCNoneAttached File-=>Added mhonarc-utf8.patch, #1936
Thu 02 Dec 2004 12:58:54 AM UTCegmontAttached File-=>Added mhonarc-utf8.tar.gz, #1933

Back to the top


Powered by Savane 3.1-cleanup1