bugMHonArc - Bugs: bug #20074, extra space in subject


You are not allowed to post comments on this tracker with your current authentication level.

bug #20074: extra space in subject

Submitted by:  Egmont Koblinger <egmont>
Submitted on:  Mon 04 Jun 2007 06:17:24 PM UTC  
Category:  Mail Parsing Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Behavior
Status:  Fixed Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Operating System:  Linux Perl Version:  5.8.8
Component Version:  2.6.16 Fixed Release:  2.6.17

( Jump to the original submission)

Sat 01 Jan 2011 11:29:10 PM UTC, comment #10: 

Yes, solution implemented is inline with comment #2.  The decoding
phase is done after the header is parsed and the $fields
hash is populated.  The decoding phase operates on the
values in the $fields hash.

Earl Hood <ehood>
Project Administrator
Sat 01 Jan 2011 11:20:08 PM UTC, comment #9: 

As far as I understand you, you implemented the solution that I recommended a long ago in comment #2 ? I agree that that's the right way to go, not my patch.

Looking forward to a new stable release, it's been soooo long :-)

Egmont Koblinger <egmont>
Sat 01 Jan 2011 10:35:10 PM UTC, comment #8: 

(mhtxthtml.pl, 2.39)
Text-encoding/decoding of non-ASCII encoded
header text is now a post-processing step after raw parsing
of header is done.  This way, the decoding function works
against an unfolded string, so handling of white space between
two encoded atoms should be handled correctly.

Earl Hood <ehood>
Project Administrator
Sat 01 Jan 2011 07:21:42 PM UTC, comment #7: 

Thanks for resubmitting the patch.  However, I'm going to
look at doing a different solution, where the decoding
is done as a post-process after the basic header parsing.
The header parsing does unfold lines, so it would
simplify things if MIME decoding is done against the $fields
hash after $fields is populated.

This should make things simpler, and avoid the use of
$' (and similiar vars) since the use of those can introduce
extra overhead in Perl's regex processing.

Also, it should avoid unneeded space dropping in the
extremely rare case where "=?" and "?=" may exist
in a message header, but are not part a MIME encoding

Earl Hood <ehood>
Project Administrator
Sat 01 Jan 2011 03:01:30 PM UTC, comment #6: 

Sent you an e-mail, using gmail's web frontend as mua.

The subject is "mhonarc bug 20074 test fóobar1 fóobar2 fóobar3 fóobar4 fóobar5"

Gmail encodes it as
Subject: =?UTF-8?Q?mhonarc_bug_20074_test_f=C3=B3obar1_f=C3=B3obar2_f=C3=B3obar3_f?=

Mhonarc decodes this with a spurious space after "f" in the 4th foobar.

It's extremely easy to construct similar examples. Take basically any mail client, and type a subject line that is long enough, and contains a couple of non-ascii characters here and there.

Egmont Koblinger <egmont>
Sat 01 Jan 2011 02:43:58 PM UTC, comment #5: 

Oh, the patch was the other way around :) Then I mean to say we didn't see any missing space (no new bug), nor any spurious one (the old one is fixed) :)

Egmont Koblinger <egmont>
Sat 01 Jan 2011 02:39:47 PM UTC, comment #4: 

Oops, I've totally missed the e-mail notification about your last comment, a year and a half ago :(

Patch re-attached.

We've been using this patch ever since I created it (3.5y ago) on lists.uhulinux.hu, and haven't seen any spurious space inserted (no new bug), but it fixed a lot of missing spaces.

I can't have any example right now (and I no longer have access to those raw mailboxes, neither can I regenerate the archive without the patch), but I do remember that many messages suffered from the problem, something along the lines of 1 out of every 10 or 20 (given that the language of the mailing lists uses accents quite often)...

Egmont Koblinger <egmont>
Sun 03 May 2009 09:52:23 PM UTC, comment #3: 

First, you could PLEASE resubmit your latest patch.
I had a brain fart and accidently deleted it from this
bug item.

Second, would you provide example data that illustrates
the problem?

One thing is to modify behavior according to what is
seen in the "wild" versus what the standard state about
this.  The core mail specs and the MIME specs come into
play about how to deal with line folding and extra spaces,
and I would have refresh my memory on how things should

Earl Hood <ehood>
Project Administrator
Tue 05 Jun 2007 11:26:13 AM UTC, comment #2: 

I think the correct and elegant solution would be to first concatenate raw header fields that belong together, and then decode the result with MAILdecode_1522_str.
MAILdecode_1522_str already takes care of dropping the space or tab if it's surrounded by two =?...?= blocks, so it would drop the leading space or tab of a physical mailbox line if and only if it should really be dropped.

Egmont Koblinger <egmont>
Tue 05 Jun 2007 11:09:05 AM UTC, comment #1: 

I update a patch. Now it seems to me (looking at more mailbox files) that the space should be dropped only if the previous line ends with a rfc1522-encoded component and the new line starts with another rfc1522 block.

(file #12955)

Egmont Koblinger <egmont>
Mon 04 Jun 2007 06:17:24 PM UTC, original submission:  

I've just upgraded from an ancient sympa to 5.3 and this one runs mhonarc with different arguments. The relevant part is:
utf-8; MHonArc::UTF8::to_utf8; MHonArc/UTF8.pm

I've just noticed a bug that we haven't faced before: in plenty of subjects an extra space or tab character is inserted somewhere in the middle of a word. For example a mail with subject "Blah Bláh Foobar" that was correctly displayed so far might now be displayed as "Blah Bláh Foob ar".

This happens when mail clients encode the long accented subject using =?x?Q?...?= or =?x?B?...?= notation. In this case they can wrap into a new physical line everywhere, even in the middle of a word. For example, the raw mailbox file might look like this:
Subject: =?utf-8?Q?Blah_Bl=C3=A1h_Foob?=

In this case mhonarc preserves the space or tab at the beginning of the new line as if it was part of the subject, though it should drop it. If there is actually a space then mail clients explicitely mention it within one of the =?.?.?...?= blocks.

Ugly but forks-for-me patch attached, I hope it doesn't have negative side effects. I haven't read and understood all the details of rfc 1522 and friends, this patch just happens to work for this situation. I bet you'll have a better solution :-)

Egmont Koblinger <egmont>


Attached Files
file #22330:  mhonarc-2.6.16-header-extra-space-v2.patch added by egmont (2KiB - text/x-patch - the fixed version)


Depends on the following items: None found

Items that depend on this one: None found


Carbon-Copy List
  • -email is unavailable- added by ehood (Posted a comment)
  • -email is unavailable- added by egmont (Submitted the item)

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.




    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2011-01-09 ehood Open/ClosedOpen => Closed
    2011-01-09 ehood StatusReady For Test => Fixed
        Fixed Release => 2.6.17
    2011-01-01 ehood StatusNeed Info => Ready For Test
    2011-01-01 egmont Attached File- => Added mhonarc-2.6.16-header-extra-space-v2.patch, #22330
    2010-12-31 ehood StatusNone => Need Info
    2009-05-03 ehood Attached File#12955 => Removed
    2007-06-05 egmont Attached File- => Added mhonarc-2.6.16-header-extra-space-v2.patch, #12955
    2007-06-04 egmont Attached File- => Added mhonarc-2.6.16-header-extra-space.patch, #12947

    Back to the top

    Powered by Savane 3.6