bugMHonArc - Bugs: bug #12314, linebreak not utf-8 aware


You are not allowed to post comments on this tracker with your current authentication level.

bug #12314: linebreak not utf-8 aware

Submitted by:  Jeff Breidenbach <jab>
Submitted on:  Mon 14 Mar 2005 07:49:37 AM UTC  
Category:  Character Sets Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Undesired Behavior
Status:  Fixed Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Operating System:  Linux Perl Version:  5.8.4
Component Version:  2.6.10 Fixed Release:  2.6.13

Mon 20 Jun 2005 04:30:24 AM UTC, comment #2: 

Applied a fix that will work with versions of Perl with
the Encode module installed (i.e Perl >=5.8).  The fix converts
the input to perl's internal utf-8 so line breaking is done on characters
vs bytes.  The data is translated back after operation.

To avoid unnecessary translations, translation is not done on
us-ascii or iso-8859 sets, common 7/8-bit charsets.  Of course,
conversion could be avoided for other 8-bit sets, but charset check
may cause more overhead then translation, so only common ones are
checked for.

Anyone who can, please test changes to make sure nothing got
broke.  Note, translation only done when maxwidth is set.
Changes will be available in next snapshot build.

Side Comment: For the future, it is worth considering that
all textual content get normalized to Perl's internal utf-8
format.  Of course, such a change would cause mhonarc to
not be compatible with versions of Perl < 5.8.

Earl Hood <ehood>
Project Administrator
Mon 14 Mar 2005 07:52:35 AM UTC, comment #1: 

> This one is interesting. We have the told Mhonarc to break lines
> once they hit 80 characters. This was for English language users
> who don't know how to hit the return key. Our CSS layout isn't
> happy when the message body gets too wide.
> However, it looks to me that the linebreak is having trouble with
> Japanese in two places. First, it doesn't seem smart enough to
> understand that a linebreak can occur between any two Japanese
> characters (instead it is looking for ASCII space characters).

Checking mhtxtplain.pl, the line breaking code is charset unaware.
Therefore, it does assume that ASCII-based octets that indicate a
white space can be broken on.

For most charsets, this is generally not a problem, with the following

  • Charsets that are not a superset of ASCII.
  • Charsets that have white space characters represented by non-ASCII

octet equivalents.

  • Multi-byte charsets: Well-designed multi-bytes charsets normally

avoid using octets between 0 and 127 within multi-byte sequences to
 be friendly to old C string functions and non-multi-byte-aware
 software.  However, this is not always the case.

 Also, multi-byte characters will throw-off the "max-width" line breaking.
 I.e.  The line-breaking code only deals with octets, not characters.
 Therefore, multi-byte character text may end up with "short" lines
 after line breaking.

> Earl, do you have any thoughts?

maxwidth option has limitations with multi-byte encodings :)

(Note: maxwidth is not applicable for flowed text messages.)

Some code analysis would need to be done to see what is the best
approach to deal with issue.  Doing such an analysis depends on
how problematic the maxwidth limitations are.  Also, line breaking
is done with flowed text messages, so any solution would have to
be generalized for all line-breaking operations.

My initial thoughts would to do something similiar as the TEXTCLIPFUNC
route.  I.e.  Have the ability to customize how line-breaking is
done, with the routine provided a charset argument so the routine
can be charset aware.

(Personal side note: TEXTCLIPFUNC may need to support a charset
argument, but I do not know if this will be easy to do.)

> Also Earl, do you have a preferred
> venue for this type of question? Private email, mhonarc-user, mhonarc-dev,
> etc?

mhonarc-user or mhonarc-dev is okay.  If technical in nature and
related to how mhonarc code may need to be changed, dev is the
better list.  When in doubt, send to mhonarc-user, and that I can
redirect the discussion to mhonarc-dev if the discussion warrants it.


Jeff Breidenbach <jab>
Mon 14 Mar 2005 07:49:37 AM UTC, original submission:  

maxwidth=80 does poorly on mixed English/Japanese UTF-8

Jeff Breidenbach <jab>


No files currently attached


Depends on the following items: None found

Items that depend on this one: None found


CC list is empty


Do you think this task is very important?
If so, you can add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.




Follow 5 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-07-06 ehood StatusReady For Test => Fixed
    Open/ClosedOpen => Closed
    Fixed ReleaseCVS => 2.6.13
2005-06-20 ehood StatusNone => Ready For Test
    Fixed Release => CVS

Back to the top

Powered by Savane 3.5