bugMHonArc - Bugs: bug #12314, linebreak not utf-8 aware

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #12314: linebreak not utf-8 aware

Submitted by:  Jeff Breidenbach <jab>
Submitted on:  Mon 14 Mar 2005 07:49:37 AM UTC  
 
Category: Character SetsSeverity: 3 - Normal
Priority: 5 - NormalItem Group: Undesired Behavior
Status: FixedPrivacy: Public
Assigned to: NoneOpen/Closed: Closed
Operating System: LinuxPerl Version: 5.8.4
Component Version: 2.6.10Fixed Release: 2.6.13

Mon 20 Jun 2005 04:30:24 AM UTC, comment #2:

Applied a fix that will work with versions of Perl with
the Encode module installed (i.e Perl >=5.8). The fix converts
the input to perl's internal utf-8 so line breaking is done on characters
vs bytes. The data is translated back after operation.

To avoid unnecessary translations, translation is not done on
us-ascii or iso-8859 sets, common 7/8-bit charsets. Of course,
conversion could be avoided for other 8-bit sets, but charset check
may cause more overhead then translation, so only common ones are
checked for.

Anyone who can, please test changes to make sure nothing got
broke. Note, translation only done when maxwidth is set.
Changes will be available in next snapshot build.

Side Comment: For the future, it is worth considering that
all textual content get normalized to Perl's internal utf-8
format. Of course, such a change would cause mhonarc to
not be compatible with versions of Perl < 5.8.

Earl Hood <ehood>
Project Administrator
Mon 14 Mar 2005 07:52:35 AM UTC, comment #1:

> This one is interesting. We have the told Mhonarc to break lines
> once they hit 80 characters. This was for English language users
> who don't know how to hit the return key. Our CSS layout isn't
> happy when the message body gets too wide.
>
> However, it looks to me that the linebreak is having trouble with
> Japanese in two places. First, it doesn't seem smart enough to
> understand that a linebreak can occur between any two Japanese
> characters (instead it is looking for ASCII space characters).


Checking mhtxtplain.pl, the line breaking code is charset unaware.
Therefore, it does assume that ASCII-based octets that indicate a
white space can be broken on.

For most charsets, this is generally not a problem, with the following
exceptions:

  • Charsets that are not a superset of ASCII.
  • Charsets that have white space characters represented by non-ASCII

octet equivalents.

  • Multi-byte charsets: Well-designed multi-bytes charsets normally

avoid using octets between 0 and 127 within multi-byte sequences to
be friendly to old C string functions and non-multi-byte-aware
software. However, this is not always the case.

Also, multi-byte characters will throw-off the "max-width" line breaking.
I.e. The line-breaking code only deals with octets, not characters.
Therefore, multi-byte character text may end up with "short" lines
after line breaking.

> Earl, do you have any thoughts?


maxwidth option has limitations with multi-byte encodings :)

(Note: maxwidth is not applicable for flowed text messages.)

Some code analysis would need to be done to see what is the best
approach to deal with issue. Doing such an analysis depends on
how problematic the maxwidth limitations are. Also, line breaking
is done with flowed text messages, so any solution would have to
be generalized for all line-breaking operations.

My initial thoughts would to do something similiar as the TEXTCLIPFUNC
route. I.e. Have the ability to customize how line-breaking is
done, with the routine provided a charset argument so the routine
can be charset aware.

(Personal side note: TEXTCLIPFUNC may need to support a charset
argument, but I do not know if this will be easy to do.)

> Also Earl, do you have a preferred
> venue for this type of question? Private email, mhonarc-user, mhonarc-dev,
> etc?


mhonarc-user or mhonarc-dev is okay. If technical in nature and
related to how mhonarc code may need to be changed, dev is the
better list. When in doubt, send to mhonarc-user, and that I can
redirect the discussion to mhonarc-dev if the discussion warrants it.

--ewh

Jeff Breidenbach <jab>
Mon 14 Mar 2005 07:49:37 AM UTC, original submission:

maxwidth=80 does poorly on mixed English/Japanese UTF-8

Jeff Breidenbach <jab>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 5 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Wed 06 Jul 2005 04:36:00 AM UTCehoodStatusReady For Test=>Fixed
  Open/ClosedOpen=>Closed
  Fixed ReleaseCVS=>2.6.13
Mon 20 Jun 2005 04:31:03 AM UTCehoodStatusNone=>Ready For Test
  Fixed Release=>CVS

Back to the top


Powered by Savane 3.1-cleanup1