bugSamizdat - Bugs: bug #21263, close after ruby-gettext-1.90.0 in...

 
 

bug #21263: close after ruby-gettext-1.90.0 in deb stable; ruby-gettext >= 1.7.0-1 < 1.90.0 converts UTF-8 mo files to other charset

Submitter:  Boud Roukema <boud>
Submitted:  Fri 05 Oct 2007 12:40:13 PM UTC
   
 
Category:  None Severity:  2 - Minor
Status:  Wont Fix Privacy:  Public
Assigned to:  boud Open/Closed:  Closed
* Mandatory Fields

Add a New Comment Rich Markup
   

Tue 04 Mar 2008 03:55:23 PM UTC, comment #5: 

ruby-gettext-1.90.0 is in Debian/unstable now, I've brought Samizdat up-to-date with it and uploaded samizdat-0.6.1 to unstable, too. They both will progress towards testing and eventually stable on the same schedule, so there's no reason to keep this bug open any longer.

Dmitry Borodaenko <angdraug>
Group administrator
Thu 21 Feb 2008 10:08:14 PM UTC, comment #4: 

On second thoughts, since we want samizdat to get into
debian stable, i think it's unreasonable to close this
bug until ruby-gettext-1.90.0 gets into debian stable.

Not many people installing samizdat and finding the bug
are going to isolate it and find out that it's a ruby-gettext
bug. Chances are they'll check the list of open samizdat
bugs and if they don't find an entry, they may either
report the bug as a new bug, or else decide that samizdat
has too many unfixed bugs and and become discouraged from
using it.

So by leaving this open, but with a clear warning in
the title:

"close after ruby-gettext-1.90.0 in deb stable; ruby-gettext >= 1.7.0-1 < 1.90.0 converts UTF-8 mo files to other charset"

hopefully we'll give sufficient info to people noticing the
bug but also it should be easy to know when to close the
bug when (i guess 18-36 months from now)
rubygettext-1.90.0 or later gets into debian stable.

i realise that strictly speaking it's not a samizdat bug,
but the bug does affect samizdat, so this info should
be placed in an obvious place.  Maybe an alternative
would be to add a file "BUGS" in the main samizdat package
and list both this rubygettext bug and the  sync.rb bug,
explaining that both are external to samizdat and giving
details of the fixes and the versions of ruby and rubygettext
in which they have been fixed?

Boud Roukema <boud>
Group Member
Mon 18 Feb 2008 12:22:46 AM UTC, comment #3: 

Masao Mutoh found a bug in ruby-gettext:

http://rubyforge.org//tracker/index.php?func=detail&aid=15513&group_id=855&atid=3378


diff -u -r1.32 gettext.rb
--- gettext.rb  29 Oct 2007 15:32:39 -0000      1.32
+++ gettext.rb  10 Nov 2007 02:31:43 -0000
@@ -86,7 +86,7 @@
     end
     opt[:locale] = opt[:locale] ? Locale::Object.new(opt[:locale])
: Locale.get
     opt[:charset] = output_charset if output_charset
-    locale.charset = opt[:charset] if opt[:charset]
+    opt[:locale].charset = opt[:charset] if opt[:charset]
     Locale.set_current(opt[:locale])
     target_key = bound_target
     manager = @@__textdomainmanagers[target_key]


This solves the bug on both systems on which it occurred
previously.

This correction appears in ruby-gettext-1.90.0 which was
released just two weeks ago on 3 Feb 2008:
http://rubyforge.org/frs/?group_id=855

It's not yet in debian unstable, but it looks like the debian
pipeline will eventually get it automatically into ruby-1.8 and ruby-1.9
within 6-12 months:
http://packages.debian.org/search?keywords=libgettext&searchon=names&suite=all&section=all

Dmitry: my guess is that you'll prefer to set this bug
as "won't fix" for this reason: let bug corrections flow
through the system systematically, and people who want to
correct them earlier have to know what they're doing (i assume
that's the policy).

So for people who "know what they're doing", the above one
line fix works perfectly as far as i have seen the bug and
tested the fix.

For other people, sit and wait for the software to flow
through the experimental-to-stable pipeline.

i'll close this bug in one week's time (25.02.2008) unless
someone objects.


Boud Roukema <boud>
Group Member
Sat 10 Nov 2007 12:05:33 AM UTC, comment #2: 

forwarded with a bit of samizdat context:
http://rubyforge.org/tracker/index.php?group_id=855&atid=3378


Boud Roukema <boud>
Group Member
Sun 04 Nov 2007 07:18:37 PM UTC, comment #1: 

Please forward this bug to Ruby-GetText developers:
http://rubyforge.org/projects/gettext/

I don't fully understand root cause of this bug, so I'm not sure what is the best way to do it.

Dmitry Borodaenko <angdraug>
Group administrator
Fri 05 Oct 2007 12:40:13 PM UTC, original submission:  

hi samizdat-devel,

PROBLEM: samizdat converts mo files created from UTF-8 po files
back into what it "thinks" is the best charset for that language.
Examples:

  • system one: es and fr are converted to

iso-8859-1 (but de remains in UTF-8)

  • system two: converting eo to a

charset which i haven't identified, but shows up as unknown
symbols when interpreted as UTF-8.

CHECKS: i checked several times that the po files are in
UTF-8 and that the conversions with msgfmt have been made
correctly and installed and read.

BUG HISTORY:
i reported this back in January earlier this year:
http://lists.gnu.org/archive/html/samizdat-devel/2007-01/msg00011.html
i thought it was solved with the 0.6.0 version, but this is
probably due to several coincidences related to the nature
of the bug (see analysis below) (e.g. i started the samizdat
init.d script from the command line in a UTF-8 environment).

VERSIONS:

  • samizdat jan 2007 (i don't remember what version): bug occurs
  • samizdat-0.6.0.20070818-1: bug occurs


ruby-gettext version:

  • 0.8.0-1 (debian sarge!) bug does NOT occur
  • 1.7.0-1 bug occurs
  • 1.10.0 - i just tested one obvious chunk - bug still occurs


ENVIRONMENT VARIABLES:
Env variables clearly have some role to play here. i
don't seem to be able to modify these from inside of
ruby, or not in any way that affected the bug, in any
case.  When setting e.g. export LC_ALL=ja_JP.UTF-8
in a shell and then stopping/starting samizdat and
apache (to clear the cache), this did prevent charset
conversion to non-UTF-8 charsets.
However, expecting sysadmin to start up samizdat
from an interactive shell would clearly be a bad solution.


FURTHER ANALYSIS:

Commenting out the following iconv line in gettext/mo.rb
fixes the bug (but would create problems for other ruby
users on the system):


str = Iconv.iconv(@output_charset, @charset, str).join if @charset


This seems to confirm the analysis of what is wrong.

ANALYSIS SUMMARY:
Based on experiments to correct the bug, it seems to me
that we could say that in some sense ruby-gettext-1.7.0
or later is "trying to be too clever",
since it decides to convert
a utf-8 charset mo file back to what is more "natural"
based on some definition (system) of what is the most
natural locale. To someone working from a terminal (shell),
this probably is a good idea. But for a daemon that's
probably not so good.


SOLUTIONS:
i tried solving it from a relatively high level,
i.e. in samizdat_bindtextdomain in engine.rb,
but this did not get transferred down to mo.rb.

Going up from mo.rb, one solution could be changing this
line:


      @@current.charset ||= @@locale_system_module.get_charset(@@current)


to be hardwired to @@current.charset = 'utf-8', though of
course this should be an override within samizdat, not
a change in the shared copy of the ruby source code.

SOLUTION FOUND: (attached) which seems to work and seems
relatively compact is in the patch attached. It's in
engine.rb and overrides the reading in of an output_charset
in the MOFile class, resetting it to nil, so that
no attempt can be made to make any conversion, since
it's unknown to what charset the conversion should be
made.


COMMENT: If someone understands better than me how to start
up samizdat (through the init.d/samizdat script) in a way
that gives it the appropriate environment variable, then
that would probably be a better solution.

cheers
boud



Boud Roukema <boud>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #14092:  MOFile_hack-0.1.patch added by boud (761B - text/x-patch)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by angdraug (Posted a comment)
  • -email is unavailable- added by boud
  • -email is unavailable- added by boud (Submitted the item)
  •  

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2008-03-04 angdraug Open/ClosedOpen Closed
    2008-02-21 boud Summaryclose after 25.02.2008 if no objections: BUG: ruby-gettext-1.7.0-1 or later converts UTF-8 mo files into another charset close after ruby-gettext-1.90.0 in deb stable; ruby-gettext >= 1.7.0-1 < 1.90.0 converts UTF-8 mo files to other charset
    2008-02-18 boud SummaryBUG: ruby-gettext-1.7.0-1 or later converts UTF-8 mo files into another charset close after 25.02.2008 if no objections: BUG: ruby-gettext-1.7.0-1 or later converts UTF-8 mo files into another charset
    2007-11-04 angdraug Severity3 - Normal 2 - Minor
        StatusWorks For Me Wont Fix
        Assigned toNone boud
    2007-10-05 boud Attached File- Added MOFile_hack-0.1.patch, #14092
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-f8d8.
    Corresponding source code