Fri 05 Oct 2007 12:40:13 PM UTC, original submission:
hi samizdat-devel,
PROBLEM: samizdat converts mo files created from UTF-8 po files
back into what it "thinks" is the best charset for that language.
Examples:
- system one: es and fr are converted to
iso-8859-1 (but de remains in UTF-8)
- system two: converting eo to a
charset which i haven't identified, but shows up as unknown
symbols when interpreted as UTF-8.
CHECKS: i checked several times that the po files are in
UTF-8 and that the conversions with msgfmt have been made
correctly and installed and read.
BUG HISTORY:
i reported this back in January earlier this year:
http://lists.gnu.org/archive/html/samizdat-devel/2007-01/msg00011.html
i thought it was solved with the 0.6.0 version, but this is
probably due to several coincidences related to the nature
of the bug (see analysis below) (e.g. i started the samizdat
init.d script from the command line in a UTF-8 environment).
VERSIONS:
- samizdat jan 2007 (i don't remember what version): bug occurs
- samizdat-0.6.0.20070818-1: bug occurs
ruby-gettext version:
- 0.8.0-1 (debian sarge!) bug does NOT occur
- 1.7.0-1 bug occurs
- 1.10.0 - i just tested one obvious chunk - bug still occurs
ENVIRONMENT VARIABLES:
Env variables clearly have some role to play here. i
don't seem to be able to modify these from inside of
ruby, or not in any way that affected the bug, in any
case. When setting e.g. export LC_ALL=ja_JP.UTF-8
in a shell and then stopping/starting samizdat and
apache (to clear the cache), this did prevent charset
conversion to non-UTF-8 charsets.
However, expecting sysadmin to start up samizdat
from an interactive shell would clearly be a bad solution.
FURTHER ANALYSIS:
Commenting out the following iconv line in gettext/mo.rb
fixes the bug (but would create problems for other ruby
users on the system):
This seems to confirm the analysis of what is wrong.
ANALYSIS SUMMARY:
Based on experiments to correct the bug, it seems to me
that we could say that in some sense ruby-gettext-1.7.0
or later is "trying to be too clever",
since it decides to convert
a utf-8 charset mo file back to what is more "natural"
based on some definition (system) of what is the most
natural locale. To someone working from a terminal (shell),
this probably is a good idea. But for a daemon that's
probably not so good.
SOLUTIONS:
i tried solving it from a relatively high level,
i.e. in samizdat_bindtextdomain in engine.rb,
but this did not get transferred down to mo.rb.
Going up from mo.rb, one solution could be changing this
line:
to be hardwired to @@current.charset = 'utf-8', though of
course this should be an override within samizdat, not
a change in the shared copy of the ruby source code.
SOLUTION FOUND: (attached) which seems to work and seems
relatively compact is in the patch attached. It's in
engine.rb and overrides the reading in of an output_charset
in the MOFile class, resetting it to nil, so that
no attempt can be made to make any conversion, since
it's unknown to what charset the conversion should be
made.
COMMENT: If someone understands better than me how to start
up samizdat (through the init.d/samizdat script) in a way
that gives it the appropriate environment variable, then
that would probably be a better solution.
cheers
boud
|