bugmldonkey, a multi-networks file-sharing client - Bugs: bug #17618, Reassigned to another tracker...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #17618: Reassigned to another tracker [was: Incorrect charset used for some locales]

Submitter:  Gang Chen <gangchen>
Submitted:  Wed 06 Sep 2006 03:42:00 AM UTC
   
 
Category:  HTTP interface Severity:  4 - Important
Item Group:  i18n issues Status:  Ready For Test
Assigned to:  None Open/Closed:  Closed
Release:  2.7.7 Release: 
Operating System:  Linux Binaries Origin:  CVS / Self compiled
CPU type:  Intel x86

Jump to the original submission

Mon 13 Nov 2006 10:13:30 AM UTC, comment #14: 

THIS ITEM WAS REASSIGNED TO PATCH #5547.

Please, do not post any new comments to this item.


spiralvoice <spiralvoice>
Group administrator
Mon 13 Nov 2006 05:54:31 AM UTC, comment #13: 

The patch works fine for me, thanks.

Gang Chen <gangchen>
Wed 08 Nov 2006 09:30:41 PM UTC, comment #12: 

Updated patch for current CVS, please test and report back if it works for you

spiralvoice <spiralvoice>
Group administrator
Sat 07 Oct 2006 03:40:57 PM UTC, comment #11: 

ping

spiralvoice <spiralvoice>
Group administrator
Thu 14 Sep 2006 05:42:10 PM UTC, comment #10: 

Applied the language detection to CVS, for the display problems
I will leave this bug open.

spiralvoice <spiralvoice>
Group administrator
Tue 12 Sep 2006 10:37:40 PM UTC, comment #9: 

To fix conversion problems enclose the string variable, which
currently outputs bad chars, with   Charset.to_utf8 ( )

spiralvoice <spiralvoice>
Group administrator
Tue 12 Sep 2006 06:45:08 PM UTC, comment #8: 

I updated two places of code with an UTF8-conv function
for the popups. Please test if this is enough or if
there are other places where incorrect chars are displayed.

Are search result filenames displayed well?

spiralvoice <spiralvoice>
Group administrator
Tue 12 Sep 2006 09:01:49 AM UTC, comment #7: 

The problem about tip is the filename are not converted to UTF-8 encoding properly when it displayed in the tip window.
For example, in the src/daemon/driver/driverCommands.ml:

(Http_server.html_real_escaped (Filename.basename impl.impl_shared_codedname))
...
(Filename.basename impl.impl_shared_codedname)

The codedname may not in UTF-8 if the system default encoding is not UTF-8, a conversion is needed before print out the filename. I found the "shorten" method did the conversion properly, so it will be a workaround by calling shorten for filename before write to buffer.
For example:
(shorten (Http_server.html_real_escaped (Filename.basename impl.impl_shared_codedname)) 999)
...
(shorten (Filename.basename impl.impl_shared_codedname) !!max_name_len)

Gang Chen <gangchen>
Tue 12 Sep 2006 07:09:22 AM UTC, comment #6: 

Hi
I verified that your fix can resolve the problem, thank you very much.
While I found another encoding issue in the tips window when mouse move over the filename. Shall I open a new bug for that one?

Thanks

Gang Chen <gangchen>
Mon 11 Sep 2006 08:15:31 AM UTC, comment #5: 

Please test the updated patch, it converts all string to uppercase before analyzing them.

spiralvoice <spiralvoice>
Group administrator
Mon 11 Sep 2006 03:39:06 AM UTC, comment #4: 

I have tested your patch but unfortunately it doesn't works.
The problem is caused by following line:
if String.sub s 0 2 = "ZH" && String.length s > 4 then
The "ZH" is used to match the locale but the locale string
starts with lowercase in most case. It will always fall into
default when the locale starts with language id in lowercase.

Since it's a convention that locale starts with lowercase,
I suggest to always use the lowercase for language id.


Gang Chen <gangchen>
Thu 07 Sep 2006 11:38:06 AM UTC, comment #3: 

"The normalize_language method should also be changed since it will trim the "_CC" in the locale string."

Example: for the German language these values exist:

de_DE (Germany)
de_AT (Austria)
de_CH (Switzerland)
de_LU (Luxemburg)
de_BE (Belgium - has German minority)

for all of them using "de" cut by the normalize function is perfectly ok, otherwise charset_list_from_language would be a
very long function.

In the case of "zh_CN" and "zh_TW" its a different story,
I changed some code so zh_CN and zh_TW are specially treated.

Please try the attached patch and report if it works for you.

spiralvoice <spiralvoice>
Group administrator
Thu 07 Sep 2006 06:15:50 AM UTC, comment #2: 

        --Buildinfo--
Version:         MLNet Multi-Network p2p client version 2.7.7
Networks:        Global Shares  Gnutella  G2  FileTP  BitTorrent  Donkey (SUI)
Ocaml version:   3.09.2 - C compiler version: 4.1.2 - C++ compiler version: 4.1.2
Build on:        Linux i686 2.6.17.11-p4 (little endian) with glibc 2.3.6
Configure args:  /usr/local/share/config.site /usr/local/etc/config.site
Features:        threads zlib-1.2.3 bzip2-1.0.3 gd(jpg/png-1.2.8) iconv no-magic no-check-bounds

        --Runinfo--
User:            admin (PW Protected) - uptime: 1d 37m 50s
Enabled nets:     Donkey Overnet BitTorrent FileTP
Server usage:    enabled
Geoip:           enabled, GeoLite data created by MaxMind, available from http://maxmind.com/
IP blocking:     no blocking list loaded
System info:     Linux lily 2.6.17.11-c3 #1 Mon Sep 4 22:53:39 CST 2006 i686
                 language: ZH - locale: GB2312 - UTC offset: +0800
                 max_string_length: 16777211 - word_size: 32 - max_array_length: 4194303
                 max file descriptors: 1024 - max useable file size: 2^63-1 bits (do the maths ;-p)

NOTE:
This is built by myself from the debian patched source with the workaround.

Gang Chen <gangchen>
Wed 06 Sep 2006 12:29:47 PM UTC, comment #1: 

Please post output of "buildinfo" and "runinfo"

spiralvoice <spiralvoice>
Group administrator
Wed 06 Sep 2006 03:42:00 AM UTC, original submission:  

THIS ITEM WAS REASSIGNED TO PATCH #5547.

Hi
I found a bug that the charset is incorrect in some environment. For example, when the locale is set to "zh_CN" or "zh_CN.GBK" for Simplified Chinese, the mldonkey should use charset GB2312 or GBK charset to convert filenames, but it indeed using the BIG5 charset to convert filenames. This will result in filenames with incorrect characters displayed on webpages.
I found the root cause of this problem is in src/utils/lib/charset.ml:
The charsets list are always set to "BIG5, ..., GBK" when the language is "zh"
      | "SH"
      | "SR" -> li := central_european :: cyrillic ::!li
      | "ZH" -> li := chinese_traditional :: chinese_simplified :: !li
      | "BE"
But please note the charset BIG5 for zh_TW is totally different from GBK for zh_CN, it's not a good idea to assume charsets for same language should always be compatible with rests.
To workaround this problem, we can remove the chinese_traditional from above code, but it's bad for poeple who using Traditional Chinese.
My suggestion is to match the locale in ll_CC format, while not only the language tag. For example:
"zh_CN" -> li := chinese_simplified :: !li
"zh_TW" -> li := chinese_traditional :: !li

The normalize_language method should also be changed since it will trim the "_CC" in the locale string.

A better solution for this issue may be that try to use the encoding for current locale provided by glibc, such as nl_langinfo(), it will returns the charset for current locale, same as result of the command: locale charmap

Thanks



Gang Chen <gangchen>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 15 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2006-11-13 spiralvoice Reassign Itemmldonkey, a multi-networks file-sharing client, bug #17618 mldonkey, a multi-networks file-sharing client, patch #5547
2006-11-08 spiralvoice StatusIn Progress Ready For Test
    Attached File- Added lang_tw_cn.patch, #11202
2006-11-08 spiralvoice Attached File#10737 Removed
2006-09-12 spiralvoice StatusReady For Test In Progress
2006-09-12 spiralvoice Attached File- Added lang_tw_cn.patch, #10737
2006-09-12 spiralvoice Attached File#10729 Removed
2006-09-11 spiralvoice Attached File- Added lang_tw_cn.patch, #10729
2006-09-11 spiralvoice Attached File#10710 Removed
2006-09-07 spiralvoice Severity3 - Normal 4 - Important
    StatusNone Ready For Test
    ReleaseNone 2.7.7
    Operating SystemNone Linux
    CPU typeNone Intel x86
    Attached File- Added lang_tw_cn.patch, #10710

Back to the top

Powered by Savane 3.13-02a9.
Corresponding source code