bugmldonkey, a multi-networks file-sharing client - Bugs: bug #17618, Reassigned to another tracker...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #17618: Reassigned to another tracker [was: Incorrect charset used for some locales]

Submitted by:  Gang Chen <gangchen>
Submitted on:  Wed 06 Sep 2006 03:42:00 AM UTC  
 
Category: HTTP interfaceSeverity: 4 - Important
Item Group: i18n issuesStatus: Ready For Test
Assigned to: NoneOpen/Closed: Closed
Release: Release: 2.7.7
Operating System: LinuxBinaries Origin: CVS / Self compiled
CPU type: Intel x86

(Jump to the original submission Jump to the original submission)

Mon 13 Nov 2006 10:13:30 AM UTC, comment #14:

THIS ITEM WAS REASSIGNED TO PATCH #5547.

Please, do not post any new comments to this item.


spiralvoice <spiralvoice>
Project Administrator
Mon 13 Nov 2006 05:54:31 AM UTC, comment #13:

The patch works fine for me, thanks.

Gang Chen <gangchen>
Wed 08 Nov 2006 09:30:41 PM UTC, comment #12:

Updated patch for current CVS, please test and report back if it works for you

spiralvoice <spiralvoice>
Project Administrator
Sat 07 Oct 2006 03:40:57 PM UTC, comment #11:

ping

spiralvoice <spiralvoice>
Project Administrator
Thu 14 Sep 2006 05:42:10 PM UTC, comment #10:

Applied the language detection to CVS, for the display problems
I will leave this bug open.

spiralvoice <spiralvoice>
Project Administrator
Tue 12 Sep 2006 10:37:40 PM UTC, comment #9:

To fix conversion problems enclose the string variable, which
currently outputs bad chars, with Charset.to_utf8 ( )

spiralvoice <spiralvoice>
Project Administrator
Tue 12 Sep 2006 06:45:08 PM UTC, comment #8:

I updated two places of code with an UTF8-conv function
for the popups. Please test if this is enough or if
there are other places where incorrect chars are displayed.

Are search result filenames displayed well?

spiralvoice <spiralvoice>
Project Administrator
Tue 12 Sep 2006 09:01:49 AM UTC, comment #7:

The problem about tip is the filename are not converted to UTF-8 encoding properly when it displayed in the tip window.
For example, in the src/daemon/driver/driverCommands.ml:

(Http_server.html_real_escaped (Filename.basename impl.impl_shared_codedname))
...
(Filename.basename impl.impl_shared_codedname)

The codedname may not in UTF-8 if the system default encoding is not UTF-8, a conversion is needed before print out the filename. I found the "shorten" method did the conversion properly, so it will be a workaround by calling shorten for filename before write to buffer.
For example:
(shorten (Http_server.html_real_escaped (Filename.basename impl.impl_shared_codedname)) 999)
...
(shorten (Filename.basename impl.impl_shared_codedname) !!max_name_len)

Gang Chen <gangchen>
Tue 12 Sep 2006 07:09:22 AM UTC, comment #6:

Hi
I verified that your fix can resolve the problem, thank you very much.
While I found another encoding issue in the tips window when mouse move over the filename. Shall I open a new bug for that one?

Thanks

Gang Chen <gangchen>
Mon 11 Sep 2006 08:15:31 AM UTC, comment #5:

Please test the updated patch, it converts all string to uppercase before analyzing them.

spiralvoice <spiralvoice>
Project Administrator
Mon 11 Sep 2006 03:39:06 AM UTC, comment #4:

I have tested your patch but unfortunately it doesn't works.
The problem is caused by following line:
if String.sub s 0 2 = "ZH" && String.length s > 4 then
The "ZH" is used to match the locale but the locale string
starts with lowercase in most case. It will always fall into
default when the locale starts with language id in lowercase.

Since it's a convention that locale starts with lowercase,
I suggest to always use the lowercase for language id.

Gang Chen <gangchen>
Thu 07 Sep 2006 11:38:06 AM UTC, comment #3:

"The normalize_language method should also be changed since it will trim the "_CC" in the locale string."

Example: for the German language these values exist:

de_DE (Germany)
de_AT (Austria)
de_CH (Switzerland)
de_LU (Luxemburg)
de_BE (Belgium - has German minority)

for all of them using "de" cut by the normalize function is perfectly ok, otherwise charset_list_from_language would be a
very long function.

In the case of "zh_CN" and "zh_TW" its a different story,
I changed some code so zh_CN and zh_TW are specially treated.

Please try the attached patch and report if it works for you.

spiralvoice <spiralvoice>
Project Administrator
Thu 07 Sep 2006 06:15:50 AM UTC, comment #2:

--Buildinfo--
Version: MLNet Multi-Network p2p client version 2.7.7
Networks: Global Shares Gnutella G2 FileTP BitTorrent Donkey (SUI)
Ocaml version: 3.09.2 - C compiler version: 4.1.2 - C++ compiler version: 4.1.2
Build on: Linux i686 2.6.17.11-p4 (little endian) with glibc 2.3.6
Configure args: /usr/local/share/config.site /usr/local/etc/config.site
Features: threads zlib-1.2.3 bzip2-1.0.3 gd(jpg/png-1.2.8) iconv no-magic no-check-bounds

--Runinfo--
User: admin (PW Protected) - uptime: 1d 37m 50s
Enabled nets: Donkey Overnet BitTorrent FileTP
Server usage: enabled
Geoip: enabled, GeoLite data created by MaxMind, available from http://maxmind.com/
IP blocking: no blocking list loaded
System info: Linux lily 2.6.17.11-c3 #1 Mon Sep 4 22:53:39 CST 2006 i686
language: ZH - locale: GB2312 - UTC offset: +0800
max_string_length: 16777211 - word_size: 32 - max_array_length: 4194303
max file descriptors: 1024 - max useable file size: 2^63-1 bits (do the maths ;-p)

NOTE:
This is built by myself from the debian patched source with the workaround.

Gang Chen <gangchen>
Wed 06 Sep 2006 12:29:47 PM UTC, comment #1:

Please post output of "buildinfo" and "runinfo"

spiralvoice <spiralvoice>
Project Administrator
Wed 06 Sep 2006 03:42:00 AM UTC, original submission:

THIS ITEM WAS REASSIGNED TO PATCH #5547.

Hi
I found a bug that the charset is incorrect in some environment. For example, when the locale is set to "zh_CN" or "zh_CN.GBK" for Simplified Chinese, the mldonkey should use charset GB2312 or GBK charset to convert filenames, but it indeed using the BIG5 charset to convert filenames. This will result in filenames with incorrect characters displayed on webpages.
I found the root cause of this problem is in src/utils/lib/charset.ml:
The charsets list are always set to "BIG5, ..., GBK" when the language is "zh"
| "SH"
| "SR" -> li := central_european :: cyrillic ::!li
| "ZH" -> li := chinese_traditional :: chinese_simplified :: !li
| "BE"
But please note the charset BIG5 for zh_TW is totally different from GBK for zh_CN, it's not a good idea to assume charsets for same language should always be compatible with rests.
To workaround this problem, we can remove the chinese_traditional from above code, but it's bad for poeple who using Traditional Chinese.
My suggestion is to match the locale in ll_CC format, while not only the language tag. For example:
"zh_CN" -> li := chinese_simplified :: !li
"zh_TW" -> li := chinese_traditional :: !li

The normalize_language method should also be changed since it will trim the "_CC" in the locale string.

A better solution for this issue may be that try to use the encoding for current locale provided by glibc, such as nl_langinfo(), it will returns the charset for current locale, same as result of the command: locale charmap

Thanks

Gang Chen <gangchen>

 

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

Do you think this task is very important?
If so, you can click here to add your encouragement to it.
This task has 0 encouragements so far.

Only logged-in users can vote.

 

Please enter the title of George Orwell's famous dystopian book (it's a date):

 

 

Follow 15 latest changes.

Date Changed By Updated Field Previous Value => Replaced By
Mon 13 Nov 2006 10:13:30 AM UTCspiralvoiceReassign Itemmldonkey, a multi-networks file-sharing client, bug #17618=>mldonkey, a multi-networks file-sharing client, patch #5547
Wed 08 Nov 2006 09:30:41 PM UTCspiralvoiceStatusIn Progress=>Ready For Test
  Attached File-=>Added lang_tw_cn.patch, #11202
Wed 08 Nov 2006 09:30:16 PM UTCspiralvoiceAttached File#10737=>Removed
Tue 12 Sep 2006 10:37:40 PM UTCspiralvoiceStatusReady For Test=>In Progress
Tue 12 Sep 2006 06:45:08 PM UTCspiralvoiceAttached File-=>Added lang_tw_cn.patch, #10737
Tue 12 Sep 2006 06:43:47 PM UTCspiralvoiceAttached File#10729=>Removed
Mon 11 Sep 2006 08:15:31 AM UTCspiralvoiceAttached File-=>Added lang_tw_cn.patch, #10729
Mon 11 Sep 2006 08:14:25 AM UTCspiralvoiceAttached File#10710=>Removed
Thu 07 Sep 2006 11:38:06 AM UTCspiralvoiceSeverity3 - Normal=>4 - Important
  StatusNone=>Ready For Test
  ReleaseNone=>2.7.7
  Operating SystemNone=>Linux
  CPU typeNone=>Intel x86
  Attached File-=>Added lang_tw_cn.patch, #10710

Back to the top


Powered by Savane 3.1-cleanup1