patchmldonkey, a multi-networks file-sharing client - Patches: patch #4406, Improved Unicode filename support

 
 

You are not allowed to post comments on this tracker with your current authentication level.

patch #4406: Improved Unicode filename support

Submitter:  spiralvoice <spiralvoice>
Submitted:  Thu 08 Sep 2005 01:32:02 PM UTC
   
 
Category:  Improved interfaces Priority:  5 - Normal
Status:  Done Assigned to:  None
Open/Closed:  Closed

Jump to the original submission

Mon 21 Nov 2005 09:41:30 PM UTC, comment #10: 

Applied mld_unicode_patch6.patch 2005/11/21

spiralvoice <spiralvoice>
Group administrator
Sun 18 Sep 2005 12:54:39 AM UTC, comment #9: 

arg another thing: we have many functions that have an implementation for *nix and one for win (using the win32 api). i guess we need to check if any are among them that handle filenames.

we could also implement all our filename handling functions like that and not use the ocaml implementations. (but i think it might in the long run be better the other way around: merge ours with ocaml)

Amorphous <amorphous>
Sun 18 Sep 2005 12:48:17 AM UTC, comment #8: 

oh an addition: the patch should work as it is, if we use the patched ocaml (that expects filenames as utf-8) on win. it's just not a clean and future-proof solution (and thus will never get accepted into ocaml).

(it will also work on linux, because of the os check, but not on win with an unpatched ocaml.)

Amorphous <amorphous>
Sun 18 Sep 2005 12:41:04 AM UTC, comment #7: 

skip to ---HERE--- for the important things, if you know
enough about charset encoding and locale stuff and how it is
on win and linux.

Ok i checked it. a locale charset encoding is called
codepage on windows. we detect it correctly (at least for an
one common codpage [german win2003 displays: "locale:
CP1252" in buildinfo]).

win has all string functions as an ansi (fooA) and an wide
(fooW) version and an wrapper that uses one of them
depending on if UNICODE is defined (see
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/intl/unicode_28c3.asp
and it's siblings for this and more windows unicode info).
the wide versions use strings of the character type wchar_t
encoded as UCS-2LE (at least on i386 i think), libiconv has
an special encoding named wchar_t for this use (see
http://www.gnu.org/software/libiconv/ ). the ansi versions
are for use with strings that are encoded with the codepage
the system is set to.

on linux the filename functions just work with anything that
is thrown at them. (an filename can contain any
char/bytevalue except / on linux, even such things as \0 .)
and how we encode that is entierly an thing of the locales
character encoding. everyone should use the utf-8 encoding,
but there are still people using other encodeings and thus
we encode any filename on write to that locale (that is
.to_locale for) or on read from that locale (i don't know if
we do that currently).

so we either need use an unpatched ocaml (that uses the ansi
functions) and output filenames in the system codepage (but
normally that is something that can only represent an
limited amount of chars, much much less than unicode) or use
an patched ocaml that uses the widechar functions and output
filenames in an unicode encoding suitable for it. an
solution to

---HERE---
the problem with the current version of the unicode patch to
ocaml is that we don't know (in mldonkey code) if we have an
ocaml that uses ansi or widechar functions and thus can't
know if need to convert to the locale or to wchar_t.

another problem with the patch is, that it does an
additional convert between utf-8 and ucs-2le (where the
later should be wchar_t anyway, even tho that is most likely
an alias to ucs_2le on many windows systems). (not doing the
additional conversion would simplyfy the patch much more.
using the UNICODE define could perhaps reduce it further.
see the msdn link above for more.)

(another solution would be to set the codepage to utf-8 and
just use the ocaml with the ansi functions, but i don't know
if one can do that. windows as something like per thread
locales "CP_THREAD_ACP", but i don't know if/how that can be
set to utf-8)

Amorphous <amorphous>
Sat 17 Sep 2005 02:09:01 PM UTC, comment #6: 

I believe Win32 uses UTF-16 in encoding things like file names.

Anonymous
Sat 17 Sep 2005 01:05:28 PM UTC, comment #5: 

Charset.to_locale is needed, as the user may use some locale other than uft-8 . (i'm not sure how that is handeled on windows, but if windows allways uses utf-8 then we should detect the windows locale as that (then logically .to_locale will just output utf-8). i guess all nt versions use utf-8 for the filesystem/locale, right? but win9x might not support it.)

Anyway i guess the problem is somewhere else in the code. It seems we add the file to shares when commiting it (with the filename with whatever encoding the string had). And the second time when we check all shares and see the filename with the encoding we converted it to (with .to_locale).
Thus we also need to convert the filename with .to_locale in the first case.

Amorphous <amorphous>
Sat 17 Sep 2005 10:38:01 AM UTC, comment #4: 

Amorphous said, some other platforms may not support utf-8.

Li-Hui Zhou <enig123>
Sat 17 Sep 2005 09:29:14 AM UTC, comment #3: 

Why did you use

+  if Sys.os_type <> "Win32" then
+    Charset.to_locale (Buffer.contents buf)

Why not change it on all platforms?

spiralvoice <spiralvoice>
Group administrator
Sat 17 Sep 2005 01:15:30 AM UTC, comment #2: 

Fix Win32 new downloaded(non-english filename) to share cause in shared_files_new.ini
have two same entry but with diff filename, and web rename download problem

CML <cml>
Tue 13 Sep 2005 09:54:57 AM UTC, comment #1: 

Applied 2005/09/13

spiralvoice <spiralvoice>
Group administrator
Thu 08 Sep 2005 01:32:02 PM UTC, original submission:  

copied from unicode patch #5 by CML

spiralvoice <spiralvoice>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #8759:  mld_unicode_patch6.patch added by cml (2KiB - text/plain)
file #9051:  mld_unicode.patch added by spiralvoice (10KiB - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

CC list is empty

 

There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

 

Follow 8 latest changes.

Date Changed by Updated Field Previous Value => Replaced by
2005-11-21 spiralvoice StatusPostponed Done
    Open/ClosedOpen Closed
2005-09-17 amorphous StatusDone Postponed
2005-09-17 spiralvoice Open/ClosedClosed Open
2005-09-17 cml Attached File- Added mld_unicode_patch6.patch, #5184
2005-09-13 spiralvoice StatusNone Done
    Open/ClosedOpen Closed
2005-09-08 spiralvoice Attached File- Added mld_unicode.patch, #5159

Back to the top

Powered by Savane 3.13-bb6a.
Corresponding source code