bugGNU Octave - Bugs: bug #53842, Handle m-files with UTF-16 and...

 
 

bug #53842: Handle m-files with UTF-16 and UTF-32 character encoding

Submitter:  Markus Mützel <mmuetzel>
Submitted:  Sat 05 May 2018 06:12:02 PM UTC
   
 
Category:  Interpreter Severity:  3 - Normal
Priority:  5 - Normal Item Group:  Incorrect Result
Status:  Confirmed Assigned to:  None
Originator Name:  Open/Closed:  * Open
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Tue 05 Mar 2019 05:21:18 AM UTC, comment #16: 


>  I personally rarely stumbled upon UTF-16 encoded files myself apart from some Windows system files (and never knowingly upon UTF-32 encoded files)


Same here. I don't think I've ever seen UTF-16 or UTF-32 source code files in the wild. I think we should defer UTF-16 and UTF-32 support until the broader source code encoding issues are sorted out, like: bug #49685, bug #35910, bug #50409, and bug #55826.


Andrew Janke <apjanke>
Fri 03 Aug 2018 07:10:50 PM UTC, comment #15: 

Re-titling to reflect that we still can't use encodings which don't consist of a sequence of chars.

Markus Mützel <mmuetzel>
Group administrator
Wed 27 Jun 2018 10:25:53 AM UTC, comment #14: 

@Andrew: As a first step, being able to read UTF-16 (or at least UCS-2) and UTF-32 encoded files at all would be an improvement. We could worry later about some heuristics to detect which flavour of UTF is actually used.
To be honest, I personally rarely stumbled upon UTF-16 encoded files myself apart from some Windows system files (and never knowingly upon UTF-32 encoded files).
But if you want to start working on that, I could try and give some feedback.

Markus Mützel <mmuetzel>
Group administrator
Mon 25 Jun 2018 12:11:35 PM UTC, comment #13: 


>  UTF-8 is an encoding that covers all Unicode characters. What do you mean by changing the default encoding from "utf-8" to "unicode"?


I mean, make the default external file encoding not strictly UTF-8, but a "meta-encoding" which means "Unicode, but choose/detect the actual UTF-whatever encoding on a per-file basis". So that you might have a project that has some .m files in UTF-8, and some files in UTF-16, and these encodings are autodetected when the individual files are read, based on their contents. And for output, previously-existing files are saved in their original encodings (to avoid introducing spurious changes into source control or whatever), and newly-created files are saved in a default Unicode encoding, probably UTF-8.

> Wrt your comment #10: Unfortunately(?) the C++ standard doesn't define any character encodings. To make matters worse wchar_t and wstring have different sizes on different platforms...


That's my understanding as well. And, yeah, "Unfortunately(?)" is right. It's a bummer that there's not standard support in base C/C++, but this Unicode stuff is so complex that whatever made it out of a C++ standards committee would probably be woefully outdated by the time it hit the streets.

What I really mean is:

  • Define an EncodedFileReader class that:
    • detects file encodings using some heuristics, or takes an encoding explicitly supplied by the caller, and then
    • does line-oriented encoding-aware input reading (based on the initial encoding detection/specification) and encoding conversion, returning the results as UTF-8 strings inside `std::string` (or whatever Octave is using for its internal Unicode string representation). So the external encoding could be UTF-16, UTF-8, UTF-32, or whatever, as long as the Unicode library supports translation between that and the native Octave representation, and line feeds could be safely detected
      • (which maybe means slurping the entire input file in to memory, transcoding the entire thing at once, and then detecting line feeds. But that's probably not an issue because source code files are relatively small.)


My understanding is that Octave is currently using gnulib for Unicode support, and that gnulib provides encoding translations, but not encoding/character-set detection. So we'd have to roll our own encoding detection, or take a dependency on a library that supports it.

> While C++ 11 added u16string and u32string, I don't know of any iostreams that support these types out of the box.


I don't know of any standard C/C++ stuff that supports this either. AFAIK, base C/C++ Unicode support is not great.

So that means rolling our own encoding-detection and Unicode-aware iostream support (which might not be so bad if all you care about is UTF-8 + UTF-16 and BMP-only support), or taking a dependency on ICU4C or similar, or deferring this to later.

Andrew Janke <apjanke>
Mon 25 Jun 2018 11:47:19 AM UTC, comment #12: 

@Andrew: The first part of your comment #11 is bug #49685 which is on hold to keep compatibility with Matlab.

I don't understand the last part of comment #11: UTF-8 is an encoding that covers all Unicode characters. What do you mean by changing the default encoding from "utf-8" to "unicode"?

Wrt your comment #10: Unfortunately(?) the C++ standard doesn't define any character encodings. To make matters worse wchar_t and wstring have different sizes on different platforms. So a compatible character encoding could be either UTF-16 or UTF-32 (not both). While C++ 11 added u16string and u32string, I don't know of any iostreams that support these types out of the box.

Markus Mützel <mmuetzel>
Group administrator
Mon 25 Jun 2018 02:46:57 AM UTC, comment #11: 

I'm wondering if this feature may actually be detrimental to the Octave user community, because it would allow (and maybe encourage) users to create M-code that is incompatible with M-code in other encodings.

Let's say I'm collaborating on a project with someone in Latvia and someone in China, and each of them have their systems set up to use their local non-Unicode encodings. I don't think there's a way for me to configure my Octave system in a way that the source code from both of them could be used in a single Octave session, if the source file encoding is a global Octave setting.

Perhaps it would be better to require that all Octave M-code source files be in Unicode encodings, or at least make that the default on all operating systems. That way all Octave .m code bases would be interoperable with all other Octave .m code bases, at least by default.

Using the system default code page would require users to be aware of the interoperability issue, and take explicit action to create their source code in Unicode format. That seems like an advanced operation that normal Octave users shouldn't have to be aware of.

Along the same lines, perhaps this:

> static std::string Vmfile_encoding = "utf-8";


should be a more generic "unicode" encoding, like this:

> static std::string Vmfile_encoding = "unicode";


to allow for easier integration of future support for autodetected UTF-16 and UTF-32 files?

Andrew Janke <apjanke>
Mon 25 Jun 2018 01:57:09 AM UTC, comment #10: 


> Even if we solved the strlen issue, it doesn't seem to be easy to read UTF-16 or UTF-32 files with std::fgets (e.g. stops reading at single byte \n which could be part of a valid 2-byte or 4-byte character).
> Does anyone have an idea what we could do?


I think you'd need to sniff the BOM at the beginning of the file when first opening it, and switch to using `std::wifstream` or `std::fgetws` for UTF-16 and UTF-32 encoded files, converting the strings to UTF-8 as they are pulled from the input.

I think that implies replacing the whole `octave_gets` use with a file input object that abstracts away the file encoding and internally switches between `std::fgets` and `std::fgetws`, since `std::FILE *` does not carry character encoding info (I think).

Andrew Janke <apjanke>
Sat 12 May 2018 09:07:20 PM UTC, comment #9: 
Markus Mützel <mmuetzel>
Group administrator
Fri 11 May 2018 08:04:11 PM UTC, comment #8: 

Change looks good, I think you can push that small fix.

I think it's fine to defer handling of UTF-16 and UTF-32 encoded files until the future, and either document or not the lack of support for now.

See also bug #53786.

Mike Miller <mtmiller>
Group Member
Thu 10 May 2018 07:08:59 PM UTC, comment #7: 

Mike: You are right. UTF-16 files aren't read correctly.
The problem seems to be with octave_fgets which reads the file as an array of bytes (using char) and uses strlen to determine the string length. That means for most files using UTF-16 it reads only the first (half) character.

Even if we solved the strlen issue, it doesn't seem to be easy to read UTF-16 or UTF-32 files with std::fgets (e.g. stops reading at single byte \n which could be part of a valid 2-byte or 4-byte character).
Does anyone have an idea what we could do? Besides maybe document that we are not able to read .m files with these encodings and emit an error when trying to set them as the mfile_encoding.

Once we should be able to read the file, the conversion to UTF-8 should be working out-of-the-box.

The attached set fixes the second issue you mentioned.


(file #44137)

Markus Mützel <mmuetzel>
Group administrator
Thu 10 May 2018 01:27:19 AM UTC, comment #6: 

This works for me on Debian with an m-file encoded in iso8859-1 or latin1.

This does not work for me on Debian with an m-file encoded in utf-16. Should that be possible with this change so far?

Also, I think functions that take a string argument setting usually return the previous value, not the new value. So for example


>> enc = __mfile_encoding__ ()
enc = utf-8
>> enc = __mfile_encoding__ ('utf-16')
enc = utf-16


both of these return values should be 'utf-8'.

Mike Miller <mtmiller>
Group Member
Wed 09 May 2018 07:27:46 PM UTC, comment #5: 

I pushed the changeset here:
http://hg.savannah.gnu.org/hgweb/octave/rev/82445187633e

Marking as ready for test.

Markus Mützel <mmuetzel>
Group administrator
Wed 09 May 2018 06:12:20 PM UTC, comment #4: 

With this patch, the default m-file encoding is "system" on Windows platforms. That is the same as the default for the file editor.

(file #44132)

Markus Mützel <mmuetzel>
Group administrator
Tue 08 May 2018 12:25:53 PM UTC, comment #3: 

The previous patches set the wrong default on Windows platforms. It should be the system codepage on Windows and utf-8 otherwise.
I'll update the patch as soon as I have something ready.

Markus Mützel <mmuetzel>
Group administrator
Sun 06 May 2018 08:15:21 PM UTC, comment #2: 

The attached modified patch avoids including the gnulib header in the wrapper.h file. This is done by using "void *" instead of "iconv_t" as proposed by jwe on IRC.

(file #44106)

Markus Mützel <mmuetzel>
Group administrator
Sat 05 May 2018 06:25:54 PM UTC, comment #1: 

The attached file adds support for converting .m files with any character encoding to UTF-8 before passing its content to the lexer. The assumed .m file encoding can be changes with _mfile_encoding_.

I'll also attach "test_iso_8859_15.m" which is encoded with ISO-8859-15 (Western). With it I see the following results in an Octave session starting with the default .m file encoding "utf-8":

>> test_iso_8859_15
ans = W�hrend Adam lacht, jagen zw�lf Boxk�mpfer Eva quer �ber den gro�en Sylter Deich - f�r satte 12.345.667,89 �uro
>> __mfile_encoding__
ans = utf-8
>> __mfile_encoding__ ("iso-8859-15")
ans = iso-8859-15
>> clear test_iso_8859_15
>> test_iso_8859_15
ans = Während Adam lacht, jagen zwölf Boxkämpfer Eva quer über den großen Sylter Deich - für satte 12.345.667,89 €uro


While all non-ASCII characters are initially displayed with the replacement character, they are correctly read and displayed after the character encoding was set accordingly.

Markus Mützel <mmuetzel>
Group administrator
Sat 05 May 2018 06:12:02 PM UTC, original submission:  

As discussed on the mailing list [1], Octave isn't aware of character encoding yet.
This bug will track the part of reading m-files with arbitrary encoding.
I will post a patch as soon as I have a bug number.

[1] http://octave.1599824.n4.nabble.com/Handle-encoding-of-Octave-strings-td4687603.html

Markus Mützel <mmuetzel>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #44137:  bug53842_previous_setting.patch added by mmuetzel (939B - application/octet-stream)
file #44132:  bug53842_mfile_encoding_v3.patch added by mmuetzel (7KiB - application/octet-stream)
file #44106:  bug53842_mfile_encoding_v2.patch added by mmuetzel (6KiB - application/octet-stream)

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by apjanke (Posted a comment)
  • -email is unavailable- added by mmuetzel (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 11 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-03-05 mtmiller Carbon-CopyRemoved 80942 -
    2018-08-03 mmuetzel StatusReady For Test Confirmed
        SummaryHandle m-files with arbitrary character encoding Handle m-files with UTF-16 and UTF-32 character encoding
    2018-05-10 mmuetzel Attached File- Added bug53842_previous_setting.patch, #44137
    2018-05-09 mmuetzel StatusPatch Submitted Ready For Test
    2018-05-09 mmuetzel Attached File- Added bug53842_mfile_encoding_v3.patch, #44132
        StatusIn Progress Patch Submitted
    2018-05-08 mmuetzel StatusPatch Submitted In Progress
    2018-05-06 mmuetzel Attached File- Added bug53842_mfile_encoding_v2.patch, #44106
        StatusNone Patch Submitted
    2018-05-06 mmuetzel Dependencies- bugs #53646 is dependent

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code