bugvile, a vi-compatible text editor - Bugs: bug #63912, WinVile problems handling UTF-8,...

 
 

bug #63912: WinVile problems handling UTF-8, UTF-16LE, and UTF-16BE

Submitter:  None
Submitted:  Sat 11 Mar 2023 06:41:25 PM UTC
   
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  None
Privacy:  Public Assigned to:  None
Open/Closed:  Open
* Mandatory Fields

Add a New Comment Rich Markup
   

Wed 15 Mar 2023 12:20:28 AM UTC, comment #5: 

I see (in my quick check, only the first line of the UTF-8 file was there - the second was blank).  At the moment I'm just collecting information (am working on xterm...).  vile (like xterm) has debugging traces which I use to actually see the problem.  Here's a fragment from one with the UTF-8 file, to illustrate:

| TEXT    0:\u1d5f3\u1d5fc\u1d5fc\u1d5ef\u1d5ee\u1d5ff^M
| TEXT    1:\ud835\uddf3\ud835\uddfc\ud835\uddfc\ud835\uddef\ud835\uddee\ud835\uddff^M
| TEXT   39:[Use ^A-h, ^X-h, or :help to get help]
| TEXT   39:

so... given your examples, I'll have to pick it apart to see where it's wrong :-)

Offhand, that second line looks like surrogate pairs
(see https://www.compart.com/en/unicode/U+DDF3), which I did not implement in vile (yet).

I was asking about the font, since Consolas is okay for coverage, but the good ones are those from Office, and I haven't a need for Office.  So if your example is readable in Consolas, that's fine.  I'll debug it in xterm anyway, and port it back to Windows (since most of the relevant logic is portable).

The bytes in the UTF-8 example look different than your expected result:

> od -c -t x1 VILE_test_for_Unicode_paste_UTF_8_with_new_line.txt

0000000 360 235 227 263 360 235 227 274 360 235 227 274 360 235 227 257
         f0  9d  97  b3  f0  9d  97  bc  f0  9d  97  bc  f0  9d  97  af
0000020 360 235 227 256 360 235 227 277  \r  \n 355 240 265 355 267 263
         f0  9d  97  ae  f0  9d  97  bf  0d  0a  ed  a0  b5  ed  b7  b3
0000040 355 240 265 355 267 274 355 240 265 355 267 274 355 240 265 355
         ed  a0  b5  ed  b7  bc  ed  a0  b5  ed  b7  bc  ed  a0  b5  ed
0000060 267 257 355 240 265 355 267 256 355 240 265 355 267 277  \r  \n
         b7  af  ed  a0  b5  ed  b7  ae  ed  a0  b5  ed  b7  bf  0d  0a
0000100

and in sending the file to xterm, I only see the same foobar/blank-line as in the quick test for vile (xterm doesn't implement surrogate pairs either).  Implementing that in vile would be useful (agreed).  As I understand it, your example has the same Unicode encoded two different ways, and winvile would write the result in just one format.

Thomas Dickey <dickey>
Group administrator
Tue 14 Mar 2023 05:19:06 PM UTC, comment #4: 

comment #3:

> It looks like "current" is 9.8x -- the script which I use for updating the direct link from the upload (in the archive area) missed that (I see where to fix this).  The 9.8y change for map/insert is something you would want.

I'm currently using 9.8y. 9.8y was used for all testing mentioned in this bug report. I did have to go hunting it down in the appropriate directory after first re-downloading 9.8x. Fixing the "current" link is appreciated.

> In a quick check, changing VIDEO_TEXT and VIDEO_CHAR to unsigned (e.g., 32-bits) and testing in xterm (Debian 11), I can see the "foobar" in the UTF-8 example.  (The UTF-16 example doesn't display, but it's something to consider).

I'm unsurprised that the UTF-16 example doesn't display. WinVile corrupted the last character in the original file when it added a CR/LF and "foobar" was pasted after opening the original, which only contained "foobar" and no CR/LF. Basically, the current length of the file is 2 bytes less than it should be. If you look at the two bytes just prior to the first 00 0D 00 0A, those bytes are D8 35, but the full character should be D8 35 DD FF (and were such in the original file that WinVile opened). That sequence should exactly match the final "r" in the file, which WinVile does encode correctly.

Not accounting for the use of "Bold (sans)", the content of both/all test files should be:
foobar
foobar

Other than the UTF-8/UTF-16 encoding difference, the text content of both files is supposed to be the same (i.e. two identical lines of "foobar" in Unicode Bold (sans), each terminated with CR/LF). The differences between the actual contents and that expected content are examples of some of the issues describe in the original comment here.

For the UTF-16 file, the contents of the file when WinVile opened it was "foobar" without a CR/LF, because I'd noted the above corruption, which doesn't happen if the original file has a CR/LF. Unfortunately, hitting "o", pasting the second "foobar", and then saving resulted in the original "r" at the end of the first line being corrupted.

UTF-8 file:
As to the UTF-8 file, which has similar expected content (i.e. two lines of "foobar", each with a CR/LF), I'm a bit surprised that it looks OK. My more structured look at this issue started because a Python program reported that it couldn't even read a file edited by WinVile, due to the file containing invalid UTF-8 encodings. I'd assumed that WinVile would have produced similarly invalid UTF-8 for the test file, but I guess it must have ended up as not actually invalid, just with characters that are not understood, except by WinVile. The programs I'd tried viewing the files with, other than the hex editor, clearly tried to auto-correct and/or guess the format, but other than WinVile, were resorting to displaying at least the second line as bytes.

The UTF-8 file was produced the same way as the UTF-16 file with the difference being that it was saved as UTF-8 and that it had a CR/LF at the end of the first line. Once opened by WinVile, "o" was pressed and the Bold (sans) "foobar" was pasted into the second line, then saved.

The content I had hoped for would have been:
F0 9D 97 B3  F0 9D 97 BC  F0 9D 97 BC  F0 9D 97 AF
F0 9D 97 AE  F0 9D 97 BF  0D 0A F0 9D  97 B3 F0 9D
97 BC F0 9D  97 BC F0 9D  97 AF F0 9D  97 AE F0 9D
97 BF 0D 0A

which is just:
foobar
foobar

How the attached example UTF-8 file is displayed in other programs is either the first "foobar" followed by a line of garbage characters or all garbage characters, with the latter typically indicating the program reading it was unable to decode it and just displayed the bytes.

> Allowing for some issues where I may have passed a Unicode value in a short which wasn't typed as one of those, it "should" work, depending on the driver configuration.  Some of the drivers wouldn't work for this (the X11 bitmap fonts for example, and possibly the Windows console version), but X11 TrueType and WinVile probably.

That does, of course, complicate things. I don't know the internals well enough to offer suggestions. There's probably not going to be much that can be done in some instances, other than convert back and forth from whatever format the specific driver is expecting, which could be a large task.

> What font are you using in WinVile?  (As I recall it -- may have changed -- there was some limitation on which ones I could get Windows to acknowledge as a fixed-pitch font).

I've tried several different fixed-width fonts. There are a few installed on this machine which are intended for things other than displaying typical text, but a bit more than 10 that are at least marginally acceptable. I've generally used Consolas for testing, as it seems to have a reasonably complete selection of glyphs.

Anonymous
Mon 13 Mar 2023 11:11:02 PM UTC, comment #3: 

It looks like "current" is 9.8x -- the script which I use for updating the direct link from the upload (in the archive area) missed that (I see where to fix this).  The 9.8y change for map/insert is something you would want.

In a quick check, changing VIDEO_TEXT and VIDEO_CHAR to unsigned (e.g., 32-bits) and testing in xterm (Debian 11), I can see the "foobar" in the UTF-8 example.  (The UTF-16 example doesn't display, but it's something to consider).

Allowing for some issues where I may have passed a Unicode value in a short which wasn't typed as one of those, it "should" work, depending on the driver configuration.  Some of the drivers wouldn't work for this (the X11 bitmap fonts for example, and possibly the Windows console version), but X11 TrueType and WinVile probably.

What font are you using in WinVile?  (As I recall it -- may have changed -- there was some limitation on which ones I could get Windows to acknowledge as a fixed-pitch font).

Thomas Dickey <dickey>
Group administrator
Mon 13 Mar 2023 07:57:44 PM UTC, comment #2: 

Thank you for taking the time to look at this issue. Yes, using an unsigned short is almost certain to cause this type of issue.

I'm sorry for the problems with this bug report. I thought I had double-checked that the report here showed the characters, which are just "foobar" converted to "Bold (sans)" using this page. I am able to download the attached files, which do have the correct content when I re-download them. Based on the display in this page, it's possible the issue is a spaces in the filename issue, so I'll rename and try re-uploading.

However, if re-uploading doesn't work, the file contents are intentionally short, so can be shown here in hex.

VILE_test_for_Unicode_paste_UTF_16_BE.txt is:

FE FF D8 35 DD F3 D8 35 DD FC D8 35 DD FC D8 35
DD EF D8 35 DD EE D8 35 00 0D 00 0A D8 35 DD F3
D8 35 DD FC D8 35 DD FC D8 35 DD EF D8 35 DD EE
D8 35 DD FF


VILE_test_for_Unicode_paste_UTF_8_with_new_line.txt is:

F0 9D 97 B3 F0 9D 97 BC F0 9D 97 BC F0 9D 97 AF
F0 9D 97 AE F0 9D 97 BF 0D 0A ED A0 B5 ED B7 B3
ED A0 B5 ED B7 BC ED A0 B5 ED B7 BC ED A0 B5 ED
B7 AF ED A0 B5 ED B7 AE ED A0 B5 ED B7 BF 0D 0A

(file #54478, file #54479)

Anonymous
Sun 12 Mar 2023 11:23:27 PM UTC, comment #1: 

Most of this (perhaps all) is a known limitation from when I implemented the Unicode support in 2007.  That uses an unsigned short for the VIDEO_TEXT and VIDEO_CHAR types, which is an "of course" for 32-bit architectures.  For 64-bits, that's not essential, and could conceivably be altered (though just changing that would probably find some issues).

The example characters don't render in this webpage, and the attachments don't work.

Thomas Dickey <dickey>
Group administrator
Sat 11 Mar 2023 06:41:25 PM UTC, original submission:  

Current issues on Windows 10 using WinVile 9.8y: "winvile-wide-ole version 9.8y for win64, built Jan 15 2023 20:12:09". However, at least very similar issues have existed for quite some time (years).

I've been having a issues handling UTF encoded files with Vile for quite some time (years), but have not really explored it until now. While WinVile is able to read and write files which it exclusively writes and have the characters be the same when read back, the encoding used isn't interoperable with other programs that read UTF, particularly UTF-8, encoded files.

The errors when read back in other programs can be just seeing other characters, corrupting existing characters in the file, or failure to decode due to invalid UTF encoding (e.g. Python when reading back a UTF-8 encoded file with a few characters added by WinVile and then saved).

The errors when a UTF, particularly UTF-8, encoded file is read by WinVile is that the characters displayed are not those which were entered in any of the other programs I've used to generate or view the files. This happens for characters which have large code point values, usually exceeding 0xFFFF. Other than the problem is seen with such characters, I did not fully explore at what code point value the problems start happening.

The problems are exclusively when going to/from WinVile. The several other programs I've tried all display/write the files in a manner which allows them all to be used interchangeably, except WinVile.

Testing:
For entering non-ASCII characters into a WinVile window, I'm almost exclusively pasting them into the window.
Testing was on Windows 10 using WinVile 9.8y: winvile-wide-ole version 9.8y for win64, built Jan 15 2023 20:12:09 with Visual C++, which was downloaded from invisible-island.net, IIRC. All I specifically recall about downloading was that the /current/ links were pointing to an older version, so I had to look through the files and specifically select the 9.8y version.

Example characters: ????????????????????????
Note: the above characters are not made bold through Markup. They are Unicode characters which are bold in themselves.

The observed behavior is identical for UTF-16BE and UTF-16LE, so only UTF-16BE files are provided.

Files were created in the stock Windows Notepad and saved as the respective format. The text "????????????????????????" was pasted into Notepad and versions were saved as UTF-8, UTF-8 with BOM, UTF-16 LE, and UTF-16 BE, and both with and without a newline after the pasted text (accidentally done, but identified another type of corruption).

The files were then individually read into WinVile. If the display in WinVile didn't have a newline after the existing characters, then "o" was used to insert starting with a new line (Some of the base files which do, in fact, have a new line don't display the new line in WinVile. If there was a new line visible, then "i" was used to begin inserting. The above "????????????????????????" text was pasted into WinVile along with a return and the file written.

Display:

  1. These characters are displayed as if they are two separate characters in WinVile. Even when the character displayed is correct, there's an apparent space between each character. Deleting the "space" corrupts the character.


Using UTF-16 (both BE and LE):

  1. If there was no CR/LF at the end of the file, then WinVile truncated the last two bytes, corrupting the last of the original characters. Otherwise, the characters pasted into WinVile and save appear the same as those pasted in Notepad and can be read correctly with multiple different programs. The file "VILE, test for Unicode paste, UTF-16 BE.txt" is attached showing the corruption of the last character in the first line.
  2. When the above happens, the last character which was added by pasting into WinVile is also corrupted when read back into WinVile if a return was not added after the paste.


Using UTF-8 (with and without BOM):

  1. The characters displayed in WinVile when the original file is opened are displayed as characters other than what was pasted in Notepad. This is the case regardless of what value I use for "file-encoding".
  2. The "????????????????????????" characters pasted in WinVile and saved are not displayed properly when the file is read by other programs. Using a hex editor, it appears that WinVile is encoding the characters into pairs of 3-byte values instead of using a 4-byte value for each character.


The file "VILE, test for Unicode paste, UTF-8 with new line.txt" is attached which exhibits the above issues with UTF-8 files.

Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by dickey (Posted a comment)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 4 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2023-03-13 None Attached File- Added VILE_test_for_Unicode_paste_UTF_16_BE.txt, #54478
        Attached File- Added VILE_test_for_Unicode_paste_UTF_8_with_new_line.txt, #54479
    2023-03-11 None Attached File- Added VILE,@test@for@Unicode@paste,@UTF-16@BE.txt, #54473
        Attached File- Added VILE,@test@for@Unicode@paste,@UTF-8@with@new@line.txt, #54474

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code