bugGNU nano - Bugs: bug #50773, [Request] improve the handling of...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #50773: [Request] improve the handling of combining characters

Submitter:  Mike Frysinger <vapier>
Submitted:  Mon 10 Apr 2017 05:04:38 AM UTC
   
 
Severity:  1 - Wish Status:  Fixed
Assigned to:  bens Open/Closed:  Closed

Jump to the original submission

Thu 03 Dec 2020 11:15:13 AM UTC, comment #11: 

Released in nano-5.4.

Benno Schulenberg <bens>
Group administrator
Sun 22 Nov 2020 09:46:48 AM UTC, comment #10: 

Thanks for the feedback, Peter.  Yes, the combining character "jumping back" onto the preceding character is the weird and confusing behavior that prompted me to implement the better behavior.  (I myself don't use combining characters, and I can only enter them via M-V 0003xx, so for me personally it has never been an issue.)

I'm not sure, though, whether all zero-width characters should be treated in this new way: the ZWJ and ZWNJ, for example.  We'll see that when people start using the new release.

Benno Schulenberg <bens>
Group administrator
Sun 22 Nov 2020 06:39:16 AM UTC, comment #9: 

As someone who works routinely with zero-width characters in nano, I'd like to say that I love the patches, and they seem to work well.

If I could define a key shortcut to revert to the old behaviour, I don't think I would ever use it, because it is hard to know the order that the zero-width characters are in (in Thai there can be 2). And to remove the base (non-zero-width) character so the zero-width character gets re-superimposed on the previous non-zero-width character makes no sense to me.

Peter Passchier <pepa65>
Tue 17 Nov 2020 10:44:42 AM UTC, comment #8: 

Improved versions of the patches have been pushed to master, commit 687efd21 and dc907bfe.

Benno Schulenberg <bens>
Group administrator
Wed 11 Nov 2020 03:26:44 PM UTC, comment #7: 

A first iteration of an implementation is attached.

(file #50247, file #50248)

Benno Schulenberg <bens>
Group administrator
Thu 04 May 2017 08:43:38 AM UTC, comment #6: 


> GTK+ wipes out the entire grapheme, Qt only removes the last
> combining character. (Both remove the entire grapheme on Delete.)


Thanks for the information, Egmont.  So... what I am suggesting is that nano should behave like Qt.  Maybe later we'll add an option to make Backspace work like in GTK+.  (But even these combining characters themselves are not high on my list of things to do.)

Benno Schulenberg <bens>
Group administrator
Wed 03 May 2017 09:26:02 PM UTC, comment #5: 

Just FYI:

GTK+ and Qt behave differently on Backspace in their simplest one-line text entry fields (e.g. gnome-terminal or konsole -> profile preferences -> rename the profile).

GTK+ wipes out the entire grapheme, Qt only removes the last combining character. (Both remove the entire grapheme on Delete.)

Egmont Koblinger <egmont>
Tue 11 Apr 2017 08:33:52 PM UTC, comment #4: 

backspace-vs-delete sounds like a reasonable compromise.  hopefully the user's keyboard provides both :).

i don't feel strongly on the char counting.  if you do, that's fine.

wrt gnulib, i'm not familiar with their modules, and a dumb grep shows they have a bunch.  a search of the manual doesn't help narrow things down.  you should just e-mail bug-gnulib@, describe what you're looking for, and if gnulib has any APIs that might help.  they're a friendly bunch :).

Mike Frysinger <vapier>
Group Member
Tue 11 Apr 2017 08:54:37 AM UTC, comment #3: 

I've read a bit on the Unicode website and encountered this: "on a given system the backspace key might delete by code point, while the delete key may delete an entire cluster." [1]  I think that is the perfect compromise: deleting the entire grapheme when <Delete> is pressed avoids the suprprise of combining character(s) jumping to the preceding character, but still allows to remove combining characters individually by using <Backspace>.  So I don't think a config option is needed there.

For moving the cursor I don't want a config option either: it should always move per grapheme.  And the character count should continue to simply count codepoints, so that the user can realize that there is a compound character somewhere when the cursor moves just one position and the character count changes by two or more.  The man page will have to explain this behavior.

So... all we now need is some complicated logic to make <Left> and <Right> move per grapheme instead of per codepoint.  Surely gnulib has a function to help with this, no?  Something like is_combining(char) or get_combining_class(char)?  :)

[1] http://www.unicode.org/reports/tr29/ -- section 3, fourth paragraph

Benno Schulenberg <bens>
Group administrator
Mon 10 Apr 2017 08:09:21 PM UTC, comment #2: 

i think either behavior wrt deletion is defensible.  i don't feel too strongly either way since i don't use/encounter them normally, so i can't speak to common workflows.  being able to delete combining characters like emacs does seem like a reasonable position, but so does deleting an entire grapheme.  maybe time for a config option ? :)

i agree that the chars field is, generally speaking, not an exact field one would care about (where as the cols field gets way more usage).  personally i use it as a rough equiv to byte count rather than character (codepoints or graphemes) count.  if nano had the ability to "move" over codepoints w/out updating the cursor, counting codepoints would be ideal as it would be clear that the cursor "moved" in the datastream even if visually it didn't move (row/col position is unchanged).

so if we had a config option that allowed users to specify which they cared about more (codepoints or graphemes), i think answers to all of this would fall out naturally ?

Mike Frysinger <vapier>
Group Member
Mon 10 Apr 2017 03:16:55 PM UTC, comment #1: 

When I open a file that contains just "0ax̊bc" (without the quotes and without a newline) with vim, it says "1L, 7C".  Seven characters!  That's even worse than nano, which correctly identifies six codepoints, six individual characters.

Anyway:  1) Yes, the "stutter" when the cursor moves over a combining character is confusing and annoying.  Although joe and Geany behave the same way as nano currently does, vim and gedit and emacs don't, and I agree it would be better if nano behaved like the latter three: treat a character plus its subsequent combining characters as a single entity when moving the cursor around.

(However, emacs is clever here: when you put the cursor on the "b" and type <Backspace>, it will delete just the combining-ring-above.  And if you put the cursor on the "x" and type <Delete>, only the x disappears and the combining-ring-above moves to the "a".  This may seem weird, but if a user knows about combining characters, she most probably is able to type them too, and then it would be strange that one can type such characters separately but not delete them separately.  I think nano should behave like emacs in this case.)

2) Don't know what I would prefer nano to count: codepoints or graphemes...  How often will a user be interested in exactly how many visual characters her file contains?  What I can imagine is that someone wants their lines to be at most a certain number of columns, but who does an M-D and wants their file to be an exact number of characters?  Also: nano counts newlines as characters, and I am certain that normal users don't count an end-of-line as a character.  So... when newlines are counted, then it is most logical to count also combining characters separately.

Benno Schulenberg <bens>
Group administrator
Mon 10 Apr 2017 05:04:38 AM UTC, original submission:  

consider this content:
$ printf '0ax\U0000030Abc\n'
0ax̊bc
there's a combining character (U+030A Combining Ring Above) after the x.

nano shows this status:
[ line 1/2 (50%), col 0/6 (16%), char 0/7 (0%) ]

there's a few weird things here:
- when moving the cursor over the combined character, it stops and lets you delete the combining character, except the position renders after the char (over the "b").  if you put in multiple combining chars, it does the same but more so.
- the chars field is arguably correct ... it's showing codepoints.  but it's also arguably incorrect ... users don't count distinct codepoints, they could graphemes.

comparing it to other programs (like vim and Chrome), they only operate on graphemes.  that means the cursor moves over x̊ in one step, deletes it with one keystroke, etc...  i think nano should do the same.

Mike Frysinger <vapier>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by pepa65 (Posted a comment)
  • -email is unavailable- added by egmont (Posted a comment)
  • -email is unavailable- added by bens (Posted a comment)
  • -email is unavailable- added by vapier (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 8 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2020-12-03 bens Open/ClosedOpen Closed
    2020-11-17 bens StatusReady For Test Fixed
    2020-11-11 bens Attached File- Added 0001-moving-skip-combining-characters-and-other-zero-widt.patch, #50247
        Attached File- Added 0002-prompt-skip-over-combining-characters-also-when-edit.patch, #50248
        StatusNone Ready For Test
        Assigned toNone bens
    2019-05-27 bens Severity3 - Normal 1 - Wish
        Summaryhandling of combining characters [Request] improve the handling of combining characters

    Back to the top

    Powered by Savane 3.13-4448.
    Corresponding source code