bugm17n - Bugs: bug #46399, Combining Hebrew diacritics do not...

 
 

bug #46399: Combining Hebrew diacritics do not work in some instances

Submitted by:  Benjamin Riefenstahl <cc_benny>
Submitted on:  Sat 07 Nov 2015 07:12:31 PM UTC  
 
Category:  None Severity:  3 - Normal
Item Group:  None Status:  Fixed
Privacy:  Public Assigned to:  Kenichi Handa <handa>
Open/Closed:  Open

Add a New Comment (Rich Markup)
   

You are not logged in

Please log in, so followups can be emailed to you.

 

( Jump to the original submission)

Sun 29 Nov 2015 03:39:08 PM UTC, comment #25: 

Right, sounds like a plan ;-)

Thanks for your support so far.

Benjamin Riefenstahl <cc_benny>
Sun 29 Nov 2015 02:17:38 PM UTC, comment #24: 

> We can close it, but I'd like Mark to verify that he does not see
> any obvious problems at this point.


Ok.  Let's close it (and this bug itself) after his verification.

> What I still like to do is:
>   Align the categories in the FLT better with what the SBL font documentation suggests.
>   Check some "interesting" arrangements of marks that I have seen in another project.
>   Check-out other fonts.
>   Check-out which mark sequences (generalized by category) actually occur in the bible texts that are available on the net.


I see.  But, please open new items for them when you are ready to
discuss on them.

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Sat 28 Nov 2015 07:48:02 PM UTC, comment #23: 

Kenichi Handa writes:

> I've just committed that version.


Thanks.

> If you have any question about FLT, please feel free to ask.


Will do, thank you.

> By the way, do you think "Emacs bug #21846" can be closed now?  Or,
> is it better to keep it opened until all your works are done?


We can close it, but I'd like Mark to verify that he does not see any
obvious problems at this point.

What I still like to do is:

  • Align the categories in the FLT better with what the SBL font

  documentation suggests.

  • Check some "interesting" arrangements of marks that I have seen in

  another project.

  • Check-out other fonts.
  • Check-out which mark sequences (generalized by category) actually

  occur in the bible texts that are available on the net.

Benjamin Riefenstahl <cc_benny>
Wed 25 Nov 2015 03:04:48 PM UTC, comment #22: 

> Appending it.


Thank you.  I've just committed that version.

> Note, that while this is already an improvement, I'd like like to work on it further.


Ok.  If you have any question about FLT, please feel free to ask.

By the way, do you think "Emacs bug #21846" can be closed now?  Or, is it better to keep it opened until all your works are done?

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Tue 24 Nov 2015 09:30:06 PM UTC, comment #21: 

Appending it.  Note, that while this is already an improvement, I'd
like like to work on it further.

(file #35538)

Benjamin Riefenstahl <cc_benny>
Mon 23 Nov 2015 03:01:24 AM UTC, comment #20: 

> Got it. It seems to fix the general problem with the displacement.


Thank you for confirming that.

> I extended this to also cover the general case, not just the SHIN
> case, and that seems to fix the problems with the FreeMono font.


Please upload that version.  I'll commit it.

> It looks like in general whenever these Hebrew fonts rely on the
> general positioning instead of the mark feature, they assume this
> ordering.


Currently FLT code does not have a facility to check the left-bearing
and right-bearing of a combining mark glyph.  If it has, we can decide
whether we should re-order glyphs or not depending on that
information... though it's a very ad-hoc solution.

> Maybe this just happens automatically on MS Windows as a result of
> bidi reordering there.


OTF spec says that combining marks should come after base glyphs even for right-to-left scripts.  Perhaps, uniscribe and harfbuzz adopt the ad-hoc solution written above.

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Sun 22 Nov 2015 10:05:18 PM UTC, comment #19: 

Got it.  It seems to fix the general problem with the displacement.
Interesting construction ;-).

I extended this to also cover the general case, not just the SHIN
case, and that seems to fix the problems with the FreeMono font.  It
looks like in general whenever these Hebrew fonts rely on the general
positioning instead of the mark feature, they assume this ordering.
Maybe this just happens automatically on MS Windows as a result of
bidi reordering there.

> By the way, when you set the env. var. MDEBUG_FLT to 3, m17n-lib
> prints various debugging information about layouting. It worth
> trying something like this: % MDEBUG_FLT=3 m17n-edit ...


That's usefull, thanks.

Benjamin Riefenstahl <cc_benny>
Sat 21 Nov 2015 03:05:24 AM UTC, comment #18: 

> It seems you uploaded the wrong file? The one you attached is the unchanged
> HEBR.flt, the one I changed and that you probably have your changes in is
> HEBR-OTF.flt.


Oops, sorry, I've just attached the correct one.

> Also, would you have an idea what may be the problem with FreeMono and
> how we can fix that?


I have not yet cheched the OTF tables in FreeMono.  I need some more time.

By the way, when you set the env. var. MDEBUG_FLT to 3, m17n-lib prints various debugging information about layouting.  It worth trying something like this:
% MDEBUG_FLT=3 m17n-edit ...

(file #35511)

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Fri 20 Nov 2015 08:08:59 PM UTC, comment #17: 

> So I attached a new version which has an extra stage to handle
> combining marks on SHIN.


It seems you uploaded the wrong file?  The one you attached is the unchanged
HEBR.flt, the one I changed and that you probably have your changes in is
HEBR-OTF.flt.

Also, would you have an idea what may be the problem with FreeMono and
how we can fix that?

Benjamin Riefenstahl <cc_benny>
Tue 17 Nov 2015 04:40:32 PM UTC, comment #16: 

> I append a new version of the patch that implements the suggestion.


Thank you for the patch.  I'll commit it soon.

> With FreeSerif (regular), it does not fix the problems with SHIN +
> SHIN/SIN DOT. This is not an ordering problem though, FreeSerif does
> not work in Emacs with SHIN + SHIN/SIN DOT even without a vowel or
> when I reorder the diacritics manually. Needless to say that Gedit
> and Harfbuzz show no problem. One difference I see between FreeSerif
> Bold and FreeSerif regular is that according to Harfbuzz' tool
> hb-shape, FreeSerif Bold generates a precomposed glyph while FreeSerif
> regular generates two glyphs.


I check the gpos table of FreeSerif.ttf.  It surely has "mark" feature for "hebr" script, but the correponding lookup table does not handle any mark on U+05E9 (SHIN).  So I attached a new version which has an extra stage to handle combining marks on SHIN.

Could you please try it?

(file #35481)

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Mon 16 Nov 2015 03:36:20 AM UTC, comment #15: 

FWIW, I see no problems with SHIN + SHIN DOT in Emacs on MS-Windows.

Eli Zaretskii <eliz>
Sun 15 Nov 2015 10:19:38 PM UTC, comment #14: 

> Regarding heb.patch, the addition
>
> + (0x05F0 0x05F2 ?C) ; yiddish ligatures
>
> should be changed to not include 0x05F1, since nikudes
> should not be combined with vov-yud.


While that is correct in principle, if we do not include U+05F1,
I would want a comment as to why and that comment alone makes it
more hassle than it's worth IMO.  Keep it simple, I'd say.

Benjamin Riefenstahl <cc_benny>
Sun 15 Nov 2015 10:11:44 PM UTC, comment #13: 

> FreeSerif does not work in Emacs with SHIN + SHIN/SIN DOT


Just to clarify, this problem also happens in m17n-edit.

Benjamin Riefenstahl <cc_benny>
Sun 15 Nov 2015 10:08:03 PM UTC, comment #12: 

I append a new version of the patch that implements the suggestion.

Background details: The canonical ordering according to Unicode does
not work.  Harfbuzz has faced the same problem in 2011/2012
<https://bugzilla.mozilla.org/show_bug.cgi?id=662055>.  Their patch at
the time points to the "SBL Hebrew manual"
<http://www.sbl-site.org/Fonts/SBLHebrewUserManual1.5x.pdf> for a
discussion of the right order of Hebrew diacritics.

The appended patch does not implement the SBL recommendations exactly
yet, because I did not want to redesign the categories in the FLT
before we know that this really is what we want.

Problem cases:

The patch so far fixes all the problems that I can see when I use
FreeSerif Bold.

It does not fix the problems we have with FreeMono.

With FreeSerif (regular), it does not fix the problems with SHIN +
SHIN/SIN DOT.  This is not an ordering problem though, FreeSerif does
not work in Emacs with SHIN + SHIN/SIN DOT even without a vowel or
when I reorder the diacritics manually.  Needless to say that Gedit
and Harfbuzz show no problem.  One difference I see between FreeSerif
Bold and FreeSerif regular is that according to Harfbuzz' tool
hb-shape, FreeSerif Bold generates a precomposed glyph while FreeSerif
regular generates two glyphs.

All the problem happen not only with Emacs but also in the same manner
with the m17n example application "m17n-edit".

(file #35466,

Benjamin Riefenstahl <cc_benny>
Tue 10 Nov 2015 11:11:36 PM UTC, comment #11: 

> Could someone post a new code along this line?


If nobody beats me to it, I will try over the weekend.

Benjamin Riefenstahl <cc_benny>
Tue 10 Nov 2015 02:11:23 PM UTC, comment #10: 

> Therefore, the place where this reordering, if one is needed, should happen is in libm17n.


I agree with that, and actually performing reordering in FLT file is quite easy.  For instance, BET SHEVA DAGESH sequence matches with "Cvd", and BET DAGESH SHEVA matches with "Cdv".  To reorder the former to the latter, what we need is an extra (first) stage for reordering.

(category ...)

;; Extra reordering stage.
(generator
  (0
    (cond
       ("(C)(v)(d)" (1 =) (3 =) (2 =))
       ...)))

;; Original stage.
(generator
  (0
    ...))

Could someone post a new code along this line?

Kenichi Handa <handa>
Project AdministratorIn charge of this item.
Mon 09 Nov 2015 08:45:30 PM UTC, comment #9: 

> Question is: Where should this ordering code live? In the application (Emacs), in libm17n or in libotf? Given that libotf already needs to know the Unicode properties data, this may be easiest there.


No, I don't think libotf is the right place.  Emacs definitely isn't.  Emacs just hands a bunch of characters to the shaping engine and expects to get a sequence of one or more font glyphs in return.  It doesn't (and shouldn't) reorder anything, because the design is that all the wisdom about complex script shaping and layout should be in the shaping engine.

Therefore, the place where this reordering, if one is needed, should happen is in libm17n.

> With the patch this works for "DOUBLE YOD" + "PATAH", but not for "DOUBLE VAV" + "PATAH".


On MS-Windows, both cases display correctly.  So it's again some issue with the shaping engine.

Eli Zaretskii <eliz>
Mon 09 Nov 2015 08:13:01 PM UTC, comment #8: 

> This also leaves open the question what to do about "FreeMono". That
> does still not work, almost all diacritics are still displaced.


With the patch this works for "DOUBLE YOD" + "PATAH", but not for
"DOUBLE VAV" + "PATAH".  The difference may be that "DOUBLE YOD" +
"PATAH" gets replaced by a precomposed glyph that exists in the font,
while "DOUBLE VAV" + "PATAH" does not have such a precomposed glyph.
The PATAH glyph is just a zero-width glyph, but Emacs seems to treat
it as a full-width character.

Benjamin Riefenstahl <cc_benny>
Mon 09 Nov 2015 08:11:56 PM UTC, comment #7: 

> Not all combinations work really well with the font, some diacritics
> are still displaced when multiple diacritics are added in the wrong
> order. In principle FLT code could be written to rearrange the
> diacritics in the canonical order, I guess.


Sections 9.1, 2.11 and 3.11 of the Unicode standard (online at
<http://www.unicode.org/versions/Unicode8.0.0/> as most of you
probably know) explain that there indeed is a canonical ordering and
it looks like the fonts that we tried so far require a re-ordering
according to the rules set by Unicode.

Question is: Where should this ordering code live?  In the application
(Emacs), in libm17n or in libotf?  Given that libotf already needs to
know the Unicode properties data, this may be easiest there.

Benjamin Riefenstahl <cc_benny>
Mon 09 Nov 2015 07:11:22 PM UTC, comment #6: 

Regarding heb.patch, the addition

+ (0x05F0 0x05F2 ?C) ; yiddish ligatures

should be changed to not include 0x05F1, since nikudes should not be combined with vov-yud.

Mark H. David <mhd>
Mon 09 Nov 2015 06:42:06 PM UTC, comment #5: 

I'm adding a couple of screen shots of Shin with various Hebrew diacritics both in Emacs (bad) and in a browser (good).  This is how it looks after applying the patch, which seemed tro try to fix things with Shin.

Mark H. David <mhd>
Mon 09 Nov 2015 06:18:44 PM UTC, comment #4: 

I applied the patch. It fixes the basic <base letter followed by single diacritic> problems.  However, the <base letter followed by two or more diacritics> problems remain, including with shin dot and sin dot, unless I'm missing something.

Mark H. David <mhd>
Sat 07 Nov 2015 09:35:45 PM UTC, comment #3: 

I'm attaching a patch that makes some of the test file work for me with "FreeSerif".  I have also tried some cleanup.

  • categories: Add the yiddish ligatures.
  • generator: Redesign to allow all diacritics in any order.  Do not allow ZWNJ inside the grapheme cluster.  For a group of diacritics without base character, put them all on a single DOTTED CIRCLE instead of giving each diacritic their own.

Note: The DOTTED CIRCLE bit does not seem to work with "FreeSerif" for some reason, but I can not make the earlier code work either.  I'm probably missing something.

Not all combinations work really well with the font, some diacritics are still displaced when multiple diacritics are added in the wrong order.  In principle FLT code could be written to rearrange the diacritics in the canonical order, I guess.

This also leaves open the question what to do about "FreeMono".  That does still not work, almost all diacritics are still displaced.

Another thought: Should we allow a combination of the DOTTED CIRCLE with diacritics, I mean where the user explicitly specifies that character?

(file #35414)

Benjamin Riefenstahl <cc_benny>
Sat 07 Nov 2015 08:01:09 PM UTC, comment #2: 

In general, BET / DAGESH / SHEVA is the correct order in which a human will type these characters, but Emacs supports both orders of the diacriticals.

FWIW, MS-Windows' Uniscribe renders both sequences correctly.

Eli Zaretskii <eliz>
Sat 07 Nov 2015 07:48:41 PM UTC, comment #1: 

Problems in Detail:

== Yiddish characters ==

From: Mark H. David

> First case:
>   Upper right first sequence (bounded by space)
>   Sequence description: mem double-yud pasekh sof-nun
>   Paste of character sequence: מײַן
>   Problem: a "base character standin" glyph appears between what should
>   be the 2nd/3rd of 3 bounding boxes, the second of which comprises two
>   Unicode characters:
>     05F2;HEBREW LIGATURE YIDDISH DOUBLE YOD
>     05B7;HEBREW POINT PATAH


This seems to be, because HEBR-OTF.flt does not mention the special yiddish base characters at all.  Those are the Unicode codepoints U+05F0 - U+05F2.  This part works once I add this to the "category" statement:

   (0x05F0 0x05F2 ?C) ; yiddish ligatures

== Displaced diacritics with FreeMono ==

> Second case:
>   Upper right 5th sequence (bounded by space)
>   Sequence description: alef pasekh pey rofe reysh yud ayin
>   Paste of character sequence:  אַפֿריקע
>   Problem: the base character alef and pey are separated from
>     their following nikuds


This works fine with "FreeSerif" but not with "FreeMono".  Maybe that font behaves different, because it is supposed to be monospaced.  Interestingly the problem with the yiddish characters is fixed here also with the above addition to HEBR-OTF.flt, so it seems that file is used even in this case.

== Order of diacritics ==

Later in the example text there are problems with the sequence

  U+05D1 U+05B0  U+05BC i.e. BET / SHEVA / DAGESH

This works fine when I write it BET / DAGESH / SHEVA instead.  It seems to me that HEBR-OTF.flt requires a fixed order of types of diacritics.  Of course humans do not obey those rules ;-), and fonts do not require them either.

I'll have to read up on the FLT syntax, but I suspect that the expressions that detect various combinations there should be written to allow basically any combination of diacritics in any order.  Let the font decide where it wants to stop to support the users crazyness ;-)

Benjamin Riefenstahl <cc_benny>
Sat 07 Nov 2015 07:12:31 PM UTC, original submission:  

Emacs has the bug #21846 "emacs hebrew script bidi nikud rendering bug".  I'm attaching the test text and
screenshots in various configurations.

To summarize, the text renders well on the MS Windows version of Emacs,
but has problems on GNU/Linux, using m17n.  Instead of Hebrew base characters
combining with Hebrew diakritics (nikud) a dotted circle appears.  Some of the
problems seem to depend on the font, so far we have tried "Droid Sans Mono",
"FreeMono" and "FreeSerif".

Benjamin Riefenstahl <cc_benny>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #35538:  HEBR-OTF.flt added by cc_benny (3KiB - application/octet-stream - Contains: Yiddish characters, ordering the marks, putting non-marked marks before the base.)
file #35511:  HEBR-OTF.flt added by handa (3KiB - application/octet-stream)
file #35481:  HEBR.flt added by handa (3KiB - application/octet-stream - New version including heb.patch.)
file #35466:  heb.patch added by cc_benny (1009B - text/x-patch)
file #35467:  m17n-edit.png added by cc_benny (13KiB - image/png)
file #35424:  Emacs Shin - bad Screenshot 2015-11-09 10.38.16.png added by mhd (15KiB - image/png - Described in comment Mon 09 Nov 2015 06:42:06 PM GMT, comment #5)
file #35425:  Browser Shin - good Screenshot 2015-11-09 10.37.40.png added by mhd (19KiB - image/png - Described in comment Mon 09 Nov 2015 06:42:06 PM GMT, comment #5)
file #35414:  heb.patch added by cc_benny (887B - application/octet-stream)
file #35409:  emacs-test.txt added by cc_benny (4KiB - text/plain)
file #35410:  droid-sans-mono.png added by cc_benny (99KiB - image/png)
file #35411:  page1-freeserif.png added by cc_benny (106KiB - image/png)
file #35412:  windows.png added by cc_benny (27KiB - image/png)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by handa (Posted a comment)
  • -email is unavailable- added by mhd (Posted a comment)
  • -email is unavailable- added by eliz (Posted a comment)
  • -email is unavailable- added by cc_benny (Submitted the item)
  • -email is unavailable- added by cc_benny
  • -email is unavailable- added by cc_benny
  • -email is unavailable- added by cc_benny
  •  

    Do you think this task is very important?
    If so, you can add your encouragement to it.
    This task has 0 encouragements so far.

    Only logged-in users can vote.

     

     

     

    Follow 17 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2016-01-11 handa StatusNone => Fixed
        Assigned toNone => handa
    2015-11-24 cc_benny Attached File- => Added HEBR-OTF.flt, #35538
    2015-11-21 handa Attached File- => Added HEBR-OTF.flt, #35511
    2015-11-17 handa Attached File- => Added HEBR.flt, #35481
    2015-11-15 cc_benny Attached File- => Added heb.patch, #35466
        Attached File- => Added m17n-edit.png, #35467
    2015-11-09 mhd Attached File- => Added Emacs Shin - bad Screenshot 2015-11-09 10.38.16.png, #35424
        Attached File- => Added Browser Shin - good Screenshot 2015-11-09 10.37.40.png, #35425
    2015-11-07 cc_benny Attached File- => Added heb.patch, #35414
    2015-11-07 cc_benny Attached File- => Added emacs-test.txt, #35409
        Attached File- => Added droid-sans-mono.png, #35410
        Attached File- => Added page1-freeserif.png, #35411
        Attached File- => Added windows.png, #35412
        Carbon-Copy- => Added eliz@gnu.org
        Carbon-Copy- => Added mhd@yv.org
        Carbon-Copy- => Added handa@gnu.org

    Back to the top


    Powered by Savane 3.5