bugThe FreeType Project - Bugs: bug #56067, PCF defaultChar changes in 2.10...

 
 

You are not allowed to post comments on this tracker with your current authentication level.

bug #56067: PCF defaultChar changes in 2.10 causes FontConfig to mis-detect(?) charsets

Submitter:  None
Submitted:  Wed 03 Apr 2019 07:17:50 PM UTC
   
 
Severity:  3 - Normal Item Group:  Incorrect behaviour
Status:  Fixed Privacy:  Public
Assigned to:  wl Open/Closed:  Closed
Planned Release:  2.10.1

Jump to the original submission

Mon 22 Apr 2019 06:34:55 PM UTC, comment #16: 

I see. Those are actually .notdef and .null rather than duplicates in the UCS misc-fixed fonts.

FreeType does not read PCF_GLYPH_NAMES table. Therefore it was not obvious.


Alexei Podtelezhnikov <podtelez>
Group Member
Mon 22 Apr 2019 05:26:37 PM UTC, comment #15: 

This doesn't work.  Glyph index 0 must be used only for the missing glyph, because `FT_Get_Next_Char' stops as soon as it encounters glyph index 0.  However, the PCF `defaultChar' is used as a real character also.  It thus must be duplicated.

Werner LEMBERG <wl>
Group administrator
Mon 22 Apr 2019 01:45:12 PM UTC, comment #14: 

Purely cosmetically, would it make sense to check if .notdef is already at offset = 0? This way it would not be duplicated.

The classic ucs-miscfixed is such fonts.

Alexei Podtelezhnikov <podtelez>
Group Member
Fri 19 Apr 2019 08:53:32 AM UTC, comment #13: 

Looks good as far as I can tell. I tested with Dina, TerminusV and a couple of artwiz fonts and I didn't notice any issues.

Thank you so much for the fix, it's much appreciated!

Daniel Eklöf <dnkl>
Fri 19 Apr 2019 05:05:35 AM UTC, comment #12: 

It should be fixed now in git, please test.

Werner LEMBERG <wl>
Group administrator
Mon 15 Apr 2019 06:14:50 PM UTC, comment #11: 

I'm working on a fix.

Werner LEMBERG <wl>
Group administrator
Mon 15 Apr 2019 03:53:43 PM UTC, comment #10: 

Werner,

In Dina font the defaultChar is also space 0x20. It is incorrect to place it at offset=0 becase this also indicates that the charmap end at the space.



Alexei Podtelezhnikov <podtelez>
Group Member
Sat 06 Apr 2019 09:35:06 AM UTC, comment #9: 

Sorry if my last couple of posts have been confusing; I've been
learning a lot about fonts in a very short time...

This will mostly be repetition, but hopefully in a more clear, and
better structured way.

Are there problems with Dina, TerminsuV and Artwiz in an unmodified
freetype 2.10?

Yes. That's what this bug report is all about; fontconfig fails to
enumerate charsets, causing it to do font fallback. See details in the
first post. I used mostly Dina in my examples there.

The problem is that pcf_cmap_char_next() returns (index) 0 when it
encounters the font's default character. This in turn causes
FT_Get_Next_Char() to stop iterating.

Thus, an application that enumerates the characters in e.g. Artwiz
gelly will see "... 1 2 3", and then stop at '4', since that's the
default character.

The patch in (which also mentions Dina, btw)
http://lists.nongnu.org/archive/html/freetype-devel/2019-03/msg00080.html
sort of solves this; it makes pcf_cmap_char_next() skip the default
character. This allows fontconfig to properly enumerate the charsets,
and that also means the fonts now are usable.

BUT. Since we skip the default character, at least fontconfig+pango
will do font fallback for the default character. So, using Artwiz
gelly as an example again, with the patch applied, we'll be able to
render "1 2 3 5 6 7 8 9" with the artwiz font, but '4' will fallback
to something else.

As a test, I hacked pcf_cmap_char_next() to return the original
default char index when we encounter the default character. This at
least lets the caller see that the character exists (but with the
wrong index, of course).

This changed the behavior in fontconfig+pango to, instead of falling
back for '4', rendering its internal "undefined character" box.

So, where are we currently?

The patch in
http://lists.nongnu.org/archive/html/freetype-devel/2019-03/msg00080.html
is a must; without it, most PCF fonts are unusable with fontconfig.

But, it is not complete. We need a way to present the default
character to the user.

I'm not sure that can be done, to be honest. For example, the
documentation for FT_Get_Char_Index() return "The glyph index.
0~means 'undefined character code'." The caller might thus do font
fallback, or something else, when it sees a character with index=0?

I'm willing to work on a fix for this, but to do that I would need to
know what the expected behavior should be. With my incredibly limited
knowledge about all this, I'd say you cant have a "regular"
character (say, '4') at index 0 - there's simply too much special
treatment for that index.

Daniel Eklöf <dnkl>
Fri 05 Apr 2019 08:57:59 PM UTC, comment #8: 

The gohu font is invalid; its default character has index 0, however, this glyph doesn't exist in the font (i.e., the font's encoding slot at position 0 is marked as 0xFFFF).

The Dina font is OK; it uses `-' as the default character.  TerminusV and Artwiz are OK, too.  Those fonts indeed use funny characters as the default; I have no idea why this is so.

I suggest to fix the gohu font by replacing the offending `0xFFFF' bytes with the proper offset to, say, the `space' glyph.

With an unmodified FreeType 2.10.0, are there problems with Dina, TerminusV, or Artwiz?

Werner LEMBERG <wl>
Group administrator
Thu 04 Apr 2019 12:00:39 PM UTC, comment #7: 

Or are the fonts broken; should they be using a different default glyph?

I've only checked a few fonts but:

TerminusV uses '.' as the default character
Artwiz gelly uses '4'

Daniel Eklöf <dnkl>
Thu 04 Apr 2019 10:50:06 AM UTC, comment #6: 

I did a quick hack where I modified pcf_cmap_char_next to return the original default character encoding offset when we encounter the default character.

This (still) allowed fontconfig to properly detect the charsets. Now, instead of falling back to a different font for the default character, we get the "invalid character boxes".

Could this be because pcf_cmap_char_index returns 0 when asked for the default character's index? How would we go about fixing that?

Daniel Eklöf <dnkl>
Thu 04 Apr 2019 08:52:33 AM UTC, comment #5: 

Hmm, the initial problem, solved by the patch, is that FT_Get_Next_Char returned *agindex == 0 for the character before the default character.

With the patch, we're simply skipping the default character. That seems wrong.

So, isn't the core problem that you swap the default character with glyph 0?

Daniel Eklöf <dnkl>
Thu 04 Apr 2019 08:21:34 AM UTC, comment #4: 

Thanks for the pointer. That patch does seem to fix most of the issues I've seen.

The output of fc-match now matches the old output, and the fonts now seem to be selected correctly by fontconfig.

However, there is still a problem with the default character; in "artwiz gelly.se" for example, the default character is 52 ('4' in ascii). Trying to render this particular character results in a font fallback. All other characters render correctly.

This is observable with

FC_DEBUG=4 pango-view --font="artwiz gelly.se 11px" -q -t '4' | grep family:


Daniel Eklöf <dnkl>
Thu 04 Apr 2019 07:33:27 AM UTC, comment #3: 

A patch was posted on the freetype-devel list:

http://lists.nongnu.org/archive/html/freetype-devel/2019-03/msg00080.html

I haven't had the time yet to review it, so please test!

Werner LEMBERG <wl>
Group administrator
Thu 04 Apr 2019 06:37:35 AM UTC, comment #2: 

Yes, those are the original fonts. (I'm the reporter btw)

Dina is generated:

    sed -i 's/microsoft-cp1252/ISO8859-1/' *.bdf

    for i in Dina_i400-*.bdf; do
        bdftopcf -t -o DinaItalic$(_ex_pt $i).pcf $i
    done
    for i in Dina_i700-*.bdf; do
        bdftopcf -t -o DinaBoldItalic$(_ex_pt $i).pcf $i
    done
    for i in Dina_r400-*.bdf; do
        bdftopcf -t -o DinaMedium$(_ex_pt $i).pcf $i
    done
    for i in Dina_r700-*.bdf; do
        bdftopcf -t -o DinaBold$(_ex_pt $i).pcf $i
    done
    gzip *.pcf


I don't know how the gohu font is generated; the arch package pulls the PCFs directly from http://font.gohu.org/, and I couldn't find any info on how they are generated.

Note that these aren't the only fonts with issues; I've had the same kind of problems with the artwiz font collection. The are pulled from http://artwizaleczapka.sourceforge.net/. I was unable to find out how they are generated.

I'm attaching the Dina font, as generated by arch. The other fonts can be downloaded from their web sites.

(file #46719)

Daniel Eklöf <dnkl>
Wed 03 Apr 2019 08:09:49 PM UTC, comment #1: 

Are these the original fonts in question?
http://font.gohu.org/
http://www.dcmembers.com/jibsen/download/61/

I wonder how PCF version of Dina was generated. The fonts are free, please attach or send them over.

Alexei Podtelezhnikov <podtelez>
Group Member
Wed 03 Apr 2019 07:17:50 PM UTC, original submission:  

There's a long thread over at the arch forums,
https://bbs.archlinux.org/viewtopic.php?id=245188, about missing
glyphs and missing bitmap fonts that pretty quickly was traced to
freetype2 being bumped from 2.9 to 2.10.

It was originally reported here:
https://savannah.nongnu.org/bugs/?func=detailitem&item_id=55979

But was closed.

However, the problem seems to be broader than just "spaces replaced
with 0200 boxes". It seems most of the time, programs that in some way
verify that glyhps exist in the uesr selected (bitmap) font, will fail
and fallback to a different (non-bitmap) font.

One way this can be observed is:

FC_DEBUG=4 pango-view --font Dina -q -t 'a'|grep family:


It will show a switch from Dina right at the end.

Turning off font fallback results in Dina being the final font:

FC_DEBUG=4 pango-view --font Dina --markup  -t '<span fallback="false">a</span>'|grep family:


Here's a diff of FontConfig's view of the "Dina" font (2.9.1 compared with 2.10):

--- /tmp/ft-2.9        2019-04-03 19:04:56.661331718 +0200
+++ /tmp/ft-2.10        2019-04-03 19:05:10.057988215 +0200
@@ -23,9 +23,9 @@ Pattern has 38 elts (size 48)
         rgba: 1(i)(w)
         scale: 1(f)(s)
         charset:
-        0000: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
+        0000: 7fffffff 00000000 00000000 00000000 00000000 00000000 00000000 00000000
 (w)
-        lang: aa|ay|bi|br|ch|da|de|en|es|eu|fj|fo|fur|fy|gd|gl|gv|ho|ia|id|ie|io|is|it|lb|mg|nb|nds|nl|nn|no|nr|oc|om|pt|rm|sma|smj|so|sq|ss|st|sv|sw|tl|ts|uz|vo|wa|xh|yap|zu|an|fil|ht|jv|kj|kwm|li|ms|ng|pap-an|pap-aw|rn|rw|sc|sg|sn|su|za(s)
+        lang: (s)
         fontversion: 0(i)(s)
         fontformat: "PCF"(w)
         embeddedbitmap: True(s)


A git bisect between VER-2-9-1 and current master resulted in this:


cba72a0b0f0256e61af8dc6cb1924b143b127dde is the first bad commit
commit cba72a0b0f0256e61af8dc6cb1924b143b127dde
Author: Werner Lemberg <wl@gnu.org>
Date:   Sat Jul 21 23:45:32 2018 +0200

    [pcf] Fix handling of the undefined glyph.

    This change makes the driver use the `defaultChar' property of PCF
    files.

    * src/pcf/pcf.h (PCF_FaceRec): Change type of `defaultChar' to
    unsigned.

    * src/pcf/pcfread.c (pcf_get_encodings): Read `defaultChar' as
    unsigned.
    Validate `defaultChar'.
    If `defaultChar' doesn't point to glyph index zero, swap glyphs with
    index zero and index `defaultChar' and adjust the encodings
    accordingly.

    * src/pcf/pcfdrivr.c (pcf_cmap_char_index, pcf_cmap_char_next,
    PCF_Glyph_Load): Undo change from 2002-06-16 which always enforced
    the first character in the font to be the default character.


Anonymous

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attached Files
file #46719:  DinaMedium10.pcf.gz added by dnkl (5KiB - application/gzip)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by crem (Adding myself to follow the issue)
  • -email is unavailable- added by wl (Posted a comment)
  • -email is unavailable- added by dnkl (Updated the item)
  • -email is unavailable- added by podtelez (Posted a comment)
  • -email is unavailable- added by dnkl
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

     

    Follow 9 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2019-04-19 wl StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 2.10.1
    2019-04-15 wl StatusNeed Info In Progress
    2019-04-08 crem Carbon-Copy- Added crem
    2019-04-04 wl StatusNone Need Info
        Assigned toNone wl
    2019-04-04 dnkl Attached File- Added DinaMedium10.pcf.gz, #46719
    2019-04-03 dnkl Carbon-Copy- Added dnkl

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code