bugGNU roff - Bugs: bug #62300, [preconv] does not handle U+00A0...

 
 

bug #62300: [preconv] does not handle U+00A0 (NBSP) as it should

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Tue 12 Apr 2022 03:14:52 AM UTC
   
 
Category:  Preprocessor preconv Severity:  3 - Normal
Item Group:  Incorrect behaviour Status:  Fixed
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Closed Planned Release:  1.23.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sat 27 Apr 2024 03:32:18 PM UTC, comment #8: 

  The glyph 'u00A0' is defined in "groff/build/font/devpdf/U-*"
except in  U-ZD:


U-AB:u00A0        280,0        0        690        uni00A0        --
U-ABI:u00A0        280,0,0,0,50        0        690        uni00A0        --
U-AI:u00A0        277,0,0,0,50        0        690        uni00A0        --
U-AR:u00A0        277,0        0        690        uni00A0        --
U-BMB:u00A0        340,0        0        690        uni00A0        --
U-BMBI:u00A0        340,0,0,0,50        0        690        uni00A0        --
U-BMI:u00A0        300,0        0        690        uni00A0        --
U-BMR:u00A0        320,0,0,0,50        0        690        uni00A0        --
U-CB:u00A0        600,0        0        690        uni00A0        --
U-CBI:u00A0        600,0,0,0,-336        0        690        uni00A0        --
U-CI:u00A0        600,0,0,0,-269        0        690        uni00A0        --
U-CR:u00A0        600,0        0        690        uni00A0        --
U-HB:u00A0        278,0        0        690        uni00A0        --
U-HBI:u00A0        278,0,0,17,-195,17        0        690        uni00A0        --
U-HI:u00A0        278,0,0,0,-163        0        690        uni00A0        --
U-HNB:u00A0        228,0        0        690        uni00A0        --
U-HNBI:u00A0        228,0,0,0,96        0        690        uni00A0        --
U-HNI:u00A0        228,0,0,0,96        0        690        uni00A0        --
U-HNR:u00A0        228,0        0        690        uni00A0        --
U-HR:u00A0        278,0        0        690        uni00A0        --
U-NB:u00A0        287,0        0        690        uni00A0        --
U-NBI:u00A0        287,0,0,0,50        0        690        uni00A0        --
U-NI:u00A0        278,0,0,0,50        0        690        uni00A0        --
U-NR:u00A0        278,0        0        690        uni00A0        --
U-PB:u00A0        250,0        0        690        uni00A0        --
U-PBI:u00A0        250,0,0,0,-75        0        690        uni00A0        --
U-PI:u00A0        250,0,0,0,-75        0        690        uni00A0        --
U-PR:u00A0        250,0        0        690        uni00A0        --
U-TB:u00A0        250,0        0        690        uni00A0        --
U-TBI:u00A0        250,0,0,0,-75        0        690        uni00A0        --
U-TI:u00A0        250,0,0,0,-75        0        690        uni00A0        --
U-TR:u00A0        250,0        0        690        uni00A0        --
U-ZCMI:u00A0        220,0,0,0,50        0        690        uni00A0        --


Bjarni Ingi Gislason <bjarniig>
Sat 27 Apr 2024 02:56:22 PM UTC, comment #7: 

comment #6:

> the font defines operators for the glyph, which results in a
> space of a certain width.


This is a fixed width?  If so, such fonts provide an undocumented exception to this statement in groff_char(7): "a no-break space... is mapped to \~, the adjustable non-breaking space escape sequence."

This ticket being closed, if the above is in fact the case, one of us should open a new bug report.

Dave <barx>
Group Member
Sat 27 Apr 2024 12:43:04 PM UTC, comment #6: 

\[u00A0] is meaningful if you are using a font which defines the glyph. (I appear to have about 67 fonts which define the glyph, some even define kern pairs for the glyph, and one tibetan font defines composites using it).

Groff does not convert it to a horizontal motion and the font defines operators for the glyph, which results in a space of a certain width.

Deri James <deri>
Group Member
Sat 27 Apr 2024 03:03:55 AM UTC, comment #5: 

comment #2:

> The input sequence '\[u00A0]' is syntactically valid...but
> like '\[uFFFF]' and '\[u0000]', it's not meaningful


Dear future me: next time you run across this comment and think "I responded to this somewhere" but can't remember where: it's in comment 13 of bug #58930.

Dave <barx>
Group Member
Wed 13 Apr 2022 10:53:56 PM UTC, comment #4: 


> Convert input U+00A0 to \~ as troff would, not to \[u00A0].


...or as troff should, were it not for bug #58962.

Dave <barx>
Group Member
Wed 13 Apr 2022 02:17:38 AM UTC, comment #3: 


commit a22ceaea8df79608cf31b8080565053c35a0c7cc
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Tue Apr 12 13:19:19 2022 +1000

    [preconv]: Fix Savannah #62300.

    * src/preproc/preconv/preconv.cpp (unicode_entity): Convert input U+00A0
      to \~ as troff would, not to \[u00A0].

    Fixes <https://savannah.gnu.org/bugs/?62300>.


G. Branden Robinson <gbranden>
Group administrator
Tue 12 Apr 2022 10:53:36 AM UTC, comment #2: 

Hi Bjarni,

comment #1:

> commit f47b7dd139525bf3b8b4fbe767c3a45816c8445a
> Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
> Date:   Sat Nov 17 15:59:09 2018 +0000
>
>     The character \[u00A0] is not recognized
>    
>       The input character "no-break space" (' ', 0xA0) is mapped by "groff"
>     to '\~' (groff_char(7)), but only the character name '\[char160]' is
>     translated in the file "tmac/troffrc".


Yes.
    

>       The "preconv" translates the no-break space to the name '\[u00A0]'.


That was an error and is the subject of this ticket.
    

> diff --git a/tmac/troffrc b/tmac/troffrc
> index 1bd4aa8c9..8895a9a01 100644
> --- a/tmac/troffrc
> +++ b/tmac/troffrc
> @@ -33,10 +33,14 @@ troffrc!X100 troffrc!X100-12 troffrc!lj4 troff!lbp troffrc!html troffrc!pdf
>  .
>  .\" Test whether we work under EBCDIC and map the no-breakable space
>  .\" character accordingly.
> -.do ie '\[char97]'a' \
> +.do ie '\[char97]'a' \{\
>  . do tr \[char160]\~
> -.el \
> +. do tr \[u00A0]\~
> +.\}
> +.el \{\
>  . do tr \[char65]\~
> +. do tr \[u0041]\~
> +.\}
>  .
>  .\" Set the hyphenation language to 'us'.
>  .do hla us
>


I'm not sure I agree with this patch.  It's preconv's job to produce valid (GNU) troff input.  It was not doing so.

The input sequence '\[u00A0]' is syntactically valid...but like '\[uFFFF]' and '\[u0000]', it's not meaningful, and should be warned about.

Here is the patch I have pending.


diff --git a/src/preproc/preconv/preconv.cpp b/src/preproc/preconv/preconv.cpp
index 83feef8f7..b1027af17 100644
--- a/src/preproc/preconv/preconv.cpp
+++ b/src/preproc/preconv/preconv.cpp
@@ -404,9 +404,13 @@ unicode_entity(int u)
   if (u < 0x80)
     putchar(u);
   else {
-    // Handle soft hyphen specially -- it is an input character only,
-    // not a glyph.
-    if (u == 0xAD) {
+    // Handle no-break space and soft hyphen specially--they are input
+    // characters only, not glyphs.  See groff_char(7).
+    if (u == 0xA0) {
+      putchar('\\');
+      putchar('~');
+    }
+    else if (u == 0xAD) {
       putchar('\\');
       putchar('%');
     }


G. Branden Robinson <gbranden>
Group administrator
Tue 12 Apr 2022 03:37:18 AM UTC, comment #1: 

commit f47b7dd139525bf3b8b4fbe767c3a45816c8445a
Author: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>
Date:   Sat Nov 17 15:59:09 2018 +0000

    The character \[u00A0] is not recognized
   
      The input character "no-break space" (' ', 0xA0) is mapped by "groff"
    to '\~' (groff_char(7)), but only the character name '\[char160]' is
    translated in the file "tmac/troffrc".
   
      The "preconv" translates the no-break space to the name '\[u00A0]'.
   
      Example:
   
    .pl 3v
    A\[char161]B\[char160]C
    .br
    .\".tr \[u00A0]\~
    A\[u00A1]B\[u00A0]C
   
      or
   
    echo ' ' | preconv
   
    Signed-off-by: Bjarni Ingi Gislason <bjarniig@rhi.hi.is>

diff --git a/tmac/troffrc b/tmac/troffrc
index 1bd4aa8c9..8895a9a01 100644
--- a/tmac/troffrc
+++ b/tmac/troffrc
@@ -33,10 +33,14 @@ troffrc!X100 troffrc!X100-12 troffrc!lj4 troff!lbp troffrc!html troffrc!pdf
 .
 .\" Test whether we work under EBCDIC and map the no-breakable space
 .\" character accordingly.
-.do ie '\[char97]'a' \
+.do ie '\[char97]'a' \{\
 . do tr \[char160]\~
-.el \
+. do tr \[u00A0]\~
+.\}
+.el \{\
 . do tr \[char65]\~
+. do tr \[u0041]\~
+.\}
 .
 .\" Set the hyphenation language to 'us'.
 .do hla us

Bjarni Ingi Gislason <bjarniig>
Tue 12 Apr 2022 03:14:52 AM UTC, original submission:  

preconv handles the soft hyphen by translating it into an appropriate escape sequence (\%), but does not do the same for the no-break space.  groff_char(7) has long defined the semantics in these as input code points (for ISO character encodings).


$ cat whaaa.man
.TH ISO_8859-2 7 2014-10-02 "Linux" "Linux Programmer's Manual"
.TS
l l l c lp-1.
240     160     A0              NO-BREAK SPACE
255     173     AD      ­       SOFT HYPHEN
.TE
$ xxd whaaa.man
00000000: 2e54 4820 4953 4f5f 3838 3539 2d32 2037  .TH ISO_8859-2 7
00000010: 2032 3031 342d 3130 2d30 3220 224c 696e   2014-10-02 "Lin
00000020: 7578 2220 224c 696e 7578 2050 726f 6772  ux" "Linux Progr
00000030: 616d 6d65 7227 7320 4d61 6e75 616c 220a  ammer's Manual".
00000040: 2e54 530a 6c20 6c20 6c20 6320 6c70 2d31  .TS.l l l c lp-1
00000050: 2e0a 3234 3009 3136 3009 4130 09c2 a009  ..240.160.A0....
00000060: 4e4f 2d42 5245 414b 2053 5041 4345 0a32  NO-BREAK SPACE.2
00000070: 3535 0931 3733 0941 4409 c2ad 0953 4f46  55.173.AD....SOF
00000080: 5420 4859 5048 454e 0a2e 5445 0a         T HYPHEN..TE.
$ groff -t -kz -man whaaa.man # groff 1.22.4
troff: whaaa.man:4: warning: can't find special character 'u00A0'
$ ./build/test-groff -ww -t -kz -man whaaa.man $ groff Git HEAD
troff:whaaa.man:4: warning: can't find special character 'u00A0'
$ preconv whaaa.man # groff 1.22.4 and Git HEAD
.lf 1 whaaa.man
.TH ISO_8859-2 7 2014-10-02 "Linux" "Linux Programmer's Manual"
.TS
l l l c lp-1.
240     160     A0      \[u00A0]        NO-BREAK SPACE
255     173     AD      \%      SOFT HYPHEN
.TE


preconv should put \~ on the output as documented in groff_char(7) even in groff 1.22.4.


       160    the ISO latin1 no‐break space is mapped to ‘\~’, the
              stretchable space character.

       173    the soft hyphen control character.  groff never uses
              this character for output (thus it is omitted in the
              table below); the input character 173 is mapped onto
              ‘\%’.


This remapping should occur because the diagnostic itself is not the problem; there are many Unicode code points that are not valid groff input; expressing them as special character escape sequences does not change that fact.

Working on this.

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by deri (Posted a comment)
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by bjarniig (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 3 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2022-04-13 gbranden StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.23.0

    Back to the top

    Powered by Savane 3.13-caa5.
    Corresponding source code