bugGNU roff - Bugs: bug #64484, [troff] .device and \X don't...

 
 

bug #64484: [troff] .device and \X don't behave the same

Submitter:  G. Branden Robinson <gbranden>
Submitted:  Fri 28 Jul 2023 10:22:42 AM UTC
   
 
Category:  Core Severity:  3 - Normal
Item Group:  Feature change Status:  Need Info
Privacy:  Public Assigned to:  gbranden
Open/Closed:  Open Planned Release:  1.24.0
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Mon 04 Mar 2024 11:48:47 PM UTC, comment #10: 

Already reopened, but for the record..


commit 190d01502ec8f8caab05bd3eb11ee04411c1e332
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Mon Mar 4 00:14:39 2024 -0600

    Revert "[troff]: Fix Savannah #64484."

    This reverts commit 9dbf227a5b3870a19c1e6d90e5b619c4ae3e7f3e.

    Reopens Savannah #64484.

    Test src/roff/groff/tests/device-control-special-character-handling.sh
    fails at this commit.


G. Branden Robinson <gbranden>
Group administrator
Tue 16 Jan 2024 07:34:17 PM UTC, comment #9: 

Having second thoughts about the direction in which I achieved parity here.

Maybe both `device` and `\X` should be reading their argument in copy mode.

Technically, though, that would break compatibility with AT&T troff.  However, there are no cross-formatter conventions for the format or content of device control commands in the first place, so I don't expect this to be a practical issue.

Proposal/discussion thread:

https://lists.gnu.org/archive/html/groff/2024-01/msg00060.html

G. Branden Robinson <gbranden>
Group administrator
Sat 13 Jan 2024 06:48:22 PM UTC, comment #8: 


commit 974c063f0a9e1ef6c0d2cac4755a3b9d6e925b0d
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sat Jan 13 08:08:16 2024 -0600

    [troff]: Unit-test device control spec chars.

    * src/roff/groff/tests/device-control-special-character-handling.sh: Add
      unit test for this feature.  We want to be able to consistently pass
      (some) special character escape sequences to device control commands,
      and we want the `device` request and `\X` escape sequences to behave
      consistently with each other.

    * src/roff/groff/groff.am (groff_TESTS): Run test.

    Test fails at this commit.  See Savannah #64484.

commit 9dbf227a5b3870a19c1e6d90e5b619c4ae3e7f3e
Author: G. Branden Robinson <g.branden.robinson@gmail.com>
Date:   Sat Jan 13 09:26:58 2024 -0600

    [troff]: Fix Savannah #64484.

    * src/roff/troff/input.cpp (encode_char_for_troff_output): Annotate the
      function's purpose.  Initially assume the character to be encoded as
      valid.  If the current token is a plain space, write a space (U+0020)
      to the output.  (This is necessary because the `device` request no
      longer reads its arguments in copy mode; see below.)  Move the `sc`
      local variable to a higher scope.  Update the new `is_char_valid`
      Boolean instead of issuing an error diagnostic at each point of
      validation failure.  When done processing the character, test
      `is_char_valid` and emit different diagnostics depending on whether
      the input was a special character escape sequence we can't handle, or
      something else.  Emit a self-quoted escape character _as a backslash_,
      not as the current *roff escape character.

      (device_request): Rewrite to operate in interpretation mode, not copy
      mode.

    * doc/groff.texi (Postprocessor Access):
    * man/groff.7.man (Request short reference):
    * NEWS: Document it.

    Fixes <https://savannah.gnu.org/bugs/?64484>.


G. Branden Robinson <gbranden>
Group administrator
Tue 09 Jan 2024 12:24:20 AM UTC, comment #7: 

My last paragraph in comment #6 was a bit mangled.

Let me try that again.

The foregoing means that I should abandon my plan to make a new fatal error in response to bug #65138 and also that the scenario in question is probably a corner case.  Generally, you won't issue device control command with a request but never even once have a partially collected line in your document.  A unit test might do that, but a practical document likely will not.

G. Branden Robinson <gbranden>
Group administrator
Mon 08 Jan 2024 04:47:39 PM UTC, comment #6: 

comment #5:

> Another oddity is that you can invoke the `device` request too early, and it won't be honored, but can use `\X` anywhere you want.


This turns out to be due to the presence of a break on a line using the \X escape sequence, because it's delimited.  That explanation may not be complete because this succeeds regardless.


\X'ps: nop'\c


So I guess what's actually happened is that a partially collected line exists and that's enough to make the output copasetic.

But that means that my plan to make a new fatal error consequent to bug #65138 and also that that is probably a corner case.  Generally, you won't issue device control command with a request but never even once have a partially collected line in your document.

G. Branden Robinson <gbranden>
Group administrator
Mon 08 Jan 2024 04:38:59 PM UTC, comment #5: 

Another oddity is that you can invoke the `device` request too early, and it won't be honored, but can use `\X` anywhere you want.


$ cat EXPERIMENTS/device-control-escape-sequence.roff
\X'ps: nop'
$ ~/groff-stable/bin/groff -Z EXPERIMENTS/device-control-escape-sequence.roff
x T ps
x res 72000 1 1
x init
p1
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X ps: nop
n12000 0
x trailer
V792000
x stop
$ cat EXPERIMENTS/device-control-request.roff
.device ps: nop
$ ~/groff-stable/bin/groff -Z EXPERIMENTS/device-control-request.roff
x T ps
x res 72000 1 1
x init
p1
x trailer
V792000
x stop


G. Branden Robinson <gbranden>
Group administrator
Mon 28 Aug 2023 01:40:28 AM UTC, comment #4: 

comment #2:

> I don't know whether the pdfmark macros rely on the asciifying
> behaviour of \X or not.


While pdfmark is now maintained externally, Keith should still be able to post to savannah bugs, so I'm cc:ing him for his input on this question.  (It's not clear the answer makes any practical difference, as the change under consideration seems to be to .device and not to \X, but it's worth having all the pertinent info to inform any decisions.)

Dave <barx>
Group Member
Sat 29 Jul 2023 09:47:45 PM UTC, comment #3: 


comment #2:

> There are 4 separate ways to send text to a postprocessor.


Thanks, Deri.  I've documented all of these but did not consider them this way.  This is good stuff.

> \X is the only one which attempts to "asciify" the parameters.


Yes.  My instinct it to change this so that the `device` request does the same processing as `\X`.

> There is a difference between the two pairs of commands, if you remove the .fl and rerun:-


> x T ps
> x res 72000 1 1
> x init
> p1
>  ! \(ti deri
> output \(ti deri
> tm \(ti deri
> V12000
> H72000
> x font 5 TR
> f5
> s10000
> V12000
> H72000
> md
> DFd
> x X X ~ deri
> wh2500
> V12000
> H74500
> x X device \(ti deri
> n12000 0
> x trailer
> V792000
> x stop


> .output and \! are now out of chronological order.


This is well worth studying.  I think that thanks to you we have at last identified a use case for the `fl` request in GNU troff (as opposed to AT&T nroff, where it had an application in interactive use).  I've seen macro packages use it pointlessly.

> In gropdf I used to rely on manual "asciifying" of parameters, the new gropdf does not, in order to support unicode (passed as \[uXXXX]) I need it to be passed untouched.
>
> so it can be converted to UTF-16 by gropdf itself (also applies to any special chars - \(xx \[xxx] \C'xxx' \N'nnn'). It also needs to maintain chronological order.


I think we might be in for some design changes here in the 1.24 cycle.

TANAKA Takuji's patches to grops in bug #62830 have what look to me like similar concerns regarding the production of UTF-16 by the output driver.

> I don't know whether the pdfmark macros rely on the asciifying behaviour of \X or not.


I don't either.

I'd like `device` and `\X` to be counterparts, as `output` and `\!` appear to be.

Even better would be for `sy`, `tm`, and other requests that access the standard error stream or the file system to use the same mechanism for encoding characters.  See bug #64071.

This stuff is going to require some thought.

G. Branden Robinson <gbranden>
Group administrator
Sat 29 Jul 2023 08:17:41 PM UTC, comment #2: 

There are 4 separate ways to send text to a postprocessor.

.ds a deri
\X'X \(ti \*a'
.device device \(ti \*a
.fl
\! ! \(ti \*a
.output output \(ti \*a
.tm tm \(ti \*a    \" output to STDERR

Which produces:-

x T ps
x res 72000 1 1
x init
p1
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X X ~ deri
wh2500
V12000
H74500
x X device \(ti deri
n12000 0
 ! \(ti deri
output \(ti deri
tm \(ti deri
x trailer
V792000
x stop

\X is the only one which attempts to "asciify" the parameters. There is a difference between the two pairs of commands, if you remove the .fl and rerun:-

x T ps
x res 72000 1 1
x init
p1
 ! \(ti deri
output \(ti deri
tm \(ti deri
V12000
H72000
x font 5 TR
f5
s10000
V12000
H72000
md
DFd
x X X ~ deri
wh2500
V12000
H74500
x X device \(ti deri
n12000 0
x trailer
V792000
x stop

.output and \! are now out of cnronological order.

In gropdf I used to rely on manual "asciifying" of parameters, the new gropdf does not, in order to support unicode (passed as \[uXXXX]) I need it to be passed untouched.so it can be converted to UTF-16 by gropdf itself (also applies to any special chars - \(xx \[xxx] \C'xxx' \N'nnn'). It also needs to maintain chronological order.

I don't know whether the pdfmark macros rely on the asciifying behaviour of \X or not.

Deri James <deri>
Group Member
Fri 28 Jul 2023 10:23:49 AM UTC, comment #1: 

Assigning to Deri to solicit his perspective on this ticket, about which I am conflicted.

G. Branden Robinson <gbranden>
Group administrator
Fri 28 Jul 2023 10:22:42 AM UTC, original submission:  

The `device` request and `X` escape sequence don't behave the same way when it comes to encoding of things that aren't representable in the page description language (groff_out(5)).

Exhibit:


$ cat EXPERIMENTS/device-vs-X.groff
.nf
foo
\X'baz: \%qux man-beast\[u1F63C]'
.device baz: \%qux man-beast\[u1F63C]
$ groff -Z EXPERIMENTS/device-vs-X.groff | grep '^x X'
x X baz: qux man-beast
x X baz: \%qux man-beast\[u1F63C]


This is not a regression from groff 1.22.4


$ /usr/bin/groff -Z EXPERIMENTS/device-vs-X.groff | grep '^x X'
troff: EXPERIMENTS/device-vs-X.groff:3: a special character is invalid within \X
x X baz: qux man-beast
x X baz: \%qux man-beast\[u1F63C]


It seems likely to me that `device` was added to GNU troff to provide a request counterpart to the \X escape sequence (an AT&T device-independent troff feature), as `brp` did for `\p`.

I think we should either (1) get our story straight here (put the behavior into parity) or (2) strongly call out the distinction in documentation and supply use cases for why you'd employ one versus the other.

My preference is for (1) right now but I'd appreciate hearing from Deri, who gives these features a lot of exercise in "pdf.tmac".

G. Branden Robinson <gbranden>
Group administrator

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

No files currently attached

 

Depends on the following items: None found

Digest:
   bug dependencies.

 

Carbon-Copy List
  • -email is unavailable- added by barx (Posted a comment)
  • -email is unavailable- added by barx (a question about pdfmark arose)
  • -email is unavailable- added by deri (Posted a comment)
  • -email is unavailable- added by gbranden (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 15 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2024-01-16 gbranden StatusFixed Need Info
        Open/ClosedClosed Open
    2024-01-14 gbranden Dependencies- bugs #63074 is dependent
    2024-01-13 gbranden Severity2 - Minor 3 - Normal
        Item GroupIncorrect behaviour Feature change
        StatusIn Progress Fixed
        Open/ClosedOpen Closed
        Planned ReleaseNone 1.24.0
    2024-01-13 gbranden StatusNone In Progress
    2023-08-28 barx Carbon-Copy- Added keithmarshall
    2023-07-29 gbranden StatusNeed Info None
    2023-07-29 deri Severity3 - Normal 2 - Minor
        Assigned toderi gbranden
    2023-07-28 gbranden StatusNone Need Info
        Assigned toNone deri

    Back to the top

    Powered by Savane 3.13-bb6a.
    Corresponding source code