bugmake - Bugs: bug #51527, Make race issue when MikTeX...

 
 

bug #51527: Make race issue when MikTeX htlatex is used

Submitter:  Vincent Belaïche <vincentb1>
Submitted:  Fri 21 Jul 2017 03:00:13 PM UTC
   
 
Severity:  3 - Normal Item Group:  Bug
Status:  Works for me Privacy:  Public
Assigned to:  None Open/Closed:  Closed
Component Version:  None Operating System:  None
Fixed Release:  None Triage Status:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Thu 27 Jul 2017 03:35:17 PM UTC, comment #5: 

I'm closing this for now; feel free to add a new comment if you see it again.

Paul D. Smith <psmith>
Group administrator
Sat 22 Jul 2017 06:52:05 PM UTC, comment #4: 

Hello,

I installed MSYS2, and the make package, which is GNU Make 4.2.1, and the issue does not reproduce itself anymore --- well, this does not prove that it has been corrected, but hopefully it has.

Thank you for your time.


Vincent Belaïche <vincentb1>
Fri 21 Jul 2017 06:19:41 PM UTC, comment #3: 

Thank you for answering.

Good to see that the problem I met is probably corrected now, as there really was some memory corruption in make at some point of time.

I think that the definite answer for me will be to install MSYS2.





Vincent Belaïche <vincentb1>
Fri 21 Jul 2017 04:53:14 PM UTC, comment #2: 

Numerous memory corruption bugs have been fixed in the 11 years since make 3.81, some with just the sort of input sensitivity you report here.  One example:

https://savannah.gnu.org/bugs/?40159

I only have anecdotal evidence but even Debian, who would backport any fix pointed out to them for any reproducible test case, had a version that was riven by such occasional crashes until we removed the blocker for them to upgrade to version 4.x under:

https://savannah.gnu.org/bugs/?33034

I don't think you'll find anyone is interested in debugging a version of make from so long ago, so I regrettably think you'd be better off redirecting your considerable efforts towards getting a newer version.  The dates on:

https://sourceforge.net/projects/mingw/files/MSYS/Base/make/

... make that project look pretty moribund.  The eponymous ez in:

https://sourceforge.net/projects/ezwinports/files/

... is Gnu make's Windows maintainer.  I haven't tried it and don't remember seeing reference to it before but maybe you'll have more luck there?  If not, then sorry.

Martin Dorey <mdorey>
Fri 21 Jul 2017 04:06:30 PM UTC, comment #1: 

== Fact 5 ==

Look at the following:


# this fails
make realclean && make fmtord.log

# /foo/bar is dummy, this path does not exists
# this will be successful
save_path=$PATH
export PATH=/foo/bar:$PATH
# this will be successful
make realclean && make fmtord.log

# this will fail again
export PATH=$save_path
make realclean && make fmtord.log


Even changing the environment in a very slight way is sufficient for the make not to fall into the pitfall.

= Fact analysis and guessing the source of the problem =

My guess is the following : htlatex.exe produces test-fmtord-l.html by running latex 2 or 3 times in sequence in subprocesses. Make detects that test-fmtord-l.html is ready after firs 1st or 2nd suprocess is over and launches the date-timestamp awk scripts before htlatex is actually over. This creates an interlock between the latex and the awk programs trying to write-access the same file at the same time, and causes the problem.

I don't know how this happens, and I don't know either whether GNU Make or MiKTeX htlatex port is to blame.

Vincent Belaïche <vincentb1>
Fri 21 Jul 2017 03:00:13 PM UTC, original submission:  

Introduction


Hello,

  • First of all, I came into this bug with MikTeX htlatex utility, but I think that this would happen with any compiler that works the same way as htlatex (see end of this report), not only MikTeX, and not only from the latex friends.


However, the problem may be easier to fix in htlatex by designing file access in a more clever way, than in make itself.

  • Second, this bug is systematic, but reproducing it needs some very specific setting. I cannot provide a minimal example for the time being


  • Third, I am using a probably outdated MSYS port of GNU make, here is its version:



$ make --version
GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for i686-pc-msys


Although this port might be outdated, I suspect that the issue I am reporting may happen with the current make provided that (1) the .NOTPARALLEL phony target is not there, and that (2) there is some compiler working in the same way as htlatex.

The reason for such thinking is that I had a looked at the latest version of the gnu make manual on the git repo, and nothing is said about such issue.

Bug description


I came a accross a very strange issue. There seems to be some real time issue in triggering rules that in some circumstances cause a deadlock that fails the build.

Fact 1

I provide two files with the log of a make session, one file log_uds.txt is successful, and the other log_nosd.txt is failed. In one case I pass some USE_DOCSTRIP=yes argument on the command line and the make works, and in the other case I do not pass this argument and the make fails. However passing or not this variable setting on the cmd-line should not do any change because pretty early in the Makefile I have some USE_DOCSTRIP?=yes statement that makes yes the default setting !

This means to me that the only change is that the the make memory dynamic allocation is slightly changed and some race issue does not happen the same way with USE_DOCSTRIP=yes. Please note that I can repeat the test many times, and the one without USE_DOCSTRIP=yes in the command line will always fail.

Fact 2

If I run the failing test in another directory with exactly same content, but w/o the same filename lengths in the root directory path, it does not fail !! But is the root directory path has the same length for each direcory in it, then it fails.

To me this means that because some absolute path saved by make in its memory do not have the same length, then the memory allocation is slightly different and this prevent failing.

I attached a script doit.sh that reproduce the bug if you are on MSYS and place doit.sh in a directory with the right filename lengths in its path (see comment in doit.sh).

Fact 3

If I set variable USE_DOCSTRIP to other allowed values no or awk, this changes the ways the LaTeX stylefiles used by the test are docstripped, but this should not have any effect in the latex compilation result itself because basically this only changes the amount of comments ignored by LaTeX that are in the style-files processed in the test. However this alternative setting of USE_DOCSTRIP also prevents the make from failing. I understand that once again if I do that the make will not allocate dynamic memory the same way and I do not fall in the pitfall.

Fact analysis and guessing the source of problem




1) it creates with the MiKTeX htlatex.exe program some file
test-fmtord-l.html from test-fmtord-l.tex in the subdirectory trunk/test/tests

2) it produces test-fmtord-l-nodate.html from test-fmtord-l.html with a lightweight AWK script to strip out any date-timestamp information.

3) Finally it compares test-fmtord-l-nodate.html to a reference file in subdirectory trunk/test/references, and the outcome of comparison give the test verdict.

My guess is the following : htlatex.exe produces test-fmtord-l.html by running latex 2 or 3 times in sequence in subprocesses. Make detects that test-fmtord-l.html is ready after firs 1st or 2nd suprocess is over and launches the date-timestamp awk scripts *before* htlatex is actually over. This creates an interlock between the latex and the awk programs trying to write-access the same file at the same time, and causes the problem.

I don't know how this happens, and I don't know either whether GNU Make or MiKTeX htlatex port is to blame. But I think that any compiler that have a similar behaviour as htlatex (ie doing several output pass of the same target files by some subprocessing) will be at the same risk.


Vincent Belaïche <vincentb1>

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #41276:  log_nosd.txt added by vincentb1 (152KiB - text/plain)
file #41277:  doit.sh added by vincentb1 (563B - application/octet-stream)
file #41278:  log_uds.txt added by vincentb1 (152KiB - text/plain)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by psmith (Posted a comment)
  • -email is unavailable- added by mdorey (Posted a comment)
  • -email is unavailable- added by vincentb1 (Submitted the item)
  • -email is unavailable- added by vincentb1
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only logged-in users can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-07-27 psmith StatusNone Works for me
        Open/ClosedOpen Closed
    2017-07-21 vincentb1 Attached File- Added log_nosd.txt, #41276
        Attached File- Added doit.sh, #41277
        Attached File- Added log_uds.txt, #41278
        Carbon-Copy- Added -email is unavailable-

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code