bugGNU Octave - Bugs: bug #49507, test: allow unit tests to signal...

 
 

bug #49507: test: allow unit tests to signal that a test is expected to fail based on arbitrary runtime check

Submitter:  Mike Miller <mtmiller>
Submitted:  Tue 01 Nov 2016 05:58:09 PM UTC
   
 
Category:  Octave Function Severity:  1 - Wish
Priority:  5 - Normal Item Group:  Feature Request
Status:  Fixed Assigned to:  None
Originator Name:  Open/Closed:  * Closed
Release:  * dev Operating System:  * Any
Fixed Release:  None Planned Release:  None
* Mandatory Fields

Add a New Comment Rich Markup
   

Jump to the original submission

Sun 19 Mar 2017 04:08:58 PM UTC, comment #6: 

That change looks like it does what I was proposing. I haven't tried it but it looks right to me.

Mike Miller <mtmiller>
Group Member
Sun 19 Mar 2017 06:27:03 AM UTC, comment #5: 

Has this bug been fixed by jwe's changeset here:


changeset:   23279:67d741321e21
user:        John W. Eaton <jwe@octave.org>
date:        Wed Mar 15 16:02:06 2017 -0400
summary:     allow skipping of tests based on run-time conditions


For example, part of jwe's cset looks like this now:


-%!testif HAVE_OPENGL, HAVE_FLTK
-%! if (! have_window_system)
-%!  return;
-%! endif
+%!testif HAVE_OPENGL, HAVE_FLTK; have_window_system




Rik <rik5>
Group administrator
Tue 06 Dec 2016 05:51:07 PM UTC, comment #4: 

Something like the attached patch might do the job. It uses the error id proposed by Mike to make test skippable conditionally on runtime.
I also attached a small test file the uses that error id (unconditionally).

Still un-documented.

(file #39162, file #39163)

Markus Mützel <mmuetzel>
Group administrator
Tue 06 Dec 2016 03:13:15 AM UTC, comment #3: 

Oh yeah, there might be packages using the HAVE_XYZ syntax which would mean a headache to track down any change there.

Dan Sebald <sebald>
Tue 06 Dec 2016 03:05:35 AM UTC, comment #2: 

I looked at test.m a bit.  I understand the proposal, but I'm not sure how well that would work.  I suppose it is doable.  ID variable is not well documented in the error() function, so I'm not sure if that is supposed to be a printf ID (i.e., output) or an error type.  So, would test.m be looking for a particular ID?  Or would it have to scan the error message for a word such as "skip"?

Another thought is if the "testif" feature were slightly more robust, it might cover everything needed.  For example, negation of test requirements and evaluation of strings might provide everything needed.  For example, in a related gnuplot issue for publish.tst, a solution might be:

%!testif {"!HAVE_OSMESA", "any(strcmp(available_graphics_toolkits, 'gnuplot')"}

which would check for HAVE_OSMESA defined to 0 in a slight variation of the current test.m behavior, but also it would evaluate any string not of that form.

In some sense it would be simpler if instead of the slightly complex code for decomposing HAVE_OSMESA there were simply an Octave function called HAVE('OSMESA') that basically does the same thing.  That would mean that there is then no special consideration for HAVE_XYZ, e.g.,

%!testif {"!HAVE('OSMESA')", "any(strcmp(available_graphics_toolkits, 'gnuplot')"}

I would think this is a very straightforward modification of test.m.  Also there doesn't seem to be but a dozen script files using the HAVE_XYZ syntax in their tests?  Those could be modified easily with opening the file in an editor and using FIND/REPLACE.

Dan Sebald <sebald>
Wed 02 Nov 2016 01:17:51 AM UTC, comment #1: 

As a first stab at a proposal, I think it would be nice to use a particular error ID that is defined by Octave to be caught by the test function. A test that needs to depend on some feature at runtime could do something like


%! if (! have_some_optional_feature ())
%!   error ("Octave:test:expected-failure", ...
%!          "error: some feature is missing, skipping test");
%! endif


This would require the test.m function to look for that particular error ID (using lasterror, since we don't implement exception objects yet), and set appropriate local variables to indicate that this is a skipped test.

This error ID would be documented and advertised so that Forge packages and user code could also use this error ID to get the same results using test.

Other thoughts?

Mike Miller <mtmiller>
Group Member
Tue 01 Nov 2016 05:58:09 PM UTC, original submission:  

This was discussed on the mailing list but not copied to a bug report.

It would be nice if unit test blocks could run arbitrary code and based on some runtime condition, raise a particular error or return a particular exit status that indicates to the test runner that the test is expected to fail / skip / do not report as a failure.

This came up in the context of Java tests. All tests are conditioned on HAVE_JAVA, which is fine as a compile-time condition, but also need to test whether the JRE can be loaded at runtime.

Similar tests could occur for existence of gnuplot, or existence of a DISPLAY / headless environment (Octave function have_window_system()), or for existence of optional functions, runtime libraries, etc.

Mike Miller <mtmiller>
Group Member

 

(Note: upload size limit is set to 16384 kB, after insertion of the required escape characters.)

Attach Files:
   
   
Comment:
   

Attached Files
file #39162:  bug49507_skiptests.patch added by mmuetzel (1KiB - application/octet-stream)
file #39163:  test_skiptest.m added by mmuetzel (162B - application/octet-stream)

 

Depends on the following items: None found

Items that depend on this one: None found

 

Carbon-Copy List
  • -email is unavailable- added by rik5 (Posted a comment)
  • -email is unavailable- added by mmuetzel (Updated the item)
  • -email is unavailable- added by sebald (Posted a comment)
  • -email is unavailable- added by cbm
  • -email is unavailable- added by mtmiller (Submitted the item)
  •  

    There are 0 votes so far. Votes easily highlight which items people would like to see resolved in priority, independently of the priority of the item set by tracker managers.

    Only group members can vote.

     

    Follow 6 latest changes.

    Date Changed by Updated Field Previous Value => Replaced by
    2017-03-20 rik5 StatusNeed Info Fixed
        Open/ClosedOpen Closed
    2017-03-19 rik5 StatusConfirmed Need Info
    2016-12-06 mmuetzel Attached File- Added bug49507_skiptests.patch, #39162
        Attached File- Added test_skiptest.m, #39163
    2016-11-01 cbm Carbon-Copy- Added cbm

    Back to the top

    Powered by Savane 3.13-d3ae.
    Corresponding source code