[tap-l] Nested tests with TAP::Harness and verbose output]

[Michael Moore]

I tried to send this earlier, but I think it bounced, so here it is again.


I've set up our batch tests to use TAP::Harness's runtests to run
sub-tests. We have many cases where BatchTestA calls BatchTestB which
calls TestC.

If I create a TAP::Harness object in both BatchTestA and BatchTestB, it
works fine (all good so far!).

I wanted to get a little more information out of our tests, so I passed
in a hash with the parameter "verbosity => 1" to each of the
TAP::Harness objects.

I found that If any TAP::Harness object other than the top level test
was set to verbose, it would break the TAP output. Is this expected
behavior?

Here is the resulting output:
=== Start output ===
./LevelA.bat
LevelB........
LevelC......
ok 1 - The test in LevelC
1..1
ok 2
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.00 usr  0.01 sys +  0.00 cusr
0.02 csys =  0.03 CPU)
Result: PASS
ok 1 - The test in LevelB
1..1
 All 1 subtests passed

Test Summary Report
-------------------
LevelB.bat (Wstat: 0 Tests: 3 Failed: 2)
  Failed test number(s):  1-2
  Parse errors: Plan (1..1) must be at the beginning or end of the TAP
output
                Tests out of sequence.  Found (1) but expected (3)
                More than one plan found in TAP output
                Bad plan.  You planned 1 tests but ran 3.
Files=1, Tests=3,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.00 cusr
0.10 csys =  0.11 CPU)
Result: FAIL
ok 1 - The test in LevelA
1..1
=== End output ===

I'm not sure if attachments are OK on this list, so I'll just put the
three tests I used to narrow down the behavior here inline. If you save
these three tests and run LevelA.bat, you should see the described
behavior. If you comment out the line "verbosity => 1" in LevelB.bat,
the problem will go away.

-----LevelA.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');

my %tapHarnessArgs = (
        verbosity => 1,
        );

my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelB.bat");
ok(1==1,"The test in LevelA");


-----LevelB.bat-----
#!/usr/bin/perl
use strict;
use warnings;
use TAP::Harness;
use Test::More;
plan('no_plan');

my %tapHarnessArgs = (
        verbosity => 1,
        );

my $batch = TAP::Harness->new(\%tapHarnessArgs);
$batch->runtests("LevelC.t");
ok(2==2,"The test in LevelB");


-----LevelC.t-----
#!/usr/bin/perl
use strict;
use warnings;
use Test::More;
plan('no_plan');
ok(3==3,"The test in LevelC");



Many Thanks,
Michael Moore