Test Groups

(edit note, I just now noticed that I had the test numbers and the ok/not ok reversed, although I'd prefer it with the numbers first, that's completely different to TAP - Fergal)

eg

1..3
ok 1
1..2 2 a block
1..3 2.1 another block
ok 2.1.1
ok 2.1.2
ok 2.1.3
ok 2.1 # end of another block
ok 2.2
ok 2 # end of a block
1..3 3 a third block
ok 3.1
ok 3.2
not ok 3 # end of a third block, planned for 3 but only ran 2 tests

Each group should have it's own optional plan

Each group counts as a test in the group which contains it, the makes it easy to replace a single test with a group of tests with no adjustment to the plan.

For a group to pass, all tests inside the gorup must pass and any plan must be verified. Any failure will cascade up through the hierarchy and cause a failure at the top level.

The results should come in order that is result x.y should come before x.(y+1) however there is no reason to require that x.(y+1) must come before (x+1).z. That is

ok 1
ok 2.1
ok 3
ok 2.2

should be valid but

ok 1
ok 2.2
ok 2.1

is is invalid. This allows threaded tests to run each thread in it's own block without no need for coordination between threads.

Related Proposals

Test Blocks - test grouping based on nested begin/end blocks

Criticisms

Breaks backwards compatibility

Not a little either. If a proposal is going to do that it had better fill some gaping void in TAP that can't be otherwise filled. So far that's only the nested plans which are useful for multi-threaded apps.

Comment on the criticism:

It's mostly backwards compatible (only the plan lines cause a problem) but yes it's not perfectly backwards compatible.

Nested plans are not only useful for multi-threaded apps, they're useful for test libraries, where a single call into the library runs multiple tests and even just for subroutines and loops in a test script.

Incrementally extending the plan can already be done

A TAP producer can already provide incremental plan extensions like the following.

    use Test::More;
    plan 'as_you_go';  # hypothetical.

    for my $num (1..2) {
        extend 2;

        like $num, qr/^\d$/;
        cmp_ok $num, ">", 0;
    }

This would output:

   # Extending plan by 2 to be 2 at foo.t line 6.
   ok 1
   ok 2
   # Extending plan by 2 to be 4 at foo.t line 6.
   ok 3
   ok 4
   1..4

A simple END guard ensures that the plan must be extended at least once. The producer can detect if too many or too few tests are run in a subplan. This can be done without any extensions, a prototype has been posted to perl-qa.

This would seem to cover all the use-cases except for nested plans.

Criticism of the criticism:

This form of plan extension has serious problems. The loop above says 1..2 but maybe it was supposed to be 1..3 either way, the plan printed at the end will be "correct". The root of the problem is that there is nowhere to plan how many times you're going to loop.