Test Blocks
Test blocks allow a number of tests - maybe the entire output from a single test script - to be grouped within a larger TAP stream.
TAP version 14
1..4
begin 1 Object creation
1..2
ok 1 Object created OK
ok 2 Object isa Flunge::Twizzler
end 1 Object creation
ok 2 Clone OK
begin 3 Methods
1..4
ok 1 has twizzle method
ok 2 has burnish method
ok 3 has spangle method
not ok 4 has frob method
end 3 Methods
ok 4 Resources released
Contents |
Notes
- each begin/end block has its own plan; leading and trailing plans are supported
- the plan count at each level includes the blocks and tests at that level
Benefits
This is the primary motivation. Current TAP doesn't scale well in the case where a single test script generates a large number of results. If you have 10,000 tests you have a flat list of 10,000 results.
Simplifies plan calculation
The top level plan declares how many naked tests and blocks are expected. Each block then provides a sub-plan that declares how many tests and nested blocks it contains.
A common idiom for data driven testing is to have an array of hashes that describe test cases and an 'engine' that iterates those cases. Each case may expand to multiple individual tests. Such tests must either be unplanned or they must
- ensure they execute a fixed number of tests per case so that the plan becomes cases x N where N is the fixed number of tests per case or
- walk the test data to calculate exactly how many tests are expected
With begin / end blocks it's necessary only for the top level plan to declare the number of cases. Each case may then calculate and plan for the number of individual tests it will execute.
Allows multiple TAP transcripts to be concatenated
Currently if you wish to combine these TAP transcripts into a single document
1..2 ok 1 spozzle mictified ok 2 clapper loaded
1..3 ok 1 tripe deprecated ok 2 frome visited not ok 3 semantics nailed
You must combine the plans and renumber the tests
ok 1 spozzle mictified ok 2 clapper loaded ok 3 tripe deprecated ok 4 frome visited not ok 5 semantics nailed 1..5
That means that information about the source of the tests is lost and tests must be renumbered. The plan must move to the end because we can't output the plan until we've seen how many individual tests there are.
With begin / end you can instead output
1..2
begin 1 Tests from t/spozzle.t
1..2
ok 1 spozzle mictified
ok 2 clapper loaded
end 1
begin 2 Tests from t/tripe.t
1..3
ok 1 tripe deprecated
ok 2 frome visited
not ok 3 semantics nailed
end 2
The original test numbering is preserved and it's still clear where each test came from.
SKIP and TODO can become blocks
Test::More provides the notion of SKIP and TODO blocks. With blocks these can be reflected in the structure of the generated TAP. Instead of
1..6 ok 1 - aligned ok 2 - locked ok 3 - loaded not ok 4 - bang ok 5 # skip not running on Windows ok 6 # skip not running on Windows
You could have
1..2 begin 1 Generic tests 1..4 ok 1 - aligned ok 2 - locked ok 3 - loaded not ok 4 bang end begin 2 Win32 specific tests 1..0 # skip not running on Windows end
Blocks are named
Allows a TAP browser UI to present a hierarchy of test results which can be folded and unfolded to drill down to the results of interest.
Can wrap older TAP versions
The contents of a begin / end block can be TAP 12 or earlier which means that begin / end blocks may be used to group and organise TAP streams generated by older producers.
Criticisms
Not backwards compatible
- if it's indented as shown above then current parsers will ignore most of the lines (if it's not indented it's hard for humans to see the nesting)
- requires 2 new keywords
- tests no longer have a unique identifier.
- it cannot be made to support forked/threaded tests because a line's position in the output stream determines what block it came from and so the output must be created in a coordinated manner.
Related Proposals
Test Groups - test grouping based on a hierarchical numbering scheme
