[tap-l] Tracking tests over time

[Andy Armstrong]

Ian Malpass wrote:
> Requiring that test numbering doesn't change seems like an awfully large
> burden. It would mean you couldn't add extra values to a loop with a
> test in it, for example (unless it happened to be at the end of your
> test). It would be difficult to enforce, easy to forget/ignore/not know
> about, and failure would be awkward.

Yeah, I wasn't advocating that :)

> A heuristic would be complicated and fragile, and if (when) it went awry
> your data would, again, go screwy.

Yeah, possibly. I think it's more or less the same problem diff solves.
Again, I'm not actually advocating it though.

> As such, I think I'd favour the persistent identifier approach, and add
> an extra argument to each test assertion:
> 
>    is ( $got, $exptected, $test_name, $id );
> 
> You could then create and use a subclass of Test::More to care about the
> extra argument. Anyone just using plain Test::More wouldn't see any
> difference. It wouldn't be particularly taxing when writing tests, and
> would only impact those who felt that it would give them some benefit.
> 
> I'm not sure how you'd deal with not using Test::More (or that use
> Test::More internally), or using things that do stuff like
> all_pod_files_ok(), but those seem like solvable problems.

I wouldn't envisage unique IDs being mandatory.

We currently have two proposals / work in progress that I think are
relevant to this:

TAP Diagnostic Syntax
http://testanything.org/wiki/index.php/TAP_diagnostic_syntax

That gives us somewhere to put a unique test id.

TAP Blocks
http://testanything.org/wiki/index.php/Test_Blocks

Has one of its benefits localising the effects of any renumbering. A
block is effectively a nested TAP document with its own plan. Any
changes to the numbering inside the block don't propagate outside the
block. That'd benefit tests that didn't use a unique ID.

-- 
Andy Armstrong, Hexten