[tap-l] Tracking tests over time
andy at hexten.net
Thu Jul 12 18:24:10 BST 2007
Ian Malpass wrote:
> Requiring that test numbering doesn't change seems like an awfully large
> burden. It would mean you couldn't add extra values to a loop with a
> test in it, for example (unless it happened to be at the end of your
> test). It would be difficult to enforce, easy to forget/ignore/not know
> about, and failure would be awkward.
Yeah, I wasn't advocating that :)
> A heuristic would be complicated and fragile, and if (when) it went awry
> your data would, again, go screwy.
Yeah, possibly. I think it's more or less the same problem diff solves.
Again, I'm not actually advocating it though.
> As such, I think I'd favour the persistent identifier approach, and add
> an extra argument to each test assertion:
> is ( $got, $exptected, $test_name, $id );
> You could then create and use a subclass of Test::More to care about the
> extra argument. Anyone just using plain Test::More wouldn't see any
> difference. It wouldn't be particularly taxing when writing tests, and
> would only impact those who felt that it would give them some benefit.
> I'm not sure how you'd deal with not using Test::More (or that use
> Test::More internally), or using things that do stuff like
> all_pod_files_ok(), but those seem like solvable problems.
I wouldn't envisage unique IDs being mandatory.
We currently have two proposals / work in progress that I think are
relevant to this:
TAP Diagnostic Syntax
That gives us somewhere to put a unique test id.
Has one of its benefits localising the effects of any renumbering. A
block is effectively a nested TAP document with its own plan. Any
changes to the numbering inside the block don't propagate outside the
block. That'd benefit tests that didn't use a unique ID.
Andy Armstrong, Hexten
More information about the tap-l