TAP diagnostic syntax

At YAPC::NA 2006 we talked about using a subset of YAML (or if you dislike YAML think of it as "mail headers with stuff") for parsable test diagnostics. The syntax described here is now implemented by TAP::Parser as YAMLish. The changes haven't made it to CPAN yet. In the mean time you can find the bleeding edge version in the subversion repo.

   not ok 2 - an important test
     ---
     file:        foo.t
     line:        45
     description: an important test
     got:         this
     expected:    that
     raw-test:    is( "this", "that", "an important test" );
     extensions:
       THAC0:     16
     ...
   ok 3

This format is easy to read, easy to parse (as long as we keep our heads about which parts of YAML we adopt) and extendable.

Diagnostics need not be only for failures. There's nothing which says you can't output them for passes.

Whether or not the ... is present, the TAP diagnostic block ends when the next test starts.

The "extensions" namespace provides a way for the TAP producer to output its own non-standard TAP diagnostics without worrying about clashing with future standard keys.

The information is indented both to make it more visually distinctive and make it unambiguous where it ends even if the ... is missing. It also makes sure there's no potential interference with older parsers:

   ok 1 
     ---
     ok: 1
     ...
   ok 2

Open Issues

• Since "# foo" is what we call a diagnostic now, perhaps this should be a diagnostic and "# foo" be more rightly called a comment (in the "do not try to parse this!" sense).
• "expected"/"actual" has been proposed as an alternative to the keyword pair "got"/"expected".
• The raw TAP output needs to be made more readable so that the eyes are drawn to the important information (got vs expected) and not the trivial (name, line and file). The harness can, of course, dress it up however it wants.

Examples

Test::Differences

Test::Differences currently outputs this:

not ok 1 - differences in text
#     Failed test (foo.t at line 14)
#     +---+----------------+----------------+
#     | Ln|Got             |Expected        |
#     +---+----------------+----------------+
#     |  1|this is line 1  |this is line 1  |
#     *  2|this is line 2  |this is line b  *
#     |  3|this is line 3  |this is line 3  |
#     +---+----------------+----------------+

Most of what Test::Differences does is display the failure in a pretty format. Display of a failure can be the TAP displayer's job. So Test::Differences can output this:

not ok 1 - differences in text
  ---
  file:  foo.t
  line:  14
  name:  differences in text
  got:   |
    this is line 1
    this is line 2
    this is line 3
  expected: |
    this is line 1
    this is line b
    this is line 3
  display: |
    +---+----------------+----------------+
    | Ln|Got             |Expected        |
    +---+----------------+----------------+
    |  1|this is line 1  |this is line 1  |
    *  2|this is line 2  |this is line b  *
    |  3|this is line 3  |this is line 3  |
    +---+----------------+----------------+
  dump:
    some_var: 42
    some_hash:
      "this is line 1": 1
      "this is line 3": 3
      "this is line b": 2
    some_array:
      - 1
      - 'b'
      - 2
  ...

"got" and "expected" is the normal raw textual output. The TAP displayer can take this output and display it any way it wants, including by running it through diff. "display" is a suggestion to the TAP displayer about how it can be displayed. The TAP displayer is free to ignore this suggestion. "dump" is a hash of variables whose values may be pretty printed by the harness.

In effect, with a decent TAP displayer Test::Differences is no longer needed.