TAP diagnostic syntax
From Test Anything Protocol
Contents |
[edit] Status
This has been accepted and, after revision, will be in a future revision of TAP. TAP::Parser has an implementation.
[edit] For
- Schwern 23:51, 4 September 2007 (BST)
- AndyArmstrong 00:01, 5 September 2007 (BST)
[edit] Against
[edit] Proposal
At YAPC::NA 2006 we talked about using a subset of YAML (or if you dislike YAML think of it as "mail headers with stuff") for parsable test diagnostics. The syntax described here is now implemented by TAP::Parser as YAMLish. The changes haven't made it to CPAN yet. In the mean time you can find the bleeding edge version in the subversion repo.
not ok 2 - an important test
---
file: foo.t
line: 45
description: an important test
found: this
wanted: that
raw_test: is( "this", "that", "an important test" );
extensions:
THAC0: 16
...
ok 3
This format is easy to read, easy to parse (as long as we keep our heads about which parts of YAML we adopt) and extendable.
Diagnostics need not be only for failures. There's nothing which says you can't output them for passes.
Whether or not the ... is present, the TAP diagnostic block ends when the next test starts.
The "extensions" namespace provides a way for the TAP producer to output its own non-standard TAP diagnostics without worrying about clashing with future standard keys.
The information is indented both to make it more visually distinctive and make it unambiguous where it ends even if the ... is missing. It also makes sure there's no potential interference with older parsers:
ok 1
---
ok: 1 # the key "ok" is used here just to illustrate there is no ambiguity
...
ok 2
[edit] Keys
TAP diagnostic is optional, it can be omitted. There are also no mandatory keys.
The following is a list of agreed upon keys and their meanings.
- found
- The actual value returned from the test. Usually paired with wanted. By convention, found should come first.
- wanted
- The value we expected from the test. Usually paired with found.
- file
- The file which produced this result. Usually a relative path.
- line
- The line of the file which produced this result. If it's a multi-line function call use the first line.
- raw_test
- The code which produced this result.
- description
- A human readable description of what was tested.
- extensions
- In order to avoid clashing with future TAP diagnostic keys you should put any new keys not listed here under extensions.
[edit] Key name conventions
- Use an underscore (_) to separate words. (example: first_name)
- Use lower-case, unless there's a specific reason for that key.
[edit] Formatting conventions
While any YAML is valid, there are some considerations to be made for human readability.
- Line up the values of keys which are on the same indentation level
- For example, it makes it easier to read long lists of keys.
ok 1
---
file: foo/bar.t
line: 42
description: A test that does some stuff with things.
found: foo
wanted: bar
...
ok 2
It also makes it much easier to compare found/wanted results to see subtle differences.
ok 1
---
found: This is a really long value and it might be hard to find a typo.
wanted: This is a realy long value and it might be hard to find a typo.
...
[edit] Open Issues
- Since "# foo" is what we call a diagnostic now, perhaps this should be a diagnostic and "# foo" be more rightly called a comment (in the "do not try to parse this!" sense).
- The raw TAP output needs to be made more readable so that the eyes are drawn to the important information (found vs wanted) and not the trivial (name, line and file). The harness can, of course, dress it up however it wants.
[edit] Examples
[edit] Test::Differences
Test::Differences currently outputs this:
not ok 1 - differences in text # Failed test (foo.t at line 14) # +---+----------------+----------------+ # | Ln|Got |Expected | # +---+----------------+----------------+ # | 1|this is line 1 |this is line 1 | # * 2|this is line 2 |this is line b * # | 3|this is line 3 |this is line 3 | # +---+----------------+----------------+
Most of what Test::Differences does is display the failure in a pretty format. Display of a failure can be the TAP displayer's job. So Test::Differences can output this:
not ok 1 - differences in text
---
file: foo.t
line: 14
name: differences in text
found: |
this is line 1
this is line 2
this is line 3
wanted: |
this is line 1
this is line b
this is line 3
display: |
+---+----------------+----------------+
| Ln|Got |Expected |
+---+----------------+----------------+
| 1|this is line 1 |this is line 1 |
* 2|this is line 2 |this is line b *
| 3|this is line 3 |this is line 3 |
+---+----------------+----------------+
dump:
some_var: 42
some_hash:
"this is line 1": 1
"this is line 3": 3
"this is line b": 2
some_array:
- 1
- 'b'
- 2
...
"found" and "wanted" is the normal raw textual output. The TAP displayer can take this output and display it any way it wants, including by running it through diff. "display" is a suggestion to the TAP displayer about how it can be displayed. The TAP displayer is free to ignore this suggestion. "dump" is a hash of variables whose values may be pretty printed by the harness.
In effect, with a decent TAP displayer Test::Differences is no longer needed.
[edit] Hackathon notes
- Should work both for success and failure
- Not a serialization format, but rather "here, look at this"
- Everything is optional
- First doc only
- Accept anything
- Uppercase Keys are extentions
- Lowercase keys are internal
- First line ("---") of yaml document decides level of indentation
url: file: (relative to CWD) line: source: for code tags: (always a list) results: have: want: compare: (have is lhs, want is rhs, always an binary operator) display: (advisory, about how to display test result)
