Talk:TAP at IETF: Draft Standard
From Test Anything Protocol
[edit] TAP Grammar
The best TAP grammar at the moment is probably the one inside TAP::Parser::Grammar. Since we're basically writing down what Test::Harness does, it seems fitting to draw from it. The current TAP version is version 13. A history of changes to TAP can be found here. --Schwern 03:02, 19 January 2009 (UTC)
[edit] Whitespace
How strict do we want to be about whitespace? TAP::Parser::Grammar typically uses \s+ to match between elements. Do we want to put this lax view of whitespace into the RFC, or use a more strict single space to separate most tokens? Being strict will allow for simpler parsing, but it feels dishonest. --Schwern 03:31, 19 January 2009 (UTC)
AndyA sez go with sloppy whitespace. --Schwern
[edit] Newlines (closed)
How do we handle newlines? Right now it's system specific. This will become a problem as TAP is used as an archive format or streaming from one machine to another. Settling on one will interfere with reading raw TAP. For example, if we choose CRLF, the Internet newline, then Unix users will have trouble reading raw TAP output. OTOH the web seems to have no problem dealing with both newlines. --Schwern 03:33, 19 January 2009 (UTC)
It has been suggested to pick a single EOL and pretend there's a preprocessor which normalizes newlines. Apparently that's what XML does. --Schwern 00:58, 21 January 2009 (UTC)
- I just incorporated that into the current description. -- Gaurav 12:01, 18 November 2009 (GMT)
[edit] Reasons
What is the set allowed in a reason / description? Right now I have 1*CHAR but that seems a bit too broad. VCHAR seems too narrow, it doesn't include whitespace does it. 1*(SP / VCHAR)? --Schwern 03:51, 19 January 2009 (UTC)
Going with UTF8 with the caveat that the interchange parties can agree otherwise, which is pretty much the current state of affairs. The allowed set is anything but EOL and "#". If you want to put weird control characters in your description, sure, shoot your own foot. --Schwern 01:00, 21 January 2009 (UTC)
[edit] Multiline descriptions and escape characters
Do we want to allow multi-line descriptions? For example `pass("Foo\nBar")` comes out as what? Currently it's `ok 1 Foo\n#Bar` but that's unsatisfying as what you get is a test named "Foo" and then the comment "Bar". Probably what we need is real escape characters. `ok 1 Foo\x0a` --Schwern 01:01, 21 January 2009 (UTC)
[edit] Escape sequence
There is a defacto escape sequence for putting a # in a description: "\#". But it's not particularly well thought out. Technically there's no way to encode a literal "\#". We need something. --Schwern 01:01, 21 January 2009 (UTC)
[edit] TODO in the plan
It's an unmaintainable way to write TODO tests. It complicates determining if a test line is a pass or fail (you can't do it grammatically). Hardly anyone uses it. Test.pm is the only thing which emits it. It's already marked obsolete, I'm tempted to drop it from the plan. Or at least downgrade it to "may". --Schwern 01:06, 22 January 2009 (UTC)
The plan is to leave it in, because it's in use, but deprecated. --Schwern 19:33, 2 April 2009 (UTC)
[edit] Optional - before description
The grammar should be altered to allow an optional "-" before the description. Test::Builder does this extensively. It makes the test result easier to read. It also solves some edge cases. For example, if you want to express an unnumbered test with the name "42". Without the dash it is "ok 42" which is ambiguous. With the dash it's "ok - 42".
[edit] Compatibility version number
TAP version should be altered to split the version number into version representing compatibility and a revision. For example, "TAP version 2.23" would be compatibility version 2 revision 23. The compatibility version is incremented when a backwards incompatible change is introduced. A backwards incompatible change is one where an older parser would get a different result for the test suite. For example, if we changed "ok" to "pass" that would be backwards incompatible.
The revision would simply represent changes to the protocol. When the compatibility number increments the revision number resets to 1.
[edit] 1..0 without a reason is valid
"1..0" without a reason or directive indicates a skip. It's used extensively in the Perl core and on CPAN.
It can be deprecated, but the grammar should mention it to be able to handle real world code.
[edit] How do define deprecation
Deprecation can be formally defined as "the writer must not write X" and "the if reader encounters X it must do one of the following things: silently ignore, silently follow the specified behavior or issue a warning about the use of a deprecated feature and follow the behavior". This follows the idea of be strict in what you emit, lax in what you accept.
[edit] What does output after BAIL_OUT mean?
What is the harness supposed to do when it sees a BAIL_OUT exactly?
Answer: Quietly ignore it all. Everything after the bail out is suspect.
AndyA said: Actually we should probably reword the spec to something like "if the relationship between the harness and the test is such that the harness can safely kill the test it may do so"
[edit] Describe reasons and descriptions
Reasons and descriptions are only described in the grammar. Their purpose needs to be defined in the text. --Schwern 19:32, 2 April 2009 (UTC)
[edit] Expand the Abstract
IETF recommends the abstract be not under 3 lines, usually 5 to 10. So our abstract needs some attention. --Schwern 19:40, 2 April 2009 (UTC)
[edit] Security Considerations
We haven't done anything for security considerations.
Tainting? Does a bad parser have any exposure to malformed content? Can you craft TAP such that it can be used to attack the box? DoS? Memory starvation?
- Memory starvation. A naive implementation would store test results in an array indexed by test number. A test of
1..1234567890 ok 1 ok 1234567890
- would create a huge in memory array. This has happened to Test::Harness. So its recommended that a non-contiguous method be used to store results, such as a tree or hash. --Schwern 21:06, 24 June 2010 (BST)
[edit] Capitalization of MUST, SHALL, etc...
1. Decide if we're going to capitalize MUST, SHALL, etc...
2. Make it consistent through the document.
As aesthetically displeasing all-caps is, my thinking is to capitalize since it highlights where the concrete rules are in a very wordy RFC. --Schwern 19:59, 2 April 2009 (UTC)
- I'm in favor of caps. Andrew Rodland 12:45, 19 December 2009 (GMT)
[edit] "test result" -> "assert"
What we refer to as a "test result" the rest of the testing universe would call an "assert" or an "assertion". Perhaps we should adopt this more standard vocabulary. --Schwern 20:06, 2 April 2009 (UTC)
[edit] American or British English?
Analyze or Analyse? Color or Colour? Are we using American or British? Pick one and make it consistent. Does the IETF say anything about this? --Schwern 20:18, 2 April 2009 (UTC)
[edit] Discuss the plan
Why do we have a plan? What purpose does it serve? What sort of bugs and test mistakes does it prevent? Why put it at the front or the back? Talk about counting strategies (ie. no_plan and done_testing from Test::More). --Schwern 20:25, 2 April 2009 (UTC)
[edit] TAP Workflow
The TAP workflow needs to be discussed more definitively. How the protocol is used in practice. How tests get run, TAP gets to the consumer, what the consumer does with it. But remain general, don't make it Perlish.
[edit] Merge "Structure" with "Grammar"
"Structure" is attempting to give an overview of a TAP document in English. Unfortunately it loses important details and makes generalizations. It would be better to merge Structure and Grammar. Break up Grammar into sections with descriptions. "Structure" becomes the comments for about is "; Overall structure" in the grammar.
[edit] Ambiguities in the description of the grammar
I'm afraid that there are some contradictions / ambiguities in the language of the document that need to be cleared up.
The text says that a test "must have a test number" -- but then it goes on to describe what happens if there isn't a test number, and indeed the grammar shows that the test number is optional.
The text should say "a test MAY have a test number", and it should address:
- What happens when there is no number?
- Is it allowed for the first test to have no number? If so, let's explicitly specify the number.
- Is it allowed for a test with no number to have a description? If so, how do we keep the parse unambiguous? (Prohibit leading numerics in descriptions?)
- Agreed, that's confusing wording. Its talking about the interpretation of the result, not the actual protocol. Test numbers are optional, but one will be assigned to it. The first test can have no number, and if it doesn't it will be assigned 1. More correctly, it will be assigned the first number in the plan, which is currently always 1 but might be something else later. Yes, you can have a test number with no description. See http://testanything.org/wiki/index.php/Talk:TAP_at_IETF:_Draft_Standard#Optional_-_before_description for discussion. That proposal has yet to be integrated into the grammar/draft. --Schwern 21:02, 24 June 2010 (BST)
The text also says that "a document must parse as per the grammar above to qualify as a TAP document" but I'm concerned that that's not really as clear as it could be when combined with the repeated statement that "a line that doesn't parse should be ignored". What are the consequences for failure to parse supposed to be?
The best thought I have would be something along the lines of "if a document can be made to match the grammar above by removing unrecognized lines, it should be considered valid TAP; otherwise, (recommended behavior for a consumer that gets bad TAP)". That does leave leeway for a consumer to throw away a malformed line that was supposed to be a test and then parse the rest, but that should at least generate a FAIL according to the "no missing numbers" rule, unless numbers aren't in use.
- A document where all lines are ignored is simply interpreted as a TAP document with no plan and no tests. Thus, it is a failure. It is not a parse error. Good nit picking, I'm writing this down here for eventual inclusion in the draft. --Schwern 21:02, 24 June 2010 (BST)
Andrew Rodland 11:08, 19 December 2009 (GMT)
