Initial IETF proposal discussion
From Test Anything Protocol
Contents |
[edit] NAME
Develop IETF Standard for TAP
[edit] VERSION
This documentation refers to IETF standard version 0.0.1
[edit] OBJECTIVE
Specify the TAP standard
The TAP standard is broken down into "core TAP" and "extended TAP". A valid TAP producer MUST produce core TAP. A valid consumer MUST understand core TAP. The purpose of "extended TAP" is to enable producers to add additional useful meaning to the test reports without affecting the core TAP pass/fail interpretation.
[edit] Establish a useful ontology
Which words do we use, and what do they mean?
[edit] Test Metadata
What information do we have to make available about the test environment? About the failing test?
[edit] TAP Defined Diagnostics vs. User Defined Diagnostics
Open for the possibility for backwards- and forwards-compatible user-supplied diagnostic information (e.g. test identification, data used in a test or other information that may be relevant for a particular test.)
- Test Location - How do we identify a specific test? URI, line? Filename? Stored Procedure? Document ID?
- We need a way to specify the source or location of code/test that is responsible for a specific test result.
- Timing - Do we want to allow for specifying test timing? A possible purpose could be for gathering test run statistics for later analysis.
- Author-specified conditional test reporting - What is the test about? How can a producer supply enough information to let the processor treat tests differently? e.g. Author who doesn't want Smoke testers to report test failures related to missing dependencies. Smoke testers should have the option to specify they are Smoke testers, and thereby follow the Author's wishes about what to report.
- Test grouping - Should we allow test tags? Should we specify certain tags as part of the standard, only allow user-defined tags, or both?
- Should we define specified contexts? (e.g. "Smoke", "Install", "Packaging", etc.)
- WTF do we do with UTF8?
- User Defined Diagnostcs/Metadata - How can we add an unambiguous mechanism for letting users add useful information to their test reports?
- Do we allow for asynchronous interleaved TAP streams?
- (e.g. for letting the consumer handle multiple streams, or requiring the producer to "serialize" the streams correctly?)
[edit] Combining multiple (possible recursive) TAP streams
[edit] Pragmas
Which problems are we trying to solve?
