Hackathon notes

From Test Anything Protocol

Jump to: navigation, search

Contents

[edit] TAP glossary

  • producer: something that prints TAP output
  • consumer: something that reads TAP output
  • archiver: something that stores TAP output in a way that it can be consumed
  • TAP statement: one single test (e.g. ok($foo == 1);)
  • TAP stream: a series of test statements as a stream (e.g. output of t/test.t)
  • TAP suite: a group of test streams (e.g. t/*.t) - currently no explicit representation of a suite in TAP

[edit] For the xUnit folk

In the xUnit world:

  • an assertion is the name for a test statement (ok, not ok, etc.)
  • a test is a group of assertions in a method testing a single feature
  • a test class is a.... class... with tests in it
  • a test suite is a group of test classes

[edit] YAML Diagnostics associated with each test

  • Should work both for success and failure
  • Not a serialization format, but rather "here, look at this"
  • Everything optional
  • We allow more than one YAML document, but we only guarantee that we'll read the first one
  • Accept anything
  • Uppercase Keys are extensions
  • Lowercase keys are for official TAP use and reserved
  • First line ("---") of yaml document decides level of indentation
    • url:
    • file: (relative to CWD)
    • line:
    • source: for code
    • tags: (always a list)
    • results:
      • have:
      • want:
      • compare: (have is lhs, want is rhs, always an binary operator)
    • display: (advisory, about how to display test result)

[edit] Notes on diagnostics

  • File is a 'common case' key when you don't want to specify a file URL.
  • Compare operator is assumed to be equality by default.

[edit] Escaping

  • Problem: "description" that include a "#" are misinterpreted
  • Backslash now escapes the # "\#"
  • Backslash Backslash now escapes to \
  • Already in T::H 3.0 - we just haven't documented it yet

[edit] Metadata

  • Displayed after the version declaration:
    • the head-metadata before the plan
    • the tail-metadata after stream of TAP statements
  • All values in the head meta-data are inherited by the tap statements so that you can declare common meta-data for all tests in a single location

[edit] Head

TAP version 15

  • hostname:
  • ip:
  • cwd:
  • programming_language: (name needs to be defined?)
  • access:
    • user/group/etc. (values to be defined)
  • platform:
    • CPU/memory/Arch/OS version/vendor? (values to be defined)
  • file:
  • url:
  • producer:
    • name: Test::Builder
    • version: 1.14
  • start_time: yaml datetime

[edit] Tail

TAP done

  • end_time:
  • status:
    • exit - platform dependent error code
    • wait - platform dependent wait code
    • normal: (YAML boolean, cross-platform exit status interpretation)
  • TAP done: comes after the deferred plan.

[edit] YAML / Formats for values

For the moment we're saying that all YAML:

[edit] DateTime

[edit] Booleans

What YAML says.

[edit] Compression

We skip by using test number compression

  • ok 3..15 # SKIP
  • ok 3..15 # TODO
  • ok 3..15 - something worked
  • Unnumbered tests can't be compressed

[edit] Skipping

  • 1..0 -- is defined as a skip
  • not ok 3 # SKIP -- defined as a parse error
  • otherwise just use compression

[edit] Things that we're putting off until we think about them some more

[edit] Nested TAP

Use cases:

  • Aggregation
  • Test suite contains test streams
  • XUnit style test methods
  • Spawning more tests
  • Forking/threading
  • Sub-plans
  • Does the sub-test know?
  • Backwards compatibility?
  • Interleaving subtests
  • header and footer description must match, and is optional

Initially adopted a variation of the [Test Blocks] proposal. By default meta-data is not inherited in the nested TAP stream. If you want it inherited then you set "inherit" to a true value. 

TAP version 15
  ---
  access: foo
  ... 
1..3
ok 1
begin 2 - fribble
  TAP version 16
    ---
    inherit:  y
    ...
  1..3
  ok 1
  not ok 2
  begin 3
    1..1
    ok 1
  ok 3  
  TAP done
not ok 2 - fribble
ok 3
TAP done

Problem:

  • The next day we spotted that we need to parse the nested TAP stream - which is bad. So we're putting it aside for now

(Should probably move this to the TAP Blocks page?)

[edit] Severity / Testing Contexts / Report vs Fail

  • Started off with the idea of a general severity attribute (e.g. severity: 0, severity: 5 ).
  • Discarded because it's context dependent (e.g. POD tests are severe if you're testing the documentation, but don't matter much when installing a module)
  • Also wanted to handle the idea of some tests that should be counted as success, but reported (e.g. highly intermittent bugs, informational tests, etc.)
  • Added concept of a testing context (e.g. smoke, install, author, default, author)
  • Added idea of actions in a particular context (e.g. set "install:ignore") for tests that should be ignored for installation.
  • Ended up with two dimensions of action:
    • quiet: y/n - do we report it to the author
    • fail: y/n - do we treat it as a failure

For example: 

   not ok 2 - Pod complete
     ---
     line: 12
     file: t/pod.t
     on_ok:
       smoke:
         quiet: n
     on_not_ok:
       install:
         fail: n
     tags:
       - documentation
     ...

Would report to the author in a smoke test, and not fail in a installation context.

Everybody started thinking that this was getting slightly too complex, and we're probably missing something. Punted for now so we can get some actual implementation done.


[edit] Other stuff we didn't discuss in day 1

  • Binary results display / typing of values in have/want (probably just use YAML)
  • Logging levels
  • TAP parser tests
  • TAP spec
  • timestamp for test statements: (TAP DateTime - when we start printing the diagnostics)
  • Global ID for tests
  • Benchmarking
  • Trace (?)
  • Test suite run
Retrieved from "http://testanything.org/wiki/index.php/Hackathon_notes"
Personal tools