[tap-l] User Supplied YAML Diagnostic Keys: Descriptive Version

[Michael G Schwern]

Executive summary:  User key collision is not a show stopper.


chromatic wrote:
> On Thursday 17 April 2008 17:56:25 Michael G Schwern wrote:
> 
>> We're working around the same issue Perl 5 is having adding new keywords. 
>> In Perl 5, since keywords and user-defined subroutines share the same
>> space, we can't add a new keyword without risking clashing with a
>> user-defined subroutine and giving it new meaning.  Thus the "use feature"
>> shame.  By reserving all the lower case keys for official TAP keys and
>> leaving everything else for users, TAP is free to add new keys.
> 
> Perl 5 also solves the problems of Bob's user-defined subroutines having the 
> same name as Chuck's user-defined subroutines.  C doesn't.  C++ does.  PHP 
> didn't until (I believe PHP 5.2).  I'm sure you can guess which language 
> feature this is.

You're right, it's sensible for a programming language to separate user 
functions and variables written by different users because programming 
languages require very precise meaning.


>> We want convergence.  If user defined keys are to have meaning for more
>> than just the user or group that defined them, they must converge on a
>> common meaning.  User key collisions encourage that.  Users and test
>> producer authors can spot the collision and work it out.  Lists of common
>> keys and meanings can be published.  TAP displayers can make use of this
>> and start to do things with common, user defined keys.
> 
>> As the wide-spread utility of a key becomes apparent, they can be lower
>> cased and made official.
> 
>> It's much like tagging, with most of the chaotic problems and benefits,
>> except that there will be a growing set of well-defined, official keys to
>> provide some stability.
> 
> It didn't work for SMTP headers.  It didn't work for HTTP headers.  Given that 
> SMTP and HTTP are somewhat more popular and widespread than TAP, what makes 
> you think it will possibly work for TAP?

Sorry, I'm not terribly familiar with the history of SMTP and HTTP and I don't 
know what you're referring to.  AFAIK HTTP has a well defined set of headers 
and I'm not aware of their defining any mechanism for user defined headers.  I 
don't know what the analogy is.

As for why it'll work with TAP, with a few exceptions (exit_status, or 
whatever we decide to call it, is currently the only one), diagnostic keys do 
not effect test parsing.  It's not a show stopper.  At worst, a displayer that 
has been customized to do something special with a user defined key might show 
some odd output.  The tests still pass and fail as before.

Prefixing brings an additional problem, how do you tell a TAP displayer to do 
anything with all those prefixed keys?  Let's say Test::Stuff has 
Perl.Test.Stuff-time.  So you program your displayer to do something with 
Perl.Test.Stuff-time.  Now the Test::Thing author thinks that's a neat idea 
and adds Perl.Test.Thing-time.  So you add in Perl.Test.Thing-time to your 
displayer to do the same thing as Perl.Test.Stuff-time.  Test::Foo picks up on 
it, and you have to add another special case.  Test::Bar adds it in, another 
special case in your displayer.  Another module, add another to the list.

You're either going to wind up with a big list of prefixes and keys, which is 
annoying work, or you're going to break down and match on /-time$/ and defeat 
the point of prefixing.


>> Human readability of the raw diagnostics is important.
> 
> I don't find these big blobs of YAML particularly readable when compared to 
> the freeform diagnostics we have now.

The choice of YAML is a compromise between machine parsability, extensibility 
and human readability.  Could be worse, could be XML.  Fortunately the TAP 
displayer can make it look all rainbows and lollipops if you like, but it's 
important to ensure it remain eyeballable.


> That said, here's a refinement: TAP keys (presumably the most common keys) 
> don't have prefixes.  Other keys do.  80% solution to readability, and a kind 
> of pressure to TAP producers to standardize their keys.

I'm unhappy with the readability consequences, still emphasizing the wrong 
thing: official vs not.  It's also a pressure in the wrong direction.  The 
pressure should be convergence, not standardization.  The standard can pluck 
keys out of the wild on it's own.

Also this still requires defining how to make a globally unique prefix that 
doesn't read like a UPC code.

Finally it falls afoul of the multiplicity of prefixes making TAP displayer 
behavior customization involve a bunch of accounting mentioned above.


>> With readability problems, specification complications and inhibiting
>> convergence, prefixing adds more problems than it solves.
> 
> In the absence of prefixing, what solves the problem of key collision between 
> unrelated producers?  Expecting users to debug these problems and convince 
> two or more producers to present their case for convergence to the TAP 
> working group in a timely fashion is only one thinly-veiled metaphor about 
> banana plantations short of the canonical 20th century Latin American novel.

User key collision is not a show stopper for the user.  As mentioned above, at 
worst a customized TAP displayer shows some confused output for that key.  So 
it's not urgent that it be worked out.

The authors of the two colliding producers simply hash it out amongst 
themselves.  There's no need to involve anyone else or get any sort of 
official blessing.  Getting the key blessed is an orthogonal problem.


-- 
100. Claymore mines are not filled with yummy candy, and it is wrong
      to tell new soldiers that they are.
     -- The 213 Things Skippy Is No Longer Allowed To Do In The U.S. Army
            http://skippyslist.com/list/