Support diagnostics *before* the ok/not ok line

[cdleonard]

Right now tap assumes that unknown messages are diagnostics for the preceding "ok"/"not ok" line. In some cases it would be very nice and natural to reverse this. It also makes it very easy to run shell scripts with arbitrary outputs as tests and interpret their exit code as pass fail.

Printing diagnostic messages at the end is not always practical. For example when working on an embedded system it's common for test failures to completely crash the target system and require an external reboot. For such cases partial test output should immediately available, even before pass/fail can be established.

I've skimmed through the subtest proposal and it seems to also require that the ok/not ok for a group is printed before the individual items.

1..2
run 1
starting foo daemon...
requesting bar
halting foo daemon
ok 1 foo::bar

run 2
initial temperature is 55 C
starting busy loop in background
new temperature is still 55 C, no difference? 
not ok 2 tempon

My proposal would be to have lines matching "run ([0-9]+) .*" mark the start of messages for a certain test, until the "ok"/"not ok" with the same number. This is simple and minimally intrusive and should also work fine if you want more structured output.

I'm curious if there are tap tools which implement similar extensions.

[kees]

The Linux kernel has just been using "#" for pre-ok diagnostics. For example:

1..2
# starting foo daemon...
# requesting bar
# halting foo daemon
ok 1 foo::bar
...

[Leont]

I think that ideally there would be some sort of being and end marker, and the body contains exactly one test result and 0 or more diagnostics. Anything else is either ambiguous or not backwards compatible.