Test anything protocol

Test reports that are detected as Test Anything Protocol (TAP) will be analyzed and transformed into the short tokens used by the extension. The initial subsystem will handle TAP version 13 and some previous versions. If the text fails to be interpreted as a valid TAP report the text will not be processed but used as is. A valid report does not imply that the report is actually parsed correctly, but that some specific markers exists.

Pickle will not recreate the test fixture for each run, thus tests will only progress until the first failure. Progressing beyond the first failure imply recreating the test fixture, which puts a much larger load on the system.

Parsing Test anything protocol are done in Github: jeblad/Pickle/includes/TAP.

Core concept

Processing of TAP reports attempts to verify whether any entry are failed (not ok), then checks if all are passed (ok), and then if any of skipped and todo are set. The purpose is to detect an overall result that will convey to the user the present state of the module. In that process a lot of information is thrown away, as it is only the final state that is important.

The output should be sufficient to be parsed by some generic TAP software, and it should also be fairly easy to create new testing frameworks. Those frameworks can be rather different from the pickle framework.

Examples

These are examples on how the reports should look like

All passed

Only a single “ok” is necessary to flag each test case as passed, and only a single “ok” is sufficient to flag the whole test set as passed.

TAP version 13
# description 1 - all ok
# description 2 - all ok
# description 3 - all ok
ok

Some failed

The tests that fail will force a full report at the actual level.

TAP version 13
# description 1 - all ok
# description 2
# it 1
ok
ok
not ok
ok
# it 2 - all ok
# description 3 - all ok

Variation

To ease interpretation the background color could be changed to convey the overall state. A reddish color would indicate failing tests, and a greenish color would indicate good tests. This can be done by selectors.

generated by LDoc 1.4.6