I'm guessing Coraline does not want to receive emails about any incidents with the Test Anything group. The email in the code of conduct should change to something/someone relevant.

The greatest single-file C test framework comes with an awk script (mentioned in the README) that will convert its verbose output into TAP version 13.

(My understanding is that the TAP support is implemented as an external filter because the author is trying to keep greatest.h small.)

If you're including cmocka in your list, I'd highly recommend also including greatest.

cmocka is a complex framework that's, at best, difficult to set up for unsupported compilers. When I reviewed a bunch of test frameworks with the intent to use them for DOS hobby projects using Open Watcom C/C++, greatest was the one I recommended as the best blend of compatibility (just needs C89, no dynamic allocation, builds and runs for a real-mode DOS target), ease of use (single file), and features.

(I've included a screenshot in my review of it, and a code sample which didn't generate it... but only because I wanted the screenshot to show more than one or two tests and I didn't want the code example to be long and repetitive.)

The key takeaway from my review was "For something that works with Open Watcom C/C++ v1.9, the experience is surprisingly reminiscent of Python’s standard library unittest module."

Hi!
I've created a fork of libtap++, and applied patches that allow it to work with modern C++ standarts:

https://gitlab.com/dhyannataraj/libtappp

I am using libtap++ in my project, and I guess I will support it for a while.

The Qt Test testing framework [0] supports output in TAP format, when the test binary is passed the -tap command-line option [1]. (I would welcome any bug reports against the output it produces, if it turns out to be out of date or otherwise imperfect.) A mention on your list of producers would be welcome ;^>

[0] https://doc.qt.io/qt-6/qttest-index.html
[1] https://doc.qt.io/qt-6/qtest-overview.html#logging-options

The current example from the Cookbook doesn't seem to aggregate results properly but simply contains the latest one. Perhaps it should be changed like this:

my $aggregate = TAP::Parser::Aggregator -> new;

for my $file (@files){
    my $parser = TAP::Parser -> new({source => $file});
    while(my $result = $parser -> next){
        printf $out_file "$file results: %s\n", $result -> as_string;
    }
    printf $out_file "\n\tPlanned: %s\n\tPassed: %s\n\tFailed: %s\n\n", $parser -> tests_planned, scalar $parser -> passed, scalar $parser -> failed;
    $aggregate -> add($file, $parser);
}

printf $out_file "\t---\nTotal: %s\nPlanned: %s\nPassed: %s\nFailed: %s\n", 
    $aggregate -> total, scalar $aggregate -> planned, scalar $aggregate -> passed, scalar $aggregate -> failed;

Thanks!

there is a tap awesome list by @mjlinxwiler and @sindresorhus. this page should be linked or even better: the content could be merged on the testanything site?

what do you think?

According to http://www.nntp.perl.org/group/perl.qa/2009/03/msg12165.html it seems that nested test notation similar to this:

1..3
ok 1
   1..3
   ok 1
   ok 2
      ok 1
      ok 2
      ok 3
      ok 4
      1..4
   ok 3 - some name
ok 2 - some name
ok 3
ok

would not be too difficult to support. But the official site linked from Wikipedia: Test Anything Protocol does not mention this as the way to go.

Is there any other notation suggested for nested tests? If not then can you add this one to the specification?

The is an issue with the site which has come up several times already with comments and bugs namely: #27

The content under: http://testanything.org/testing-with-tap/

Is replicated / split to the consumers and producers page. I think this will solve the main goal of #9 without needing non standard headings.

I suggest the following:

• Change the producers and consumers page to be dedicated to explaining the use of these
  • Add links to all the language specific pages within the consumer and producer pages
• Change the language specific pages to describe the available resourced for those languages
  • Add the links to the language specific pages on the home page for the most populated pages in C, C++, Python, PHP, Perl, Java, JavaScript, and others.
• Implement a catch all page for the languages that are too short on resources for a whole page

What do you think @kinow?

GitHub upgraded Jekyll to version 3.0. In the process, they dropped support for Pygments (which is a shame because it's language support is way better than Rouge). https://github.com/blog/2100-github-pages-now-faster-and-simpler-with-jekyll-3-0

The config needs to be updated to use rouge.

This page appears to now be missing from the site:

http://testanything.org/How_TAP_was_named.html

This is linked from:
http://testanything.org/history.html

Non Proliferation
TAP producer authors should be aware of (and hopefully sign) the TAP Namespace Nonproliferation Treaty

These comments in the pages have existed for a long time.

I would like to remove them because:

• I don't think it helps making TAP a usable format across other languages
• I don't think it actually helps solve any issue
• I think it makes the TAP community look closed and require a license somewhat

Discuss!

I'm writing yet another C library for TAP, and since I come from Perl, I've also been using Test::More's POD on metacpan (https://metacpan.org/pod/Test::More) for documentation. However, Test::More has a todo_skip feature that combines both TODO and SKIP into one crazy feature, but the documentation for TAP on this website (https://testanything.org/tap-version-13-specification.html) makes no mention of this feature.

I wanted to know what this feature looks like in actual TAP output, but getting there in the newer Test2 based version is harder, but backpan has older releases. In the source from the last release before Test2, Test::Builder's todo_skip() method's code indicates that it would generate TAP output something like: (https://metacpan.org/source/RJBS/Test-Simple-0.99/lib/Test/Builder.pm)

not ok <testnum> # TODO & SKIP <why>

This seems significantly different than regular TODO and SKIP, so TAP parsers not familiar with todo_skip will probably just parse it as a TODO and include the "& SKIP" part in the part. However, in todo_skip's documenation in Test::More, it says that todo_skip works like a SKIP, but parsers unaware of todo_skip will most likely treat it like a TODO that passed.

Should todo_skip() be documented? It's existed for 16 years since 2001, and nobody has pointed this out to you guys. Should todo_skip() be added to the TAP history page? (http://testanything.org/history.html)? Could you "backdate" adding it with one of the other version numbers, because you probably couldn't get away with adding it as version 14, but maybe that would actually work just fine. Do any other TAP libraries in other nonPerl languages support todo_skip?

The printing of result is inside quotes which will fail to interpolate ->.

print $out_file "$file results: $result->as_string\n";

Also, not sure but the aggregate doesn't seem to work.

BUGS
Please report any bugs or feature requests to bug-tap at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=TAP. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.

This is wrong and also the list isn't accessible either.

For TAP13, the specification mentions:

Currently (2007/03/17) the format of the data structure represented by a YAML block has not been standardized.

Since that was over a decade ago, was it ever standardized?

Could I ask for adding two Python projects:

• bayeux is a module generating TAP stream (includes also unittest_tap which works as a shell over the standard Python unittest package)
• yamlish is a shell over PyYaml for parsing and generating Yamlish stream.

In the examples in the specification, they're all indented several spaces. In one example, under "Unknown amount and failures", the TAP Version 13 is indented one space more than the lines that follow, which I think is invalid.

    TAP version 13
   ok 1 - retrieving servers from the database
   # need to ping 6 servers

In order to make it easier to copy and paste these examples into test programs, it would be better if they were not indented with spaces. If the spacing is for visual esthetics, then CSS would be a better tool.

https://nodejs.org/dist/latest-v18.x/docs/api/test.html

I'm writing a TAP consumer. While attempting to make it conform to the TAP version 13 spec I noticed a contradiction in the section on plan lines.

Second 'graf: "It must appear once, whether at the beginning or end of the output." But after the example, "The plan is optional but if there is a plan before the test points..."

So which is it?

https://github.com/substack/tape

With the advent of GitHub's PR review capability, I think that LGTM could be removed. As it stands, it's not currently working anyway. I've approved PRs and the service is not recognizing the approval even though I'm on the MAINTAINERS list. In those cases, I've had to override some angry looking warnings from the GitHub PR system.

There are a lot of resources now for using TAP with Node.js. Specifically

Tap-producing test hardness for Node -- https://github.com/substack/tape
Nodeschool workshop tutorial in progress -- https://github.com/tomgco/tapeshopper

Would be great to add these resources to the website.

https://github.com/hadley/testthat
and
https://cran.r-project.org/web/packages/RUnit/index.html
??

For implementers, please provide a document containing list of breaking changes (or compatibility issues/concerns) between the two specification versions.

I saw reference to a Slack group for the first time in an issue comment. This is not documented on the site and it's not clear how to get on there. When I clicked the link, it looked like I have to be added by an administrator.

I think the site would benefit from a page that describes how to get involved with TAP, the spec, the site, whatever.

See https://github.com/substack/tape

As of June 4, 2017 I am the new owner of the MyTAP repository. Please update the link from theory.github.io/mytap to hepabolu.github.io/mytap

It is impossible to give a direct link to Python implementation on this page - http://testanything.org/producers.html

There are seem to be few ways to do this:

1. (templates)
   http://www.testically.org/2011/11/09/github-pages-how-to-do-a-simple-in-page-anchor-navigation-with-jekyll-and-liquid/
2. (jQuery)
   http://ben.balter.com/2014/03/13/pages-anchor-links/
3. (manual markup + css)
   http://stackoverflow.com/questions/6695439/how-do-you-create-link-to-a-named-anchor-in-multimarkdown

GitHub is deprecating its other Markdown libraries available for GitHub pages. The TestAnything site will need to switch to kramdown before May 1, 2016.

I'm not sure if the site uses anything specific to redcarpet so it may be as simple of a switch as updating the config file.

https://github.com/blog/2100-github-pages-now-faster-and-simpler-with-jekyll-3-0

SCSS-Lint was added to the Ruby section. The tool was written in Ruby, but it's not for Ruby. It's for Sass. I think Sass should get its own section.

TAP is great. Simplicity is its strength. Adding complexity should trigger caution. That said, I would suggest extending the directives capability.

The objective is to allow test harnesses to adjust their testing suites to their purposes without reinventing the wheel. The need arose for us when we wanted to integrate our TAP based testing in our devops workflow, binding a test run to a software release. All our projects are hosted on Github (whether open source or enterprise) and we wanted to keep track of associated issues, projects, milestones and tags.

All this could be achieved by simply adding custom directives to our existing GNU Autotools based build process and qualifying the build : nightly build, latest stable, release x.y.z, ...

Likewise, by binding our tests to specific issues, we can assert that bugs have been fixed, and that new features have been integrated; and that everything has tested.

Here are my suggestions:

• Directive tags are no longer case sensitive and must be uppercase
• Directives are introduced by a hash symbol followed by whitespace
• As per TAP13, directives can be followed by a textual description, provided that string is preceded by at least one whitespace character
• Directives can be added to the test line (as per TAP13) or immediately follow a test line
• Each test line can have multiple associated directives, one per line
• Directives are not pre-defined in the TAP specifications (except perhaps, historical TODO and SKIP)
• Harnesses only process directives they know about and ignore others (à la HTML/SGML)

This allows, in particular, to adjust test suites to devops workflows. In the example below we introduce a PATCH directive which is synonymous to TODO on next patch release, and a FEATURE directive which is synonymous to SKIP unless preparing next major/minor release. The ISSUE directive binds a given test line to a reported issue, for instance on Github.

not ok 21 # PATCH patch-856 Git branch for patch
# ISSUE 123456789 Memory leak
# ISSUE 123456798 Invalid linked array

not ok 22 # FEATURE 12.1 New feature for release 12.1
# ISSUE 123456987 Command line option
# - ISSUE 123456987 This is a comment and not a directive

On could argue that such information can easily be integrated in an associated YAML block. I would disagree, for two major reasons:

1. Directives are part of the test harness as they bring light on the true/false status of a test, categorising it and eventually inverting the result. A given test harness will not operate optimally without its specific directives, whereas it can probably live without YAML blocks.

2. But more importantly, YAML blocks incur significant parsing overhead, while parsing directives, thanks to the TAP syntax, is far easier -- almost a one liner in Korn shell.

Working with my intern last summer, we wrote a simple CI/CD dashboard that conforms to NASA's agency restrictions and guidelines to assist project level unit and regression testing. We are almost though the open-source release process. When we get approval, I would like to see if someone can do a review before it is fully released to the public. Any takers? As a note, this will, as far as I know, end up being an official NASA open source product.

I propose that Test::More-style subtests be mentioned in some manner in the specification.

That is, for an example:

1..3
ok 1 - First test
    # Subtest: An example subtest
    1..2
    ok 1 - This is a subtest
    ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test

Regular Test::Simple/Test::More/... tests usually output their diagnostic messages after a failed test. I.e.:

1..5
ok 1 - new() works
ok 2 - Title() get
ok 3 - Director() get
not ok 4 - Rating() get
#   Failed test 'Rating() get'
#   in t/film.t at line 14.
ok 5 - NumExplodingSheep() get
# Looks like you failed 1 tests of 5

I.e., if a TAP runner wishes to report debug information about the failed test to the programmer, it would usually report what follows the given test. However, for subtests, reporting the output of the subtest itself in addition to the final end result seems like it'd provide the developer with better output than simply reporting what is output after the failed test.

While a TAP-parser can function perfectly fine not knowing about these, someone writing a parser would probably want to know about this, so that they could provide proper support.

The current TAP copyrights are 'wrong' and wouldn't be binding in any court of law due to the work having been done by far more contributors than is in the list and actually even Larry Wall should have some part of the copyright also.

Copyright 2003-2007 by Michael G Schwern [email protected], Andy Lester [email protected], Andy Armstrong [email protected]. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See http://www.perl.com/perl/misc/artistic.html.

I suggest a move to:

• Assign the rights to a group which has governing power over the TAP specifications
  • or make it a copyleft with the agreement
• Make a separate acknowledgement page for (authors, acknowledgements and copyright) which needs mention to old contributors and perhaps automated from github for new contributions.

This would obviously need agreement etc but ensures future development can drive forward with more people acknowledged.

/cc @isaacs

there are 3 C TAP producers.

• https://www.shlomifish.org/open-source/projects/libtap/ (i call it A)
• http://github.com/zorgnax/libtap (i call it B)
• http://www.eyrie.org/~eagle/software/c-tap-harness/ (i call it C)

the page of A says B seems better and there is no information about how it compares to C. we could contact the authors and ask them to clarify?

thanks for all the work