testanything / testanything.github.io Goto Github PK
View Code? Open in Web Editor NEWTest anything protocol website
Home Page: testanything.org
License: Other
Test anything protocol website
Home Page: testanything.org
License: Other
According to http://www.nntp.perl.org/group/perl.qa/2009/03/msg12165.html it seems that nested test notation similar to this:
1..3
ok 1
1..3
ok 1
ok 2
ok 1
ok 2
ok 3
ok 4
1..4
ok 3 - some name
ok 2 - some name
ok 3
ok
would not be too difficult to support. But the official site linked from Wikipedia: Test Anything Protocol does not mention this as the way to go.
Is there any other notation suggested for nested tests? If not then can you add this one to the specification?
I'm writing a TAP consumer. While attempting to make it conform to the TAP version 13 spec I noticed a contradiction in the section on plan lines.
Second 'graf: "It must appear once, whether at the beginning or end of the output." But after the example, "The plan is optional but if there is a plan before the test points..."
So which is it?
there are 3 C TAP producers.
the page of A says B seems better and there is no information about how it compares to C. we could contact the authors and ask them to clarify?
thanks for all the work
BUGS
Please report any bugs or feature requests to bug-tap at rt.cpan.org, or through the web interface at http://rt.cpan.org/NoAuth/ReportBug.html?Queue=TAP. I will be notified, and then you'll automatically be notified of progress on your bug as I make changes.
This is wrong and also the list isn't accessible either.
TAP is great. Simplicity is its strength. Adding complexity should trigger caution. That said, I would suggest extending the directives capability.
The objective is to allow test harnesses to adjust their testing suites to their purposes without reinventing the wheel. The need arose for us when we wanted to integrate our TAP based testing in our devops workflow, binding a test run to a software release. All our projects are hosted on Github (whether open source or enterprise) and we wanted to keep track of associated issues, projects, milestones and tags.
All this could be achieved by simply adding custom directives to our existing GNU Autotools based build process and qualifying the build : nightly build, latest stable, release x.y.z, ...
Likewise, by binding our tests to specific issues, we can assert that bugs have been fixed, and that new features have been integrated; and that everything has tested.
Here are my suggestions:
This allows, in particular, to adjust test suites to devops workflows. In the example below we introduce a PATCH
directive which is synonymous to TODO
on next patch release, and a FEATURE
directive which is synonymous to SKIP
unless preparing next major/minor release. The ISSUE
directive binds a given test line to a reported issue, for instance on Github.
not ok 21 # PATCH patch-856 Git branch for patch
# ISSUE 123456789 Memory leak
# ISSUE 123456798 Invalid linked array
not ok 22 # FEATURE 12.1 New feature for release 12.1
# ISSUE 123456987 Command line option
# - ISSUE 123456987 This is a comment and not a directive
On could argue that such information can easily be integrated in an associated YAML block. I would disagree, for two major reasons:
Directives are part of the test harness as they bring light on the true/false status of a test, categorising it and eventually inverting the result. A given test harness will not operate optimally without its specific directives, whereas it can probably live without YAML blocks.
But more importantly, YAML blocks incur significant parsing overhead, while parsing directives, thanks to the TAP syntax, is far easier -- almost a one liner in Korn shell.
Non Proliferation
TAP producer authors should be aware of (and hopefully sign) the TAP Namespace Nonproliferation Treaty
These comments in the pages have existed for a long time.
I would like to remove them because:
Discuss!
The greatest single-file C test framework comes with an awk script (mentioned in the README) that will convert its verbose output into TAP version 13.
(My understanding is that the TAP support is implemented as an external filter because the author is trying to keep greatest.h
small.)
If you're including cmocka in your list, I'd highly recommend also including greatest.
cmocka is a complex framework that's, at best, difficult to set up for unsupported compilers. When I reviewed a bunch of test frameworks with the intent to use them for DOS hobby projects using Open Watcom C/C++, greatest was the one I recommended as the best blend of compatibility (just needs C89, no dynamic allocation, builds and runs for a real-mode DOS target), ease of use (single file), and features.
(I've included a screenshot in my review of it, and a code sample which didn't generate it... but only because I wanted the screenshot to show more than one or two tests and I didn't want the code example to be long and repetitive.)
The key takeaway from my review was "For something that works with Open Watcom C/C++ v1.9, the experience is surprisingly reminiscent of Python’s standard library unittest
module."
For implementers, please provide a document containing list of breaking changes (or compatibility issues/concerns) between the two specification versions.
I propose that Test::More-style subtests be mentioned in some manner in the specification.
That is, for an example:
1..3
ok 1 - First test
# Subtest: An example subtest
1..2
ok 1 - This is a subtest
ok 2 - So is this
ok 2 - An example subtest
ok 3 - Third test
Regular Test::Simple
/Test::More
/... tests usually output their diagnostic messages after a failed test. I.e.:
1..5
ok 1 - new() works
ok 2 - Title() get
ok 3 - Director() get
not ok 4 - Rating() get
# Failed test 'Rating() get'
# in t/film.t at line 14.
ok 5 - NumExplodingSheep() get
# Looks like you failed 1 tests of 5
I.e., if a TAP runner wishes to report debug information about the failed test to the programmer, it would usually report what follows the given test. However, for subtests, reporting the output of the subtest itself in addition to the final end result seems like it'd provide the developer with better output than simply reporting what is output after the failed test.
While a TAP-parser can function perfectly fine not knowing about these, someone writing a parser would probably want to know about this, so that they could provide proper support.
The printing of result is inside quotes which will fail to interpolate ->
.
print $out_file "$file results: $result->as_string\n";
Also, not sure but the aggregate doesn't seem to work.
For TAP13, the specification mentions:
Currently (2007/03/17) the format of the data structure represented by a YAML block has not been standardized.
Since that was over a decade ago, was it ever standardized?
It is impossible to give a direct link to Python implementation on this page - http://testanything.org/producers.html
There are seem to be few ways to do this:
In the examples in the specification, they're all indented several spaces. In one example, under "Unknown amount and failures", the TAP Version 13
is indented one space more than the lines that follow, which I think is invalid.
TAP version 13
ok 1 - retrieving servers from the database
# need to ping 6 servers
In order to make it easier to copy and paste these examples into test programs, it would be better if they were not indented with spaces. If the spacing is for visual esthetics, then CSS would be a better tool.
The Qt Test testing framework [0] supports output in TAP format, when the test binary is passed the -tap
command-line option [1]. (I would welcome any bug reports against the output it produces, if it turns out to be out of date or otherwise imperfect.) A mention on your list of producers would be welcome ;^>
[0] https://doc.qt.io/qt-6/qttest-index.html
[1] https://doc.qt.io/qt-6/qtest-overview.html#logging-options
There are a lot of resources now for using TAP with Node.js. Specifically
Tap-producing test hardness for Node -- https://github.com/substack/tape
Nodeschool workshop tutorial in progress -- https://github.com/tomgco/tapeshopper
Would be great to add these resources to the website.
This page appears to now be missing from the site:
http://testanything.org/How_TAP_was_named.html
This is linked from:
http://testanything.org/history.html
I'm writing yet another C library for TAP, and since I come from Perl, I've also been using Test::More's POD on metacpan (https://metacpan.org/pod/Test::More) for documentation. However, Test::More has a todo_skip feature that combines both TODO and SKIP into one crazy feature, but the documentation for TAP on this website (https://testanything.org/tap-version-13-specification.html) makes no mention of this feature.
I wanted to know what this feature looks like in actual TAP output, but getting there in the newer Test2 based version is harder, but backpan has older releases. In the source from the last release before Test2, Test::Builder's todo_skip() method's code indicates that it would generate TAP output something like: (https://metacpan.org/source/RJBS/Test-Simple-0.99/lib/Test/Builder.pm)
not ok <testnum> # TODO & SKIP <why>
This seems significantly different than regular TODO and SKIP, so TAP parsers not familiar with todo_skip will probably just parse it as a TODO and include the "& SKIP" part in the part. However, in todo_skip's documenation in Test::More, it says that todo_skip works like a SKIP, but parsers unaware of todo_skip will most likely treat it like a TODO that passed.
Should todo_skip() be documented? It's existed for 16 years since 2001, and nobody has pointed this out to you guys. Should todo_skip() be added to the TAP history page? (http://testanything.org/history.html)? Could you "backdate" adding it with one of the other version numbers, because you probably couldn't get away with adding it as version 14, but maybe that would actually work just fine. Do any other TAP libraries in other nonPerl languages support todo_skip?
With the advent of GitHub's PR review capability, I think that LGTM could be removed. As it stands, it's not currently working anyway. I've approved PRs and the service is not recognizing the approval even though I'm on the MAINTAINERS list. In those cases, I've had to override some angry looking warnings from the GitHub PR system.
The current TAP copyrights are 'wrong' and wouldn't be binding in any court of law due to the work having been done by far more contributors than is in the list and actually even Larry Wall should have some part of the copyright also.
Copyright 2003-2007 by Michael G Schwern [email protected], Andy Lester [email protected], Andy Armstrong [email protected]. This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See http://www.perl.com/perl/misc/artistic.html.
I suggest a move to:
This would obviously need agreement etc but ensures future development can drive forward with more people acknowledged.
/cc @isaacs
The current example from the Cookbook doesn't seem to aggregate results properly but simply contains the latest one. Perhaps it should be changed like this:
my $aggregate = TAP::Parser::Aggregator -> new;
for my $file (@files){
my $parser = TAP::Parser -> new({source => $file});
while(my $result = $parser -> next){
printf $out_file "$file results: %s\n", $result -> as_string;
}
printf $out_file "\n\tPlanned: %s\n\tPassed: %s\n\tFailed: %s\n\n", $parser -> tests_planned, scalar $parser -> passed, scalar $parser -> failed;
$aggregate -> add($file, $parser);
}
printf $out_file "\t---\nTotal: %s\nPlanned: %s\nPassed: %s\nFailed: %s\n",
$aggregate -> total, scalar $aggregate -> planned, scalar $aggregate -> passed, scalar $aggregate -> failed;
Thanks!
there is a tap awesome list by @mjlinxwiler and @sindresorhus. this page should be linked or even better: the content could be merged on the testanything site?
what do you think?
Hi!
I've created a fork of libtap++, and applied patches that allow it to work with modern C++ standarts:
https://gitlab.com/dhyannataraj/libtappp
I am using libtap++ in my project, and I guess I will support it for a while.
As of June 4, 2017 I am the new owner of the MyTAP repository. Please update the link from theory.github.io/mytap to hepabolu.github.io/mytap
I saw reference to a Slack group for the first time in an issue comment. This is not documented on the site and it's not clear how to get on there. When I clicked the link, it looked like I have to be added by an administrator.
I think the site would benefit from a page that describes how to get involved with TAP, the spec, the site, whatever.
The is an issue with the site which has come up several times already with comments and bugs namely: #27
The content under: http://testanything.org/testing-with-tap/
Is replicated / split to the consumers and producers page. I think this will solve the main goal of #9 without needing non standard headings.
I suggest the following:
in C, C++, Python, PHP, Perl, Java, JavaScript, and others.
What do you think @kinow?
SCSS-Lint was added to the Ruby section. The tool was written in Ruby, but it's not for Ruby. It's for Sass. I think Sass should get its own section.
GitHub is deprecating its other Markdown libraries available for GitHub pages. The TestAnything site will need to switch to kramdown before May 1, 2016.
I'm not sure if the site uses anything specific to redcarpet so it may be as simple of a switch as updating the config file.
https://github.com/blog/2100-github-pages-now-faster-and-simpler-with-jekyll-3-0
GitHub upgraded Jekyll to version 3.0. In the process, they dropped support for Pygments (which is a shame because it's language support is way better than Rouge). https://github.com/blog/2100-github-pages-now-faster-and-simpler-with-jekyll-3-0
The config needs to be updated to use rouge.
Working with my intern last summer, we wrote a simple CI/CD dashboard that conforms to NASA's agency restrictions and guidelines to assist project level unit and regression testing. We are almost though the open-source release process. When we get approval, I would like to see if someone can do a review before it is fully released to the public. Any takers? As a note, this will, as far as I know, end up being an official NASA open source product.
I'm guessing Coraline does not want to receive emails about any incidents with the Test Anything group. The email in the code of conduct should change to something/someone relevant.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.