perl-toolchain-gang / test-harness Goto Github PK
View Code? Open in Web Editor NEWRun Perl standard test scripts with statistics
Home Page: http://testanything.org/
Run Perl standard test scripts with statistics
Home Page: http://testanything.org/
Test-Harness 3.24 INSTALLATION To install Test::Harness using ExtUtils::MakeMaker do: perl Makefile.PL make make test make install This will install Test::Harness and the "prove" program. Type prove --help for more information. COPYRIGHT AND LICENCE Copyright (C) 2006, 2007 Curtis "Ovid" Poe This program is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
Tests may be run via prove directly or via make
with make test
.
In latter case, it is not possible to pass additional arguments to the script.
But this may be necessary for the IDE integration. IDE may need to pass it's own formatter, plugin or modify execution mode, like jobs, shufflle, etc.
Makefile.PL
should not depend on development environment, because different users may have different preferences, so would be nice to be able to pass additional arguments via environment variables.
One thing that bothers me is that these variable are expected to have higher precedence than command line argument, so IDE could tune execution, but this feels wrong... May be two variables could be introduced, e.g PROVE_OPTS
and FORCE_PROVE_OPTS
where first has lesser precedence and latter - higher.
I want to develop a prove plugin that needs more flexibility with respect to the arguments passed to the plugin. Especially the comma should at some places be used as is. Currently there exists now way to escape it.
my @args = ();
if ( $name =~ /^(.*?)=(.*)/ ) {
$name = $1;
@args = split( /,/, $2 );
}
Do you see any way to implement this? A number of different approaches are described here split on delimiter unless escaped.
CC @rocky
Over at Test-More/test-more#319 we've been talking about ways to get Test::More to do the whole "stop on fail" thing. What occurred to me is that while stopping immediately after a failed result has issues, stopping the test suite at the end of a failing test is easy and it solves 90% of the problem.
The problems being...
Test::Harness can have a prove flag/environment variable which stops running the rest of the .t files if one of them fails. Like an implied BAIL_OUT. Then a user could do something like...
HARNESS_BAIL_ON_FAIL=1 cpan Catalyst
Unless a failing test file is very large or very slow, this should greatly reduce the amount of failure garbage and the time spent completing failing test suites. It also doesn't matter how the tests are written.
What say?
Hi, I have some tests that produce the following output (along with other stuff), I'm planning to extend our ci in openQA to have a bit of better support for console based tests, however, while playing with Test::Harness, I'm struggling to figure out how subtests are being processed... I could find that there are some tests that are trying to parse at least subtests but nothing else.
I could not find any docs regarding Subtests in the TAP docs or related sites (perhaps I didn't look well enough?)
Any hints?
So long story short, I'm loading the file with code like this:
use TAP::Parser;
use feature 'say';
use Carp;
my $parser = TAP::Parser->new({tap => "t/source_tests/tap_format_example_complex.tap"});
confess "Failed " . $parser->parse_errors if $parser->parse_errors;
while ( my $result = $parser->next ) {
say $result->type . ' - '.$result->as_string;
}
And upon running my simple test, I get:
test - not ok 24 - group filter
test - ok 25 - POST /api/v1/isos
test - ok 26 - 200 OK
test - not ok 27 - 10 new jobs created
test - ok 28 - GET /api/v1/jobs
test - not ok 29 - server_32 is only parent of client1_32
test - not ok 30 - server_32 is only parent of client2_32
test - not ok 31 - server_32 has no parents
test - ok 32 - kde is not created for 32bit machine
test - ok 33 - advanced_kde is not created for 32bit machine
test - not ok 34 - server_64 is only parent of client1_64
test - not ok 35 - server_64 is only parent of client2_64
test - not ok 36 - server_64 has no parents
test - ok 37 - no (unexpected) warnings (via END block)
unknown - Dubious, test returned 255 (wstat 65280, 0xff00)
unknown - Failed 9/37 subtests
unknown -
unknown - Test Summary Report
unknown - -------------------
unknown - t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
unknown - Failed tests: 23-24, 27, 29-31, 34-36
unknown - Non-zero exit status: 255
unknown - Parse errors: No plan found in TAP output
unknown - Files=1, Tests=37, 9 wallclock secs ( 0.02 usr 0.01 sys + 5.05 cusr 0.18 csys = 5.26 CPU)
unknown - Result: FAIL
Also the following results not being shown is kinda strange... :
File:
t/api/02-iso.t ..
ok 1 - GET /api/v1/jobs/99927
...
ok 21 - 200 OK
ok 22 - we have no gru download tasks to start with
# Subtest: Multi Machine job dependencies
ok 1 - POST /api/v1/isos
not ok 2 - 200 OK
not ok 3 - only one job created due to group filter
1..3
not ok 23 - Multi Machine job dependencies
# Subtest: group filter
ok 1 - POST /api/v1/isos
ok 2 - 200 OK
ok 3 - no jobs created if group invalid
ok 4 - POST /api/v1/isos
ok 5 - 200 OK
not ok 6 - only one job created due to group filter
ok 7 - POST /api/v1/isos
ok 8 - 200 OK
not ok 9 - only one job created due to group filter (by ID)
1..9
not ok 24 - group filter
...
ok 32 - kde is not created for 32bit machine
ok 33 - advanced_kde is not created for 32bit machine
not ok 34 - server_64 is only parent of client1_64
not ok 35 - server_64 is only parent of client2_64
not ok 36 - server_64 has no parents
ok 37 - no (unexpected) warnings (via END block)
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 9/37 subtests
Test Summary Report
-------------------
t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
Failed tests: 23-24, 27, 29-31, 34-36
Non-zero exit status: 255
Parse errors: No plan found in TAP output
Files=1, Tests=37, 9 wallclock secs ( 0.02 usr 0.01 sys + 5.05 cusr 0.18 csys = 5.26 CPU)
Result: FAIL
Test-Harness used to come with a Makefile.PL and a NotBuild.PL. The Makefile.PL forces you to use NotBuild.PL for authoring purposes (in particular, for the dist and distmeta actions). I don't see the usefulness of this setup.
Every time aggregate_tests is called, it calls prepare, which sets $longest to 0, finds the longest test description, and then sets the _longest attribute.
If aggregate_tests is called multiple times with the same formatter, then _longest can end up being less than the actual longest description, causing a negative repeat count warning.
The fix to this would be to start with the current value of _prepare instead of starting at 0 every time.
Thanks for creating and maintaining some great modules!
After some recent upgrades, the --archive option to prove seems to be broken:
$ prove --archive /tmp/test.tgz
Can't locate object method "new" via package "TAP::Harness::Archive::SUPER" at /usr/local/share/perl/5.14.2/TAP/Harness/Archive.pm line 96.
Line 96 looks like this:
my $self = $class->SUPER::new($args);
Versions:
OS: Debian 7.6
Perl: v5.14.2 built for x86_64-linux-gnu-thread-multi
TAP::Harness 3.33
TAP::Harness::Archive 0.15
I have tested this with TAP::Harness::Archive 0.15 and TAP::Harness 3.23, and that seems to work. Not quite sure if this is mainly a problem with TAP::Harness or TAP::Harness::Archive...
Please let me know if more details are needed.
This is currently not listed as a valid argument to TAP::Harness->new(). I think it should be documented there - both for completeness and because it's incredibly useful. Or at least link to the Test::Harness::Beyond page.
I guess App::Prove::Plugin::* needs more hook points like 'after_test', 'after_runtests'.
For example, App::Prove::Plugin::Growl hacks to get a testing result.
And, I want to write a plugin to post testing result to report server.
This ticket continues the discussion begun in the Perl issue tracker at Perl/perl5#21455. I'm moving the discussion here because:
My analysis indicates that the problem lies in Test-Harness's own test suite and is merely reproduced when Test-Harness is synched into core; and
Test-Harness's documentation on CPAN indicates that bug tickets should be filed on RT, but that queue consists of 62 old reports and I suspect filing here in the Perl-Toolchain-Gang organization will get more attention. (If you want me to file on RT instead, just let me know.)
The discussion so far
Briefly, the discussion in Perl/perl5#21455: For the last month, when I configure, build and test Perl on FreeBSD (and other OSes) and then call git status
at the end of the test suite, the status output includes:
Untracked files:
(use "git add <file>..." to include in what will be committed)
cpan/Test-Harness/perl.core
This core dump does not cause any test failures and appears to be compiler-independent, but is nonetheless real:
-rw------- 1 jkeenan jkeenan 11415552 Sep 5 21:12 cpan-Test-Harness.freebsd.clang14.perl.core
I do not observe this core dump on Linux: neither by ls
nor by git status
.
I haven't definitively bisected this problem to see when it entered the Perl 5 core distribution, but I strongly suspect that it occurred when I synched Test-Harness-3.46 into core on August 13.
commit 6e0b8bd69336ecc95a6ca480c4b2ab58c0080bc2
Author: Leon Timmermans <[email protected]>
AuthorDate: Sat Aug 12 17:05:48 2023 -0400
Commit: James E Keenan <[email protected]>
CommitDate: Sun Aug 13 06:56:43 2023 -0400
cpan/Test-Harness - Update to version 3.46
I performed that synching on Linux. All tests passed. git status
was clean, so I had no reason to repeat that process on any other OS. If I weren't regularly building and testing on FreeBSD-13, I wouldn't have stumbled across this problem. My expectation is that on any (non-Windows, at least) OS to which I have access, after I run perl's make test_harness
, all tests should PASS and git status
should show no untracked files.
Research in the Test-Harness repository on GitHub
I brought my own fork of Test-Harness on GitHub up-to-date with PTC's repository, then created a branch so that I could isolate the code generating the segfault in a way by which I could compare Linux and FreeBSD output more precisely. That 'segfault-analysis-20230908' branch is here.
I acknowledge that I have rarely peered into Test-Harness's code; hence, there may be weaknesses in the diagnostic code I added in my branch. All my changes are confined to t/harness.t
:
$ git diff -w master..segfault-analysis-20230908 -- t/ |cat
diff --git a/t/harness.t b/t/harness.t
index b84fa27..e81f23f 100644
--- a/t/harness.t
+++ b/t/harness.t
@@ -6,6 +6,7 @@ BEGIN {
use strict;
use warnings;
+use Data::Dumper;
use Test::More;
use IO::c55Capture;
@@ -539,12 +540,39 @@ for my $test_args ( get_arg_sets() ) {
skip "ASAN doesn't passthrough SEGV", 1
if "$Config{cc} $Config{ccflags} $Config{optimize}" =~ /-fsanitize\b/;
+print STDERR "XXX:\n";
+warn "XXX1: perl.core file exists" if (-e './perl.core');
+
@output = ();
- _runtests( $harness_failures, "$sample_tests/segfault" );
+
+ # The following line creates './perl.core' on FreeBSD:
+ #
+ # _runtests( $harness_failures, "$sample_tests/segfault" );
+ #
+ # Our objective is to isolate the code generating the segfault so that
+ # we can understand why it (seemingly) creates a core dump on FreeBSD
+ # et al. but not on Linux.
+
+print STDERR "FFF1: \$harness_failures is a ", ref $harness_failures, "\n";
+print STDERR "FFF2: ", scalar keys (%$harness_failures), " elements in \$harness_failures\n";
+
+ # First, run only the TAP::Harness objects seen so far:
+ _runtests( $harness_failures );
+warn "XXX2: perl.core file exists" if (-e './perl.core');
+
+ # Next, run the segfault test:
+ # First argument to _runtests must be a TAP::Harness object (I think)
+
+ my $this_harness = $HARNESS->new;
+ _runtests( $this_harness, "$sample_tests/segfault" );
+warn "XXX3: perl.core file exists" if (-e './perl.core');
+
+print STDERR Dumper(\@output);
my $out_str = join q<>, @output;
like( $out_str, qr<SEGV>, 'SIGSEGV is parsed out' );
+print STDERR "YYY:\n";
}
#XXXX
All the changes are debugging code except splitting _runtests( $harness_failures, "$sample_tests/segfault" );
into two separate invocations of _runtests()
, for the second of which I had to create a new TAP::Harness object.
Results
If I build Test::Harness on each of Linux and FreeBSD and exercise the file generating the segfault on FreeBSD, I get these comparative results:
Ubuntu Linux 22.04 LTS at release-3.45_01-12-gffc5f49
$ perl Makefile.PL && make
...
Manifying 17 pod documents
$ rm perl.core; prove -b t/harness.t; ls -ltr . | tail
rm: cannot remove 'perl.core': No such file or directory
t/harness.t .. 1/133 XXX:
FFF1: $harness_failures is a TAP::Harness
FFF2: 12 elements in $harness_failures
$VAR1 = [
'Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'Result: NOTESTS',
't/sample-tests/segfault ..',
'No subtests run',
'Test Summary Report',
'-------------------',
't/sample-tests/segfault (Wstat: 139 (Signal: SEGV, dumped core) Tests: 0 Failed: 0)',
'Non-zero wait status: 139',
'Parse errors: No plan found in TAP output',
'Files=1, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'Result: FAIL'
];
YYY:
t/harness.t .. ok
All tests successful.
Files=1, Tests=133, 0 wallclock secs ( 0.01 usr 0.00 sys + 0.07 cusr 0.03 csys = 0.11 CPU)
Result: PASS
-rw-rw-r-- 1 jkeenan jkeenan 2244 Aug 12 16:45 Makefile.PL
-rw-rw-r-- 1 jkeenan jkeenan 6346 Aug 12 16:45 HACKING.pod
drwxrwxr-x 2 jkeenan jkeenan 4096 Aug 12 16:45 bin
-rw-rw-r-- 1 jkeenan jkeenan 44232 Sep 8 17:17 Changes
-rw-rw-r-- 1 jkeenan jkeenan 774 Sep 9 13:10 MYMETA.yml
-rw-rw-r-- 1 jkeenan jkeenan 1295 Sep 9 13:10 MYMETA.json
-rw-r--r-- 1 jkeenan jkeenan 45309 Sep 9 13:10 Makefile
drwxrwxr-x 8 jkeenan jkeenan 4096 Sep 9 13:10 blib
-rw-rw-r-- 1 jkeenan jkeenan 0 Sep 9 13:10 pm_to_blib
drwxrwxr-x 9 jkeenan jkeenan 4096 Sep 9 13:10 t
FreeBSD-13 at release-3.45_01-12-gffc5f49
$ perl Makefile.PL && make
...
Manifying 50 pod documents
$ rm perl.core; prove -b t/harness.t; ls -ltr . | tail
rm: perl.core: No such file or directory
t/harness.t .. 1/133 XXX:
FFF1: $harness_failures is a TAP::Harness
FFF2: 12 elements in $harness_failures
XXX3: perl.core file exists at t/harness.t line 568.
$VAR1 = [
'Files=0, Tests=0, 0 wallclock secs ( 0.00 usr + 0.00 sys = 0.00 CPU)',
'Result: NOTESTS',
't/sample-tests/segfault ..',
'No subtests run',
'Test Summary Report',
'-------------------',
't/sample-tests/segfault (Wstat: 139 (Signal: SEGV, dumped core) Tests: 0 Failed: 0)',
'Non-zero wait status: 139',
'Parse errors: No plan found in TAP output',
'Files=1, Tests=0, 0 wallclock secs ( 0.00 usr 0.01 sys + 0.00 cusr 0.01 csys = 0.02 CPU)',
'Result: FAIL'
];
YYY:
t/harness.t .. ok
All tests successful.
Files=1, Tests=133, 0 wallclock secs ( 0.04 usr 0.01 sys + 0.20 cusr 0.05 csys = 0.30 CPU)
Result: PASS
drwxr-xr-x 3 jkeenan jkeenan 3 Sep 8 21:21 reference
drwxr-xr-x 2 jkeenan jkeenan 26 Sep 8 21:21 smoke
drwxr-xr-x 4 jkeenan jkeenan 4 Sep 8 21:21 xt
-rw-r--r-- 1 jkeenan jkeenan 44090 Sep 9 17:05 Makefile
-rw-r--r-- 1 jkeenan jkeenan 1295 Sep 9 17:05 MYMETA.json
-rw-r--r-- 1 jkeenan jkeenan 774 Sep 9 17:05 MYMETA.yml
drwxr-xr-x 8 jkeenan jkeenan 8 Sep 9 17:05 blib
-rw-r--r-- 1 jkeenan jkeenan 0 Sep 9 17:05 pm_to_blib
-rw------- 1 jkeenan jkeenan 11337728 Sep 9 17:05 perl.core
drwxr-xr-x 9 jkeenan jkeenan 56 Sep 9 17:05 t
Note that only on FreeBSD do I get this statement in the output from my branch:
XXX3: perl.core file exists at t/harness.t line 568.
... and that a core dump is created only on FreeBSD, as shown from this line in ls
:
-rw------- 1 jkeenan jkeenan 11337728 Sep 9 17:05 perl.core
Inferences and Next Questions
t/sample-tests/segfault
:$ cat t/sample-tests/segfault
#!/usr/bin/perl
print "1..1\n";
print "ok 1\n";
kill 'SEGV', $$;
Why does this create a segfault file (./perl.core
) on FreeBSD but not on Linux?
On FreeBSD, Test-Harness's own make clean
(which appears to be a straightforward derivative from ExtUtils::MakeMaker) and git clean -dfx
will both remove the core dump file, after which git status
will be clean. So what is annoying is that, on one OS but not the other, calling git status
before running either of those cleanup commands will report an "untracked file." That annoyance would affect anyone doing maintenance work on Test-Harness, but nobody else. But this will be really annoying to anyone running Perl's own test suite on FreeBSD (or any other OS so affected).
Should we consider modifying t/harness.t
so that if it generates a core dump, the test file tidies up after itself and removes perl.core
once it's no longer needed?
Thank you very much.
Jim Keenan
until system ('set PATH=c:\Dwimperl\perl\bin;%PATH%');
it's strange, because, when I run in cmd it works
use FindBin '$RealBin';
system ('set PATH=c:\Dwimperl\perl\bin;%PATH%');
my $cmd= qq{prove --rc=.proverc :: --project ${dir_4_test} --path_2_test_dir "$RealBin/${dir_4_test}/jobs/Jobs" > ${dir_4_test}.html};
system ($cmd);
What --state does when combined with file arguments is underdocumented
and sometimes surprising. The following in the Test-Harness
distribution directory:
Options + Arguments | what is does | ok/surprise |
---|---|---|
rm .prove | ||
--state=slow,save | refuses to test | surprise[0] |
rm .prove | ||
--state=save | runs all tests | ok |
--state=save,fast t/parse.t | runs all tests | surprise[1] |
rm .prove | ||
--state=save,fast t/parse.t | runs parse.t | surprise[2] |
--state=save,fast t/base.t | runs parse.t and base.t | |
--state=save,all,fast t/base.t | runs parse.t and base.t | surprise[3] |
[0] because there are tests on disk and they would be run with
state=save alone; this is one of the cases where it would be helpful
to just run all tests from disk.
[1] what happens is that the argument t/parse.t is merged with all
files in .prove. Apparently there is no way to run a subset of the
known tests with state options without editing the .prove file.
[2] Given that surprise [0] above refused to work, it's now
surprising that this incantation does the right thing
[3] keyword "all" apparently does not mean all tests on disk. Not
mentioned in the docs.
I believe the currently undocumented rules are:
(1) as long as no statefile exists, any combination of state arguments
turn prove into a noop unless there are explicit file or directory
arguments on the commandline. E.g.
prove --state=save,all # noop
prove --state=save,all t/1.t # runs 1.t
(2) if a statefile exists, the set of tests to be considered to be run
is a union of the test files in the state file and files or
directories on the command line. From this union set all tests on
the command line are run unconditionally while files from the
statefile are run according to the rule. E.g.
rm -f .prove
prove --state=save t/1.t t/2.t # runs 1.t and 2.t
prove --state=save,failed t/1.t # runs 1.t and whatever failed above
(3) the keyword "all" refers to the files in the statefile, not to the
files on disk
I'm unsure whether one would prefer the rules to change or get these
rules documented. I find the rules hard to understand and to document
and also disputable.
This might be something others either ignore or maybe it doesn't happen often enough for anyone to say something. Or it's possible it's been mentioned but wontfix'ed because it breaks the flow of the routines.
I had the occasion to see it a few times when running prove manually from the wrong directory (prove -lv t/testfile.t when working directory was the t/ dir)
I'm not sure where a good place for the check could be, but it would be nice if this was smart enough to say if ( !fileexist && sourcerefundefined ) confess("file/method not found.");
Right now this is the error you get. It's factual but not very helpful.
rdrake@test:~/Test-Harness [master]$ perl -Ilib bin/prove asdf
Cannot detect source of 'asdf'! at lib/TAP/Parser/IteratorFactory.pm line 261.
TAP::Parser::IteratorFactory::detect_source(TAP::Parser::IteratorFactory=HASH(0x18d12c0), TAP::Parser::Source=HASH(0x18d11b8)) called at lib/TAP/Parser/IteratorFactory.pm line 211
TAP::Parser::IteratorFactory::make_iterator(TAP::Parser::IteratorFactory=HASH(0x18d12c0), TAP::Parser::Source=HASH(0x18d11b8)) called at lib/TAP/Parser.pm line 472
TAP::Parser::_initialize(TAP::Parser=HASH(0x18a7028), HASH(0x16bb7c0)) called at lib/TAP/Object.pm line 55
TAP::Object::new("TAP::Parser", HASH(0x16bb7c0)) called at lib/TAP/Object.pm line 130
TAP::Object::_construct(TAP::Harness=HASH(0x15e8e68), "TAP::Parser", HASH(0x16bb7c0)) called at lib/TAP/Harness.pm line 852
TAP::Harness::make_parser(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Scheduler::Job=HASH(0x188d160)) called at lib/TAP/Harness.pm line 651
TAP::Harness::_aggregate_single(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Aggregator=HASH(0x1687048), TAP::Parser::Scheduler=HASH(0x188d238)) called at lib/TAP/Harness.pm line 743
TAP::Harness::aggregate_tests(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Aggregator=HASH(0x1687048), "asdf") called at lib/TAP/Harness.pm line 558
TAP::Harness::__ANON__() called at lib/TAP/Harness.pm line 571
TAP::Harness::runtests(TAP::Harness=HASH(0x15e8e68), "asdf") called at lib/App/Prove.pm line 546
App::Prove::_runtests(App::Prove=HASH(0x1184bd0), HASH(0x15b4640), "asdf") called at lib/App/Prove.pm line 504
App::Prove::run(App::Prove=HASH(0x1184bd0)) called at bin/prove line 9
I use the -v switch in the prove utility.
DOC
but it has no effect. Then I dump my %ENV and found:
HARNESS_ACTIVE 1,
HARNESS_IS_VERBOSE 1,
HARNESS_VERSION "3.36_01",
HARNESS_IS_VERBOSE
is used instead of HARNESS_VERBOSE
The code comment refer to: https://rt.cpan.org/Public/Bug/Display.html?id=12458
Is this bad doc or bad fix?
--exec is documented in TAP::Harness as being something you might use to request a specific perl ... but when I do that, -I flags passed to the command line don't get handed through. i.e.
$ prove --exec $PERL_BINARY -I$LIB_PATH t
doesn't pass the -I through to the perl binary. Admittedly,
$ prove --exec $PERL_BINARY\ -I$LIB_PATH t
works fine since the --exec option is allowed to contain switches, but I can't find any documentation saying it's intentional to throw them away so I'm wondering if this is a bug.
Had a bug in my code because I assumed ->all_passed was the same as "everything is good". But this method returns false if all tests were skipped, like with plan skip_all => 'some reason'.
The end of the output here shows the ->all_passed
returned false when the test file had skipped all tests. I would've expected it to return true.
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -v | grep version
This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-linux-multi
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -MTAP::Harness -wlE 'say TAP::Harness->VERSION'
3.36
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ cat foo.t
use Test::More;
plan skip_all => 'some reason';
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -MTAP::Harness -wlE 'my $res = TAP::Harness->new->runtests("foo.t"); say "--------\nAll tests passed: " . $res->all_passed'
foo.t ..
foo.t ..
skipped: some reason
Files=1, Tests=0, 1 wallclock secs ( 0.05 usr 0.01 sys + 0.03 cusr 0.01 csys = 0.10 CPU)
Result: NOTESTS
--------
All tests passed: 0
zoffix@leliana:/tmp/tmp.jx6I5dc73b$
I was briefly surprised to discover that my regular unit tests run in parallel (with HARNESS_OPTIONS set to -j9 in the environment), yet my extra tests (run via dzil xtest, which uses App::Prove) chugged along serially.
I'll patch, if someone can point me to where/how this can be done.
I like to keep a tab open that runs my unit tests repeatedly. When I've made changes I can switch to that tab and see the current results of all my tests. Coloring green or red makes it really easy to know if I've broken something. I tried running prove
inside of a continuous watch
like so:
watch --color 'prove --color t/*.t'
It appears that prove
attempts to be smart and not output ANSI if the target isn't a TTY. This can also be shown with:
prove --color t/*.t > /tmp/output.txt
This is related to #85 and #101 but it's a more general issue.
First, there's a regression in a8fb517 (Make prove respect environmental variables #28, 2013-10-27), when we started paying attention to HARNESS_OPTIONS
in prove
having the options set to jN
where N > 1
suppresses the --verbose
output completely. E.g. (running this on the git.git
test suite):
HARNESS_OPTIONS=j2 perl -I ~/g/Test-Harness/lib ~/g/Test-Harness/bin/prove --verbose t0002-gitfile.sh :: -
-verbose
ok 1 - initial setup
ok 2 - bad setup: invalid .git file format
ok 3 - bad setup: invalid .git file path
ok 4 - final setup + check rev-parse --git-dir
[...]
But before a8fb517 it would show the verbose output, curiously if you pipe it to | cat
it shows up, so there's some isatty()
check somewhere.
Instead it should at the very least be smart enough to figure out that your -jN
setting doesn't matter in the case of a single test.
When you run N
tests at a time it will struggle with some ASCII-art to represent the concurrency, and then dump the non-verbose output (under --verbose
!) for whatever test it happened to run last:
$ HARNESS_OPTIONS= perl -I ~/g/Test-Harness/lib ~/g/Test-Harness/bin/prove --shuffle -j8 --verbose t000*
.sh :: --verbose
===( 1;0 0/? 1/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 5;0 0/? 1/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 11;0 0/? 1/? 0/? 0/? 0/? 0/? 0/? 0/? )================ok
===( 29;0 0/? 0/? 0/? 10/? 3/? 0/? 1/? )====================ok
===( 34;0 0/? 0/? 0/? 11/? 3/? 0/? )=========================ok
===( 83;0 19/? 11/? 0/? 18/? 1/? )============================ok
===( 180;0 55/? 29/? 0/? 36/? )=================================ok
===( 262;1 61/? 9/? 49/? )=======================================ok
===( 497;4 266/? 28/? )===========================================ok
t0008-ignores.sh .....
ok 343 - beyond a symlink from subdirectory with --verbose
ok 344 - beyond a symlink from subdirectory with --verbose -n
ok 345 - beyond a symlink from subdirectory with --verbose --non-matching
ok 346 - submodule
ok 347 - submodule with -q
[...]
There's a lot more than the "ok" output under --verbose
in that test, so the verbose output is being stripped. This is another case where piping it to | cat
gives expected results.
Not submitting this as a PR since I'm not familiar with the code, but this seems to fix the above issues:
diff --git a/lib/TAP/Harness.pm b/lib/TAP/Harness.pm
index a9a865d..105f846 100644
--- a/lib/TAP/Harness.pm
+++ b/lib/TAP/Harness.pm
@@ -453,6 +453,8 @@ Any keys for which the value is C<undef> will be ignored.
local $default_class{formatter_class} = 'TAP::Formatter::File'
unless -t ( $arg_for{stdout} || \*STDOUT ) && !$ENV{HARNESS_NOTTY};
+ local $default_class{formatter_class} = 'TAP::Formatter::File'
+ if $self->jobs > 1 && $arg_for{verbosity};
while ( my ( $attr, $class ) = each %default_class ) {
$self->$attr( $self->$attr() || $class );
I had opened houseabsolute/test-class-moose#108 on Test::Class::Moose
, but @exodist suggested that I also open a ticket here, since the problem described above may actually be an issue with prove
.
The original tickets demonstrates this issue and also contains a repository which can be used to replicate it.
Despite using select the whole architecture of TAP::Harness is based on blocking IO, which is a really poor combination with multiprocessing in particular. I'm not sure how to solve this. TAP::Parser is fundamentally a pull-architecture, while it should have been a push-based architecture. I'm not sure this can be fixed without rewriting TAP::Parser, TAP::Parser::Iterator*, TAP::Parser::Grammar and TAP::Parser::Multiplexer. The entire flow of control/data needs to be reversed :-(.
I have a small problem that crops up from time to time. Sometimes I write a TODO
test to temporarily stop a test failure until such time that I can get business signoff on a change. However, I have to remember to remove the TODO
test at a later date and having to manually handle this is fraught with error. I would like to be able to do this:
my $builder = Test::Builder->new;
$builder->todo_start( message => $message, until => 'YYYY-MM-DD' );
ok !defined $object->begin_thermonuclear_war,
'We should never allow thermonuclear war';
$builder->todo_end;
That subtly alters todo_start
behavior, but I think it can be made backwards-compatible (at the risk of additional complexity) with this untested code:
sub todo_start {
my $self = shift;
if (1 == @_) {
@_ = ( message => $_[0] );
}
my %arg_for = @_;
return if $self->_on_or_after_date($arg_for{until});
$self->{Start_Todo}++;
if( $self->in_todo ) {
push @{ $self->{Todo_Stack} } => $self->todo;
}
$self->{Todo} = $arg_for{message};
return;
}
Obviously, the _on_or_after_date
functionality could be used in multiple places, such as with SKIP tests. See also Devel::Deprecate for the rationale.
prove --source File --file-option extensions=.txt ./tap.txt
fails, while prove --source File --file-option extensions=.txt --file-option extensions=.tmp ./tap.txt
does not.
-> ao% perl -v | grep version
This is perl 5, version 24, subversion 1 (v5.24.1) built for darwin-thread-multi-2level
-> ao% perl -MTAP::Harness -E 'say TAP::Harness->VERSION'
3.36_01
-> ao% cat tap.txt
1..1
ok 1
-> ao% prove --source File --file-option extensions=.txt ./tap.txt
Can't use string (".txt") as an ARRAY ref while "strict refs" in use at /usr/local/Cellar/perl/5.24.1/lib/perl5/5.24.1/TAP/Parser/SourceHandler/File.pm line 74.
-> ao% prove --source File --file-option extensions=.txt --file-option extensions=.tmp ./tap.txt
./tap.txt .. ok
All tests successful.
Files=1, Tests=1, 0 wallclock secs ( 0.01 usr + 0.00 sys = 0.01 CPU)
Result: PASS
prove --version TAP::Harness v3.38 and Perl v5.26.0
Steps to Reproduce:
Read this section in the prove docs about --rules:
https://metacpan.org/pod/prove#-rules
The most practical use is likely to specify that some tests are not "parallel-ready".
That's exactly what I'm trying to do here.
I have 3 tests, all which sleep five seconds and then issue a pass().
prove -j3 --rules='seq=a.t' --rules='par=b.t' --rules='par=c.t' a.t b.t c.t
a.t ... ok
b.t ... ok
c.t .. ok
All tests successful.
Files=3, Tests=3, 6 wallclock secs ( 0.07 usr 0.03 sys + 0.31 cusr 0.10 csys = 0.51 CPU)
Result: PASS
As we can see above, the tests took 6s to run, which suggests every one of them ran in parallel (It should have taken 11s (1s prove overhead, 5s seq run, 5s par run).
This made me suspect the matching for full strings without globs was broken. However, in the prove docs it states:
Any test which does not match a rule will be run in sequence at the end of the run.
This did not occur, ergo the seq rule for a.t should have matched. Furthermore, since they were all run in parallel either way, that explanation didn't hold water. I then suspected that the matching order may be covering up a broken match:
"First match wins". The first rule that matches a test will be the one that applies.
However, I found no indication in the TAP::Parser::Scheduler source suggesting that specifying the test's full name would not result in a match. Nor did I see evidence that the order of rule processing was mishandled. In fact, TAP::Parser::Scheduler::_set_rules appeared to correctly classify my input:
#Output from dumping $self->{scheduler} before the return() in App::Prove::Scheduler->new()
$VAR1 = [
[
[
[
bless( {
'description' => 'a.t',
'filename' => 'a.t'
}, 'TAP::Parser::Scheduler::Job' )
]
]
],
[
bless( {
'description' => 'b.t',
'filename' => 'b.t'
}, 'TAP::Parser::Scheduler::Job' )
],
[
bless( {
'filename' => 'c.t',
'description' => 'c.t'
}, 'TAP::Parser::Scheduler::Job' )
]
];
After my initial shock at the dubious usefulness of such a data structure, I realized that the hugely nested first entry signified that the first entry indeed had walked the seq => codepath in _rule_clause, effectively ruling out any possibility of a match failure.
After more analysis of both TAP::Parser::Scheduler and TAP::Harness, the reason for all this becomes quite clear. The helper subs of Tap::Harness::aggregate_tests, _aggregate_parallell and _aggregate_single are just calling $scheduler->get_job*() and iterating in the sensible manner for the respective runmode.
However, the get_job() and get_jobs() subs return no information whatsoever about whether the job should be run in parallel or in sequence, and the helpers do nothing but call those subs. As such, the --rules information is effectively ignored (aside from throttling down -j to 1 when nothing but seq rules are passed). Having no information to work on it simply works the same way it did before the --rules feature was added.
As far as I can tell the right course of action would be:
The hard part with the latter, of course, is that we'd basically have to dummy up new TAP::Parser::Schedulers to feed into the aggregate_ helpers with the jobs restricted to each schedule "batch". When I say batch, I mean something like the third rules example in Tap::Parser::Scheduler:
{
seq => [
{ seq => 't/startup/*.t' }, #batch 1
{ par => ['t/a/*.t','t/b/*.t','t/c/*.t'], } #batch 2
{ seq => 't/shutdown/*.t' }, #batch3 ...
],
},
The data structure formed inside Tap::Parser::Scheduler::_set_rules is nowhere near as nice as that (it's deeply nested arrays as shown above in dumper output). That should be straightforward to clean up, given hardly anything is actually consulting it in practice.
skip_todo event arrives with the following object:
$VAR1 = bless( {
'description' => '',
'type' => 'test',
'explanation' => '& SKIP Testing skip',
'test_num' => 3,
'raw' => 'not ok 3 # TODO & SKIP Testing skip',
'directive' => 'TODO',
'ok' => 'not ok'
}, 'TAP::Parser::Result::Test' );
I believe explanation should be TODO & SKIP
?
Api description or something.
Also, there is a link in pod to non-existant TAP::Parser::Formatter
in prove
pod
prove
has many features, but for my case I'm just interested in the part where it aggregates a TAP stream (originating from a non-Perl toolchain) and reports its little summary at the bottom with Result: FAIL/PASS.
The shortest working solution I could find is
printf "1..1\nok 1\n" | # example TAP emitter
prove -e 'cat -' /dev/null
which is lame. The report includes the dummy filename because prove insists on having a filename.
Streams should have first-class support in prove
, somehow.
There's this piece of code in Perl source handler:
if ( my $shebang = $file->{shebang} ) {
return 0.9 if $shebang =~ /^#!.*\bperl/;
# We favour Perl as the interpreter for any shebang to preserve
# previous semantics: we used to execute everything via Perl and
# relied on it to pass the shebang off to the appropriate
# interpreter.
return 0.3;
}
but actually, the $file->{shebang}
contains the content of the first line of test file anyway, even if it is not the shebang - like "use strict" or whatever the first line.
Which makes the source handler vote for 0.3 for any .t file that has any content in the first line, instead of 0.9 which should be considered much more reliable.
Guess it could be rewritten as:
return 0.3 if $shebang =~ /^#!/;
I'd like to have a plugin to do some test suite-wide setup for a project and considered an App::Prove::Plugin::MyPlugin
might be the way to go.
The -I
value isn't made 'live' for the _find_module
function. Is this intended?
.proverc
-It/lib
-l
-r
-PMyPlugin
t
behaviour
$ prove
Can't load module MyPlugin at /sw/perlbrew-0.76/perls/perl-5.26.0/bin/prove line 13
Current workaround.
$ PERL5OPT=-It/lib prove
<snip>
All tests successful.
Files=14, Tests=142, 1 wallclock secs ( 0.04 usr 0.02 sys + 0.57 cusr 0.18 csys = 0.81 CPU)
Result: PASS
prove version: TAP::Harness v3.38 and Perl v5.26.0
I wanted to check in my test script which test harness is in use. I would like to distinguish Test::Harness
(ExtUtils::MakeMaker
manged test) and TAP::Harness
(prove
managed test). How can I do this? I have checked the environment (HARNESS_* environment variables) without success.
I have installed Win32::Console (and Term::ANSIColor just in case), but do not see color output. Googling around I got a more detailed SO question. I know that color output otherwise works because ack
now has colored output, and it also uses Win32::Console to achieve this.
Using Windows 7, Strawberry 5.18.1.1, Test::Harness 3.29.
Enabling test suite parallelism for installations from CPAN is problematic.
There is only the option of running all tests of a distribution in parallel, without any way for the distribution itself to influence that, by setting HARNESS_OPTIONS
( Test::Harness ) or creating a ~/.proverc
file ( App::Prove ). This mechanism is currently used for two purposes:
This is unfortunate, because not all test suites are safe for running in parallel and there is currently no way for specifying within a distribution how its test suite can be run in parallel.
A distribution should be able to opt-in into running (parts of) its test suite in parallel, instead of being forced to be run in parallel.
Currently, Test::Harness (together with App::Prove and make test
) has only the most basic support for running the test suite of a distribution in parallel. Supplying additional information about which tests can be run in parallel is possible for App::prove ( but maybe see #75 ) on the command line and for Test::Harness by presence of a testrules.yml
file. App::prove could respect a testrules.yml
file with some small changes.
Ideally, specifying the maximum level of parallelism does not automatically enable full/blind test suite parallelism in Test::Harness and App::Prove. A solution to allow distributions and the local user to configure test suite parallelism could be the presence or absence of a testrules.yml
file:
a) If a testrules.yml
is present, this should be respected together with the -j
option from HARNESS_OPTIONS
b) If HARNESS_OPTIONS
contains (say) --parallel-default=~/.testrules.yml
, this is used in the case where no testrules.yml
exists in the distribution
c) -j8
does not enable test suite parallelism on its own anymore
Of course, c) means breaking behaviour, which may or may not be that great. It can be worked around by implementing something like b) .
When using TAP::Parser::Iterator::Stream with the output handle of the pipe()
function, in a parallel testing scenario, the parallellization could benefit from having the stream return the handle from get_select_handles()
.
Please add this enhancement.
shlomif[black-hole]:~/Test-Harness$ export HARNESS_VERBOSE=1
shlomif[black-hole]:~/Test-Harness$ prove t/*.t
t/000-load.t ........... 13/97 # Testing Test::Harness 3.42, Perl 5.028001, /usr/bin/perl5.28.1
t/000-load.t ........... ok
t/aggregator.t ......... ok
t/bailout.t ............ ok
t/base.t ............... ok
t/callbacks.t .......... ok
t/console.t ............ ok
t/env_opts.t ........... ok
t/errors.t ............. ok
t/file.t ............... 1/56 ^C
shlomif[black-hole]:~/Test-Harness$ runprove t/*.t 2>&1 | head -10
t/000-load ........... 1..97
ok 1 - use TAP::Parser;
ok 2 - ... and TAP::Parser should have the correct version
ok 3 - use App::Prove;
ok 4 - ... and App::Prove should have the correct version
ok 5 - use App::Prove::State;
ok 6 - ... and App::Prove::State should have the correct version
ok 7 - use App::Prove::State::Result;
ok 8 - ... and App::Prove::State::Result should have the correct version
ok 9 - use App::Prove::State::Result::Test;
shlomif[black-hole]:~/Test-Harness$
I just wasted many minutes trying to understand why it doesn;t work on travis.
It is hard to watch for library changes without it. GH notifies about releases, not tags.
s/test/the/
test simply checks whether STDIN
is a tty.
Such test doesn't check anything and should be removed.
Hi All,
I've been exploring the sitecustomize.pl
feature. When I have my sitecustomize.pl
installed then t/compat/test-harness-compat.t
fails the "switches" test. A bit of bisecting shows me that a sitecustomize.pl
file containing the single line of use strict;
seems to be all that is necessary to cause the test to fail.
Of course, my sitecustomize.pl starts with
use strict;
use warnings;
and then goes on to use
other modules which presumably also use strict;
I believe that the test is simply checking if strict.pm has been loaded and that it expects it to not have been loaded. If so then the test will fail if anything (e.g. a sitecustomize.pl
) has caused strict.pm to be loaded.
Does that seem like a correct analysis?
If so, is there anything to do to make the test more robust?
Test::Harness version 3.36_01, Test::More version 1.302177
prove'ing a test which forks a daemon process doesn't exit until the daemon is killed. This is new since updating Test:: libs to all latest versions. Previously the test would exit.
This is a bit of a problem as I use a function in our test suite to restart some daemons if code has changed. This is no longer usable since the first test never exits.
Example: with prove
this hangs until the sleep ends but exits immediately when run with perl
use Test::More; system('sleep 5000 &');
Also posted to Test::More: Test-More/test-more#868 (comment)
We're currently bunding a fairly old version of Test::More (0.72, from 2007, the same that shipped with v5.10.0). I don't think this is particularly helpful, specially considering we're not upgrading it and pretty much everyone has a more recent T::M installed. Test::More has been in core since 5.8.0 (and was shipped with 5.6.2 too), I don't think we'd have serious bootstrapping issues on any perl we support.
With perl 5.8.9 to perl 5.14.4 I'm seeing test failures in t/source_handler.t. This probably affects some other Ye Olde Perles with old versions of some toolchain modules:
Can't exec "t/source_tests/source.sh": Permission denied at /home/david/cpantesting/perl-5.8.9/lib/5.8.9/IPC/Open3.pm line 268.
open3: exec of t/source_tests/source.sh failed at /home/david/cpantesting/perl-5.8.9/.cpan/build/Test-Harness-3.42-DyMKY5/blib/lib/TAP/Parser/Iterator
/Process.pm line 165
# Failed test '... line 1'
# at t/source_handler.t line 386.
# got: undef
# expected: '1..1'
# Failed test '... line 2'
# at t/source_handler.t line 386.
# got: undef
# expected: 'ok 1 - source.sh'
Can't exec "t/source_tests/source_args.sh": Permission denied at /home/david/cpantesting/perl-5.8.9/lib/5.8.9/IPC/Open3.pm line 268.
open3: exec of t/source_tests/source_args.sh foo failed at /home/david/cpantesting/perl-5.8.9/.cpan/build/Test-Harness-3.42-DyMKY5/blib/lib/TAP/Parser
/Iterator/Process.pm line 165
# Failed test '... line 1'
# at t/source_handler.t line 386.
# got: undef
# expected: '1..1'
# Failed test '... line 2'
# at t/source_handler.t line 386.
# got: undef
# expected: 'ok 1 - source_args.sh foo'
# Looks like you failed 4 tests of 79.
t/source_handler.t ..............
Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/79 subtests
(less 4 skipped subtests: 71 okay)
It seems that even though t/source_tests/source_args.sh has the execute bit set in the tarball, something in the old toolchain is removing it:
perl-5.8.9 $ ls -l .cpan/build/Test-Harness-3.42-ud1QgD/t/source_tests/
total 60
-rw-r--r-- 1 david david 75 Jul 7 2013 harness
-rw-r--r-- 1 david david 112 Jul 7 2013 harness_badtap
-rw-r--r-- 1 david david 117 Jul 7 2013 harness_complain
-rw-r--r-- 1 david david 179 Jul 7 2013 harness_directives
-rw-r--r-- 1 david david 235 Jul 7 2013 harness_failure
-rw-r--r-- 1 david david 443 Sep 11 2016 psql.bat
-rw-r--r-- 1 david david 149 Jul 7 2013 source
-rw-r--r-- 1 david david 21 Jul 7 2013 source.1
-rw-r--r-- 1 david david 54 Sep 11 2016 source_args.sh
-rw-r--r-- 1 david david 103 Jul 7 2013 source.bat
-rw-r--r-- 1 david david 70 Jul 7 2013 source.pl
-rw-r--r-- 1 david david 46 Sep 11 2016 source.sh
-rw-r--r-- 1 david david 69 Jul 23 2017 source.t
-rw-r--r-- 1 david david 23 Jul 7 2013 source.tap
-rw-r--r-- 1 david david 131 Jul 23 2017 test.tap
You may want to twiddle the test to explicitly turn the execute bit on in this case.
I seem prove --harness TAP::Harness t/00_compile.t
should work. But the command says following with TAP::Harness 3.31:
Unknown arguments to TAP::Harness::new (harness_class) at /home/tokuhirom/.plenv/versions/5.20.0/lib/perl5/site_perl/5.20.0/TAP/Harness/Env.pm line 118.
It works with TAP::Harness 3.30.
I think following change may break this option.
Add harness_class argument to TAP::Harness::Env (Leon Timmermans)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.