This comment:

seems to no longer be relevant.

Tests may be run via prove directly or via make with make test.
In latter case, it is not possible to pass additional arguments to the script.

But this may be necessary for the IDE integration. IDE may need to pass it's own formatter, plugin or modify execution mode, like jobs, shufflle, etc.

Makefile.PL should not depend on development environment, because different users may have different preferences, so would be nice to be able to pass additional arguments via environment variables.

One thing that bothers me is that these variable are expected to have higher precedence than command line argument, so IDE could tune execution, but this feels wrong... May be two variables could be introduced, e.g PROVE_OPTS and FORCE_PROVE_OPTS where first has lesser precedence and latter - higher.

I want to develop a prove plugin that needs more flexibility with respect to the arguments passed to the plugin. Especially the comma should at some places be used as is. Currently there exists now way to escape it.

my @args = ();
if ( $name =~ /^(.*?)=(.*)/ ) {
  $name = $1;
  @args = split( /,/, $2 );
}

Do you see any way to implement this? A number of different approaches are described here split on delimiter unless escaped.

CC @rocky

Over at Test-More/test-more#319 we've been talking about ways to get Test::More to do the whole "stop on fail" thing. What occurred to me is that while stopping immediately after a failed result has issues, stopping the test suite at the end of a failing test is easy and it solves 90% of the problem.

The problems being...

• I'm just installing the module, all I care about is if its going to work.
• I'm debugging the module, please don't throw a tremendous amount of garbage on my screen.

Test::Harness can have a prove flag/environment variable which stops running the rest of the .t files if one of them fails. Like an implied BAIL_OUT. Then a user could do something like...

HARNESS_BAIL_ON_FAIL=1 cpan Catalyst

Unless a failing test file is very large or very slow, this should greatly reduce the amount of failure garbage and the time spent completing failing test suites. It also doesn't matter how the tests are written.

What say?

Hi, I have some tests that produce the following output (along with other stuff), I'm planning to extend our ci in openQA to have a bit of better support for console based tests, however, while playing with Test::Harness, I'm struggling to figure out how subtests are being processed... I could find that there are some tests that are trying to parse at least subtests but nothing else.

I could not find any docs regarding Subtests in the TAP docs or related sites (perhaps I didn't look well enough?)

Any hints?

So long story short, I'm loading the file with code like this:

use TAP::Parser;
use feature 'say';
use Carp;

my $parser = TAP::Parser->new({tap => "t/source_tests/tap_format_example_complex.tap"});
confess "Failed " . $parser->parse_errors if $parser->parse_errors;

while ( my $result = $parser->next ) {
    say $result->type . ' - '.$result->as_string;
}

And upon running my simple test, I get:

test - not ok 24 - group filter
test - ok 25 - POST /api/v1/isos
test - ok 26 - 200 OK
test - not ok 27 - 10 new jobs created
test - ok 28 - GET /api/v1/jobs
test - not ok 29 - server_32 is only parent of client1_32
test - not ok 30 - server_32 is only parent of client2_32
test - not ok 31 - server_32 has no parents
test - ok 32 - kde is not created for 32bit machine
test - ok 33 - advanced_kde is not created for 32bit machine
test - not ok 34 - server_64 is only parent of client1_64
test - not ok 35 - server_64 is only parent of client2_64
test - not ok 36 - server_64 has no parents
test - ok 37 - no (unexpected) warnings (via END block)
unknown - Dubious, test returned 255 (wstat 65280, 0xff00)
unknown - Failed 9/37 subtests 
unknown - 
unknown - Test Summary Report
unknown - -------------------
unknown - t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
unknown -   Failed tests:  23-24, 27, 29-31, 34-36
unknown -   Non-zero exit status: 255
unknown -   Parse errors: No plan found in TAP output
unknown - Files=1, Tests=37,  9 wallclock secs ( 0.02 usr  0.01 sys +  5.05 cusr  0.18 csys =  5.26 CPU)
unknown - Result: FAIL

Also the following results not being shown is kinda strange... :

• unknown - Failed 9/37 subtests
• unknown - Result: FAIL

File:

t/api/02-iso.t .. 
ok 1 - GET /api/v1/jobs/99927
...
ok 21 - 200 OK
ok 22 - we have no gru download tasks to start with
# Subtest: Multi Machine job dependencies
    ok 1 - POST /api/v1/isos
    not ok 2 - 200 OK
    not ok 3 - only one job created due to group filter
    1..3
not ok 23 - Multi Machine job dependencies
# Subtest: group filter
    ok 1 - POST /api/v1/isos
    ok 2 - 200 OK
    ok 3 - no jobs created if group invalid
    ok 4 - POST /api/v1/isos
    ok 5 - 200 OK
    not ok 6 - only one job created due to group filter
    ok 7 - POST /api/v1/isos
    ok 8 - 200 OK
    not ok 9 - only one job created due to group filter (by ID)
    1..9
not ok 24 - group filter
...
ok 32 - kde is not created for 32bit machine
ok 33 - advanced_kde is not created for 32bit machine
not ok 34 - server_64 is only parent of client1_64
not ok 35 - server_64 is only parent of client2_64
not ok 36 - server_64 has no parents
ok 37 - no (unexpected) warnings (via END block)
Dubious, test returned 255 (wstat 65280, 0xff00)
Failed 9/37 subtests 

Test Summary Report
-------------------
t/api/02-iso.t (Wstat: 65280 Tests: 37 Failed: 9)
  Failed tests:  23-24, 27, 29-31, 34-36
  Non-zero exit status: 255
  Parse errors: No plan found in TAP output
Files=1, Tests=37,  9 wallclock secs ( 0.02 usr  0.01 sys +  5.05 cusr  0.18 csys =  5.26 CPU)
Result: FAIL

Test-Harness used to come with a Makefile.PL and a NotBuild.PL. The Makefile.PL forces you to use NotBuild.PL for authoring purposes (in particular, for the dist and distmeta actions). I don't see the usefulness of this setup.

Every time aggregate_tests is called, it calls prepare, which sets $longest to 0, finds the longest test description, and then sets the _longest attribute.

If aggregate_tests is called multiple times with the same formatter, then _longest can end up being less than the actual longest description, causing a negative repeat count warning.

The fix to this would be to start with the current value of _prepare instead of starting at 0 every time.

Thanks for creating and maintaining some great modules!

After some recent upgrades, the --archive option to prove seems to be broken:

$ prove --archive /tmp/test.tgz
Can't locate object method "new" via package "TAP::Harness::Archive::SUPER" at /usr/local/share/perl/5.14.2/TAP/Harness/Archive.pm line 96.

Line 96 looks like this:

my $self = $class->SUPER::new($args);

Versions:
OS: Debian 7.6
Perl: v5.14.2 built for x86_64-linux-gnu-thread-multi
TAP::Harness 3.33
TAP::Harness::Archive 0.15

I have tested this with TAP::Harness::Archive 0.15 and TAP::Harness 3.23, and that seems to work. Not quite sure if this is mainly a problem with TAP::Harness or TAP::Harness::Archive...

Please let me know if more details are needed.

This is currently not listed as a valid argument to TAP::Harness->new(). I think it should be documented there - both for completeness and because it's incredibly useful. Or at least link to the Test::Harness::Beyond page.

I guess App::Prove::Plugin::* needs more hook points like 'after_test', 'after_runtests'.

For example, App::Prove::Plugin::Growl hacks to get a testing result.
And, I want to write a plugin to post testing result to report server.

This ticket continues the discussion begun in the Perl issue tracker at Perl/perl5#21455. I'm moving the discussion here because:

• My analysis indicates that the problem lies in Test-Harness's own test suite and is merely reproduced when Test-Harness is synched into core; and

• Test-Harness's documentation on CPAN indicates that bug tickets should be filed on RT, but that queue consists of 62 old reports and I suspect filing here in the Perl-Toolchain-Gang organization will get more attention. (If you want me to file on RT instead, just let me know.)

The discussion so far

Briefly, the discussion in Perl/perl5#21455: For the last month, when I configure, build and test Perl on FreeBSD (and other OSes) and then call git status at the end of the test suite, the status output includes:

Untracked files:
  (use "git add <file>..." to include in what will be committed)
	cpan/Test-Harness/perl.core

This core dump does not cause any test failures and appears to be compiler-independent, but is nonetheless real:

-rw-------  1 jkeenan  jkeenan  11415552 Sep  5 21:12 cpan-Test-Harness.freebsd.clang14.perl.core

I do not observe this core dump on Linux: neither by ls nor by git status.

I haven't definitively bisected this problem to see when it entered the Perl 5 core distribution, but I strongly suspect that it occurred when I synched Test-Harness-3.46 into core on August 13.

commit 6e0b8bd69336ecc95a6ca480c4b2ab58c0080bc2
Author:     Leon Timmermans <[email protected]>
AuthorDate: Sat Aug 12 17:05:48 2023 -0400
Commit:     James E Keenan <[email protected]>
CommitDate: Sun Aug 13 06:56:43 2023 -0400

    cpan/Test-Harness - Update to version 3.46

I performed that synching on Linux. All tests passed. git status was clean, so I had no reason to repeat that process on any other OS. If I weren't regularly building and testing on FreeBSD-13, I wouldn't have stumbled across this problem. My expectation is that on any (non-Windows, at least) OS to which I have access, after I run perl's make test_harness, all tests should PASS and git status should show no untracked files.

Research in the Test-Harness repository on GitHub

I brought my own fork of Test-Harness on GitHub up-to-date with PTC's repository, then created a branch so that I could isolate the code generating the segfault in a way by which I could compare Linux and FreeBSD output more precisely. That 'segfault-analysis-20230908' branch is here.

I acknowledge that I have rarely peered into Test-Harness's code; hence, there may be weaknesses in the diagnostic code I added in my branch. All my changes are confined to t/harness.t:

$ git diff -w master..segfault-analysis-20230908 -- t/ |cat
diff --git a/t/harness.t b/t/harness.t
index b84fa27..e81f23f 100644
--- a/t/harness.t
+++ b/t/harness.t
@@ -6,6 +6,7 @@ BEGIN {
 
 use strict;
 use warnings;
+use Data::Dumper;
 
 use Test::More;
 use IO::c55Capture;
@@ -539,12 +540,39 @@ for my $test_args ( get_arg_sets() ) {
         skip "ASAN doesn't passthrough SEGV", 1
           if "$Config{cc} $Config{ccflags} $Config{optimize}" =~ /-fsanitize\b/;
 
+print STDERR "XXX:\n";
+warn "XXX1: perl.core file exists" if (-e './perl.core');
+
         @output = ();
-        _runtests( $harness_failures, "$sample_tests/segfault" );
+
+        # The following line creates './perl.core' on FreeBSD:
+        #
+        # _runtests( $harness_failures, "$sample_tests/segfault" );
+        #
+        # Our objective is to isolate the code generating the segfault so that
+        # we can understand why it (seemingly) creates a core dump on FreeBSD
+        # et al. but not on Linux.
+
+print STDERR "FFF1: \$harness_failures is a ", ref $harness_failures, "\n";
+print STDERR "FFF2: ", scalar keys (%$harness_failures), " elements in \$harness_failures\n";
+
+        # First, run only the TAP::Harness objects seen so far:
+        _runtests( $harness_failures );
+warn "XXX2: perl.core file exists" if (-e './perl.core');
+
+        # Next, run the segfault test:
+        # First argument to _runtests must be a TAP::Harness object (I think)
+
+        my $this_harness = $HARNESS->new;
+        _runtests( $this_harness, "$sample_tests/segfault" );
+warn "XXX3: perl.core file exists" if (-e './perl.core');
+
+print STDERR Dumper(\@output);
 
         my $out_str = join q<>, @output;
 
         like( $out_str, qr<SEGV>, 'SIGSEGV is parsed out' );
+print STDERR "YYY:\n";
     }
 
     #XXXX

All the changes are debugging code except splitting _runtests( $harness_failures, "$sample_tests/segfault" ); into two separate invocations of _runtests(), for the second of which I had to create a new TAP::Harness object.

Results

If I build Test::Harness on each of Linux and FreeBSD and exercise the file generating the segfault on FreeBSD, I get these comparative results:

Ubuntu Linux 22.04 LTS at release-3.45_01-12-gffc5f49

$ perl Makefile.PL && make
...
Manifying 17 pod documents

$ rm perl.core; prove -b t/harness.t; ls -ltr . | tail
rm: cannot remove 'perl.core': No such file or directory
t/harness.t .. 1/133 XXX:
FFF1: $harness_failures is a TAP::Harness
FFF2: 12 elements in $harness_failures
$VAR1 = [
          'Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)',
          'Result: NOTESTS',
          't/sample-tests/segfault ..',
          'No subtests run',
          'Test Summary Report',
          '-------------------',
          't/sample-tests/segfault (Wstat: 139 (Signal: SEGV, dumped core) Tests: 0 Failed: 0)',
          'Non-zero wait status: 139',
          'Parse errors: No plan found in TAP output',
          'Files=1, Tests=0,  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)',
          'Result: FAIL'
        ];
YYY:
t/harness.t .. ok       
All tests successful.
Files=1, Tests=133,  0 wallclock secs ( 0.01 usr  0.00 sys +  0.07 cusr  0.03 csys =  0.11 CPU)
Result: PASS
-rw-rw-r-- 1 jkeenan jkeenan  2244 Aug 12 16:45 Makefile.PL
-rw-rw-r-- 1 jkeenan jkeenan  6346 Aug 12 16:45 HACKING.pod
drwxrwxr-x 2 jkeenan jkeenan  4096 Aug 12 16:45 bin
-rw-rw-r-- 1 jkeenan jkeenan 44232 Sep  8 17:17 Changes
-rw-rw-r-- 1 jkeenan jkeenan   774 Sep  9 13:10 MYMETA.yml
-rw-rw-r-- 1 jkeenan jkeenan  1295 Sep  9 13:10 MYMETA.json
-rw-r--r-- 1 jkeenan jkeenan 45309 Sep  9 13:10 Makefile
drwxrwxr-x 8 jkeenan jkeenan  4096 Sep  9 13:10 blib
-rw-rw-r-- 1 jkeenan jkeenan     0 Sep  9 13:10 pm_to_blib
drwxrwxr-x 9 jkeenan jkeenan  4096 Sep  9 13:10 t

FreeBSD-13 at release-3.45_01-12-gffc5f49

$ perl Makefile.PL && make
...
Manifying 50 pod documents

$ rm perl.core; prove -b t/harness.t; ls -ltr . | tail
rm: perl.core: No such file or directory
t/harness.t .. 1/133 XXX:
FFF1: $harness_failures is a TAP::Harness
FFF2: 12 elements in $harness_failures
XXX3: perl.core file exists at t/harness.t line 568.
$VAR1 = [
          'Files=0, Tests=0,  0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)',
          'Result: NOTESTS',
          't/sample-tests/segfault ..',
          'No subtests run',
          'Test Summary Report',
          '-------------------',
          't/sample-tests/segfault (Wstat: 139 (Signal: SEGV, dumped core) Tests: 0 Failed: 0)',
          'Non-zero wait status: 139',
          'Parse errors: No plan found in TAP output',
          'Files=1, Tests=0,  0 wallclock secs ( 0.00 usr  0.01 sys +  0.00 cusr  0.01 csys =  0.02 CPU)',
          'Result: FAIL'
        ];
YYY:
t/harness.t .. ok       
All tests successful.
Files=1, Tests=133,  0 wallclock secs ( 0.04 usr  0.01 sys +  0.20 cusr  0.05 csys =  0.30 CPU)
Result: PASS
drwxr-xr-x  3 jkeenan  jkeenan         3 Sep  8 21:21 reference
drwxr-xr-x  2 jkeenan  jkeenan        26 Sep  8 21:21 smoke
drwxr-xr-x  4 jkeenan  jkeenan         4 Sep  8 21:21 xt
-rw-r--r--  1 jkeenan  jkeenan     44090 Sep  9 17:05 Makefile
-rw-r--r--  1 jkeenan  jkeenan      1295 Sep  9 17:05 MYMETA.json
-rw-r--r--  1 jkeenan  jkeenan       774 Sep  9 17:05 MYMETA.yml
drwxr-xr-x  8 jkeenan  jkeenan         8 Sep  9 17:05 blib
-rw-r--r--  1 jkeenan  jkeenan         0 Sep  9 17:05 pm_to_blib
-rw-------  1 jkeenan  jkeenan  11337728 Sep  9 17:05 perl.core
drwxr-xr-x  9 jkeenan  jkeenan        56 Sep  9 17:05 t

Note that only on FreeBSD do I get this statement in the output from my branch:

XXX3: perl.core file exists at t/harness.t line 568.

... and that a core dump is created only on FreeBSD, as shown from this line in ls:

-rw-------  1 jkeenan  jkeenan  11337728 Sep  9 17:05 perl.core

Inferences and Next Questions

1. The test code which is generating the segfault for testing purposes appears to be this simple program, t/sample-tests/segfault:

$ cat t/sample-tests/segfault 
#!/usr/bin/perl

print "1..1\n";
print "ok 1\n";
kill 'SEGV', $$;

Why does this create a segfault file (./perl.core) on FreeBSD but not on Linux?

2. On FreeBSD, Test-Harness's own make clean (which appears to be a straightforward derivative from ExtUtils::MakeMaker) and git clean -dfx will both remove the core dump file, after which git status will be clean. So what is annoying is that, on one OS but not the other, calling git status before running either of those cleanup commands will report an "untracked file." That annoyance would affect anyone doing maintenance work on Test-Harness, but nobody else. But this will be really annoying to anyone running Perl's own test suite on FreeBSD (or any other OS so affected).

3. Should we consider modifying t/harness.t so that if it generates a core dump, the test file tidies up after itself and removes perl.core once it's no longer needed?

Thank you very much.
Jim Keenan

until system ('set PATH=c:\Dwimperl\perl\bin;%PATH%');
it's strange, because, when I run in cmd it works

use FindBin '$RealBin';
system ('set PATH=c:\Dwimperl\perl\bin;%PATH%');
my $cmd= qq{prove --rc=.proverc  :: --project ${dir_4_test} --path_2_test_dir "$RealBin/${dir_4_test}/jobs/Jobs" > ${dir_4_test}.html};
system ($cmd);

What --state does when combined with file arguments is underdocumented
and sometimes surprising. The following in the Test-Harness
distribution directory:

[0] because there are tests on disk and they would be run with
state=save alone; this is one of the cases where it would be helpful
to just run all tests from disk.

[1] what happens is that the argument t/parse.t is merged with all
files in .prove. Apparently there is no way to run a subset of the
known tests with state options without editing the .prove file.

[2] Given that surprise [0] above refused to work, it's now
surprising that this incantation does the right thing

[3] keyword "all" apparently does not mean all tests on disk. Not
mentioned in the docs.

I believe the currently undocumented rules are:

(1) as long as no statefile exists, any combination of state arguments
turn prove into a noop unless there are explicit file or directory
arguments on the commandline. E.g.

prove --state=save,all         # noop
prove --state=save,all t/1.t   # runs 1.t

(2) if a statefile exists, the set of tests to be considered to be run
is a union of the test files in the state file and files or
directories on the command line. From this union set all tests on
the command line are run unconditionally while files from the
statefile are run according to the rule. E.g.

rm -f .prove
prove --state=save        t/1.t t/2.t # runs 1.t and 2.t
prove --state=save,failed t/1.t       # runs 1.t and whatever failed above

(3) the keyword "all" refers to the files in the statefile, not to the
files on disk

I'm unsure whether one would prefer the rules to change or get these
rules documented. I find the rules hard to understand and to document
and also disputable.

This might be something others either ignore or maybe it doesn't happen often enough for anyone to say something. Or it's possible it's been mentioned but wontfix'ed because it breaks the flow of the routines.

I had the occasion to see it a few times when running prove manually from the wrong directory (prove -lv t/testfile.t when working directory was the t/ dir)

I'm not sure where a good place for the check could be, but it would be nice if this was smart enough to say if ( !fileexist && sourcerefundefined ) confess("file/method not found.");

Right now this is the error you get. It's factual but not very helpful.

rdrake@test:~/Test-Harness [master]$ perl -Ilib bin/prove asdf
Cannot detect source of 'asdf'! at lib/TAP/Parser/IteratorFactory.pm line 261.
        TAP::Parser::IteratorFactory::detect_source(TAP::Parser::IteratorFactory=HASH(0x18d12c0), TAP::Parser::Source=HASH(0x18d11b8)) called at lib/TAP/Parser/IteratorFactory.pm line 211
        TAP::Parser::IteratorFactory::make_iterator(TAP::Parser::IteratorFactory=HASH(0x18d12c0), TAP::Parser::Source=HASH(0x18d11b8)) called at lib/TAP/Parser.pm line 472
        TAP::Parser::_initialize(TAP::Parser=HASH(0x18a7028), HASH(0x16bb7c0)) called at lib/TAP/Object.pm line 55
        TAP::Object::new("TAP::Parser", HASH(0x16bb7c0)) called at lib/TAP/Object.pm line 130
        TAP::Object::_construct(TAP::Harness=HASH(0x15e8e68), "TAP::Parser", HASH(0x16bb7c0)) called at lib/TAP/Harness.pm line 852
        TAP::Harness::make_parser(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Scheduler::Job=HASH(0x188d160)) called at lib/TAP/Harness.pm line 651
        TAP::Harness::_aggregate_single(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Aggregator=HASH(0x1687048), TAP::Parser::Scheduler=HASH(0x188d238)) called at lib/TAP/Harness.pm line 743
        TAP::Harness::aggregate_tests(TAP::Harness=HASH(0x15e8e68), TAP::Parser::Aggregator=HASH(0x1687048), "asdf") called at lib/TAP/Harness.pm line 558
        TAP::Harness::__ANON__() called at lib/TAP/Harness.pm line 571
        TAP::Harness::runtests(TAP::Harness=HASH(0x15e8e68), "asdf") called at lib/App/Prove.pm line 546
        App::Prove::_runtests(App::Prove=HASH(0x1184bd0), HASH(0x15b4640), "asdf") called at lib/App/Prove.pm line 504
        App::Prove::run(App::Prove=HASH(0x1184bd0)) called at bin/prove line 9

I use the -v switch in the prove utility. DOC

but it has no effect. Then I dump my %ENV and found:

    HARNESS_ACTIVE                     1,
    HARNESS_IS_VERBOSE                 1,
    HARNESS_VERSION                    "3.36_01",

HARNESS_IS_VERBOSE is used instead of HARNESS_VERBOSE

The code comment refer to: https://rt.cpan.org/Public/Bug/Display.html?id=12458

Is this bad doc or bad fix?

--exec is documented in TAP::Harness as being something you might use to request a specific perl ... but when I do that, -I flags passed to the command line don't get handed through. i.e.

$ prove --exec $PERL_BINARY -I$LIB_PATH t

doesn't pass the -I through to the perl binary. Admittedly,

$ prove --exec $PERL_BINARY\ -I$LIB_PATH t

works fine since the --exec option is allowed to contain switches, but I can't find any documentation saying it's intentional to throw them away so I'm wondering if this is a bug.

Had a bug in my code because I assumed ->all_passed was the same as "everything is good". But this method returns false if all tests were skipped, like with plan skip_all => 'some reason'.

The end of the output here shows the ->all_passed returned false when the test file had skipped all tests. I would've expected it to return true.

zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -v | grep version
This is perl 5, version 24, subversion 0 (v5.24.0) built for x86_64-linux-multi
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -MTAP::Harness -wlE 'say TAP::Harness->VERSION'
3.36
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ cat foo.t 
use Test::More;
plan skip_all => 'some reason';
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ perl -MTAP::Harness -wlE 'my $res = TAP::Harness->new->runtests("foo.t"); say "--------\nAll tests passed: " . $res->all_passed'
foo.t .. 
              
foo.t .. 
skipped: some reason

Files=1, Tests=0,  1 wallclock secs ( 0.05 usr  0.01 sys +  0.03 cusr  0.01 csys =  0.10 CPU)

Result: NOTESTS

--------
All tests passed: 0
zoffix@leliana:/tmp/tmp.jx6I5dc73b$ 

I was briefly surprised to discover that my regular unit tests run in parallel (with HARNESS_OPTIONS set to -j9 in the environment), yet my extra tests (run via dzil xtest, which uses App::Prove) chugged along serially.

I'll patch, if someone can point me to where/how this can be done.

I like to keep a tab open that runs my unit tests repeatedly. When I've made changes I can switch to that tab and see the current results of all my tests. Coloring green or red makes it really easy to know if I've broken something. I tried running prove inside of a continuous watch like so:

watch --color 'prove --color t/*.t'

It appears that prove attempts to be smart and not output ANSI if the target isn't a TTY. This can also be shown with:

prove --color t/*.t > /tmp/output.txt

This is related to #85 and #101 but it's a more general issue.

Regression with using ::Env

First, there's a regression in a8fb517 (Make prove respect environmental variables #28, 2013-10-27), when we started paying attention to HARNESS_OPTIONS in prove having the options set to jN where N > 1 suppresses the --verbose output completely. E.g. (running this on the git.git test suite):

HARNESS_OPTIONS=j2 perl -I ~/g/Test-Harness/lib ~/g/Test-Harness/bin/prove --verbose t0002-gitfile.sh :: -
-verbose

ok 1 - initial setup
ok 2 - bad setup: invalid .git file format
ok 3 - bad setup: invalid .git file path
ok 4 - final setup + check rev-parse --git-dir
[...]

But before a8fb517 it would show the verbose output, curiously if you pipe it to | cat it shows up, so there's some isatty() check somewhere.

Instead it should at the very least be smart enough to figure out that your -jN setting doesn't matter in the case of a single test.

The --verbose output doesn't make much sense under -jN

When you run N tests at a time it will struggle with some ASCII-art to represent the concurrency, and then dump the non-verbose output (under --verbose !) for whatever test it happened to run last:

$ HARNESS_OPTIONS= perl -I ~/g/Test-Harness/lib ~/g/Test-Harness/bin/prove --shuffle -j8 --verbose t000*
.sh :: --verbose
===(       1;0  0/?  1/?  0/?  0/?  0/?  0/?  0/?  0/? )================ok
===(       5;0  0/?  1/?  0/?  0/?  0/?  0/?  0/?  0/? )================ok
===(      11;0  0/?  1/?  0/?  0/?  0/?  0/?  0/?  0/? )================ok
===(      29;0  0/?  0/?  0/?  10/?  3/?  0/?  1/? )====================ok
===(      34;0  0/?  0/?  0/?  11/?  3/?  0/? )=========================ok
===(      83;0  19/?  11/?  0/?  18/?  1/? )============================ok
===(     180;0  55/?  29/?  0/?  36/? )=================================ok
===(     262;1  61/?  9/?  49/? )=======================================ok
===(     497;4  266/?  28/? )===========================================ok
t0008-ignores.sh ..... 
ok 343 - beyond a symlink from subdirectory with --verbose
ok 344 - beyond a symlink from subdirectory with --verbose -n
ok 345 - beyond a symlink from subdirectory with --verbose --non-matching
ok 346 - submodule
ok 347 - submodule with -q
[...]

There's a lot more than the "ok" output under --verbose in that test, so the verbose output is being stripped. This is another case where piping it to | cat gives expected results.

Patch?

Not submitting this as a PR since I'm not familiar with the code, but this seems to fix the above issues:

diff --git a/lib/TAP/Harness.pm b/lib/TAP/Harness.pm
index a9a865d..105f846 100644
--- a/lib/TAP/Harness.pm
+++ b/lib/TAP/Harness.pm
@@ -453,6 +453,8 @@ Any keys for which the value is C<undef> will be ignored.
 
         local $default_class{formatter_class} = 'TAP::Formatter::File'
           unless -t ( $arg_for{stdout} || \*STDOUT ) && !$ENV{HARNESS_NOTTY};
+        local $default_class{formatter_class} = 'TAP::Formatter::File'
+          if $self->jobs > 1 && $arg_for{verbosity};
 
         while ( my ( $attr, $class ) = each %default_class ) {
             $self->$attr( $self->$attr() || $class );

I had opened houseabsolute/test-class-moose#108 on Test::Class::Moose, but @exodist suggested that I also open a ticket here, since the problem described above may actually be an issue with prove.

The original tickets demonstrates this issue and also contains a repository which can be used to replicate it.

It hangs and hangs and hangs.

Despite using select the whole architecture of TAP::Harness is based on blocking IO, which is a really poor combination with multiprocessing in particular. I'm not sure how to solve this. TAP::Parser is fundamentally a pull-architecture, while it should have been a push-based architecture. I'm not sure this can be fixed without rewriting TAP::Parser, TAP::Parser::Iterator*, TAP::Parser::Grammar and TAP::Parser::Multiplexer. The entire flow of control/data needs to be reversed :-(.

I have a small problem that crops up from time to time. Sometimes I write a TODO test to temporarily stop a test failure until such time that I can get business signoff on a change. However, I have to remember to remove the TODO test at a later date and having to manually handle this is fraught with error. I would like to be able to do this:

my $builder = Test::Builder->new;
$builder->todo_start( message => $message, until => 'YYYY-MM-DD' );
ok !defined $object->begin_thermonuclear_war, 
    'We should never allow thermonuclear war';
$builder->todo_end;

That subtly alters todo_start behavior, but I think it can be made backwards-compatible (at the risk of additional complexity) with this untested code:

sub todo_start {
    my $self = shift;
    if (1 == @_) {
        @_ = ( message => $_[0] );
    }
    my %arg_for = @_;
    return if $self->_on_or_after_date($arg_for{until});

    $self->{Start_Todo}++;
    if( $self->in_todo ) {
        push @{ $self->{Todo_Stack} } => $self->todo;
    }
    $self->{Todo} = $arg_for{message};

    return;
}

Obviously, the _on_or_after_date functionality could be used in multiple places, such as with SKIP tests. See also Devel::Deprecate for the rationale.

prove --source File --file-option extensions=.txt ./tap.txt fails, while prove --source File --file-option extensions=.txt --file-option extensions=.tmp ./tap.txt does not.

-> ao% perl -v | grep version
This is perl 5, version 24, subversion 1 (v5.24.1) built for darwin-thread-multi-2level
-> ao% perl -MTAP::Harness -E 'say TAP::Harness->VERSION'
3.36_01
-> ao% cat tap.txt
1..1
ok 1
-> ao% prove --source File --file-option extensions=.txt ./tap.txt
Can't use string (".txt") as an ARRAY ref while "strict refs" in use at /usr/local/Cellar/perl/5.24.1/lib/perl5/5.24.1/TAP/Parser/SourceHandler/File.pm line 74.
-> ao% prove --source File --file-option extensions=.txt --file-option extensions=.tmp ./tap.txt
./tap.txt .. ok
All tests successful.
Files=1, Tests=1,  0 wallclock secs ( 0.01 usr +  0.00 sys =  0.01 CPU)
Result: PASS

prove --version TAP::Harness v3.38 and Perl v5.26.0

Steps to Reproduce:

Read this section in the prove docs about --rules:
https://metacpan.org/pod/prove#-rules
The most practical use is likely to specify that some tests are not "parallel-ready".
That's exactly what I'm trying to do here.

I have 3 tests, all which sleep five seconds and then issue a pass().

prove -j3 --rules='seq=a.t' --rules='par=b.t' --rules='par=c.t' a.t b.t c.t
a.t ... ok
b.t ... ok
c.t .. ok
All tests successful.
Files=3, Tests=3, 6 wallclock secs ( 0.07 usr 0.03 sys + 0.31 cusr 0.10 csys = 0.51 CPU)
Result: PASS

As we can see above, the tests took 6s to run, which suggests every one of them ran in parallel (It should have taken 11s (1s prove overhead, 5s seq run, 5s par run).

This made me suspect the matching for full strings without globs was broken. However, in the prove docs it states:

Any test which does not match a rule will be run in sequence at the end of the run.

This did not occur, ergo the seq rule for a.t should have matched. Furthermore, since they were all run in parallel either way, that explanation didn't hold water. I then suspected that the matching order may be covering up a broken match:

"First match wins". The first rule that matches a test will be the one that applies.

However, I found no indication in the TAP::Parser::Scheduler source suggesting that specifying the test's full name would not result in a match. Nor did I see evidence that the order of rule processing was mishandled. In fact, TAP::Parser::Scheduler::_set_rules appeared to correctly classify my input:

#Output from dumping $self->{scheduler} before the return() in App::Prove::Scheduler->new()
$VAR1 = [
[
[
[
bless( {
'description' => 'a.t',
'filename' => 'a.t'
}, 'TAP::Parser::Scheduler::Job' )
]
]
],
[
bless( {
'description' => 'b.t',
'filename' => 'b.t'
}, 'TAP::Parser::Scheduler::Job' )
],
[
bless( {
'filename' => 'c.t',
'description' => 'c.t'
}, 'TAP::Parser::Scheduler::Job' )
]
];

After my initial shock at the dubious usefulness of such a data structure, I realized that the hugely nested first entry signified that the first entry indeed had walked the seq => codepath in _rule_clause, effectively ruling out any possibility of a match failure.

After more analysis of both TAP::Parser::Scheduler and TAP::Harness, the reason for all this becomes quite clear. The helper subs of Tap::Harness::aggregate_tests, _aggregate_parallell and _aggregate_single are just calling $scheduler->get_job*() and iterating in the sensible manner for the respective runmode.

However, the get_job() and get_jobs() subs return no information whatsoever about whether the job should be run in parallel or in sequence, and the helpers do nothing but call those subs. As such, the --rules information is effectively ignored (aside from throttling down -j to 1 when nothing but seq rules are passed). Having no information to work on it simply works the same way it did before the --rules feature was added.

As far as I can tell the right course of action would be:

• Actually check which jobs are parallel and which are seq in _aggregate_tests, by iterating over $scheduler->{schedule}, which has the information we want.
• Then call _aggregate_parallel or _aggregate_single, depending on what that test "wants", and whether our -j > 1.

The hard part with the latter, of course, is that we'd basically have to dummy up new TAP::Parser::Schedulers to feed into the aggregate_ helpers with the jobs restricted to each schedule "batch". When I say batch, I mean something like the third rules example in Tap::Parser::Scheduler:

{
seq => [
{ seq => 't/startup/*.t' }, #batch 1
{ par => ['t/a/*.t','t/b/*.t','t/c/*.t'], } #batch 2
{ seq => 't/shutdown/*.t' }, #batch3 ...
],
},

The data structure formed inside Tap::Parser::Scheduler::_set_rules is nowhere near as nice as that (it's deeply nested arrays as shown above in dumper output). That should be straightforward to clean up, given hardly anything is actually consulting it in practice.

skip_todo event arrives with the following object:

$VAR1 = bless( {
                 'description' => '',
                 'type' => 'test',
                 'explanation' => '& SKIP Testing skip',
                 'test_num' => 3,
                 'raw' => 'not ok 3 # TODO & SKIP Testing skip',
                 'directive' => 'TODO',
                 'ok' => 'not ok'
               }, 'TAP::Parser::Result::Test' );

I believe explanation should be TODO & SKIP ?

Api description or something.
Also, there is a link in pod to non-existant TAP::Parser::Formatter in prove pod

prove has many features, but for my case I'm just interested in the part where it aggregates a TAP stream (originating from a non-Perl toolchain) and reports its little summary at the bottom with Result: FAIL/PASS.

The shortest working solution I could find is

printf "1..1\nok 1\n" |         # example TAP emitter
    prove -e 'cat -' /dev/null

which is lame. The report includes the dummy filename because prove insists on having a filename.

Streams should have first-class support in prove, somehow.

There's this piece of code in Perl source handler:

    if ( my $shebang = $file->{shebang} ) {
        return 0.9 if $shebang =~ /^#!.*\bperl/;

        # We favour Perl as the interpreter for any shebang to preserve
        # previous semantics: we used to execute everything via Perl and
        # relied on it to pass the shebang off to the appropriate
        # interpreter.
        return 0.3;
    }

but actually, the $file->{shebang} contains the content of the first line of test file anyway, even if it is not the shebang - like "use strict" or whatever the first line.

Which makes the source handler vote for 0.3 for any .t file that has any content in the first line, instead of 0.9 which should be considered much more reliable.

Guess it could be rewritten as:

    return 0.3 if $shebang =~ /^#!/;

I'd like to have a plugin to do some test suite-wide setup for a project and considered an App::Prove::Plugin::MyPlugin might be the way to go.

The -I value isn't made 'live' for the _find_module function. Is this intended?

.proverc

-It/lib
-l
-r
-PMyPlugin
t

behaviour

$ prove
Can't load module MyPlugin at /sw/perlbrew-0.76/perls/perl-5.26.0/bin/prove line 13

Current workaround.

$ PERL5OPT=-It/lib prove
<snip>
All tests successful.
Files=14, Tests=142,  1 wallclock secs ( 0.04 usr  0.02 sys +  0.57 cusr  0.18 csys =  0.81 CPU)
Result: PASS

setup

prove version: TAP::Harness v3.38 and Perl v5.26.0

I wanted to check in my test script which test harness is in use. I would like to distinguish Test::Harness (ExtUtils::MakeMaker manged test) and TAP::Harness (prove managed test). How can I do this? I have checked the environment (HARNESS_* environment variables) without success.

I have installed Win32::Console (and Term::ANSIColor just in case), but do not see color output. Googling around I got a more detailed SO question. I know that color output otherwise works because ack now has colored output, and it also uses Win32::Console to achieve this.
Using Windows 7, Strawberry 5.18.1.1, Test::Harness 3.29.

Enabling test suite parallelism for installations from CPAN is problematic.

There is only the option of running all tests of a distribution in parallel, without any way for the distribution itself to influence that, by setting HARNESS_OPTIONS ( Test::Harness ) or creating a ~/.proverc file ( App::Prove ). This mechanism is currently used for two purposes:

1. Specifying a maximum limit for the test suite parallelism
2. Enabling test suite parallelism for the test suite

This is unfortunate, because not all test suites are safe for running in parallel and there is currently no way for specifying within a distribution how its test suite can be run in parallel.

A distribution should be able to opt-in into running (parts of) its test suite in parallel, instead of being forced to be run in parallel.

Currently, Test::Harness (together with App::Prove and make test) has only the most basic support for running the test suite of a distribution in parallel. Supplying additional information about which tests can be run in parallel is possible for App::prove ( but maybe see #75 ) on the command line and for Test::Harness by presence of a testrules.yml file. App::prove could respect a testrules.yml file with some small changes.

Ideally, specifying the maximum level of parallelism does not automatically enable full/blind test suite parallelism in Test::Harness and App::Prove. A solution to allow distributions and the local user to configure test suite parallelism could be the presence or absence of a testrules.yml file:

a) If a testrules.yml is present, this should be respected together with the -j option from HARNESS_OPTIONS
b) If HARNESS_OPTIONS contains (say) --parallel-default=~/.testrules.yml , this is used in the case where no testrules.yml exists in the distribution
c) -j8 does not enable test suite parallelism on its own anymore

Of course, c) means breaking behaviour, which may or may not be that great. It can be worked around by implementing something like b) .

When using TAP::Parser::Iterator::Stream with the output handle of the pipe() function, in a parallel testing scenario, the parallellization could benefit from having the stream return the handle from get_select_handles().

Please add this enhancement.

shlomif[black-hole]:~/Test-Harness$ export HARNESS_VERBOSE=1
shlomif[black-hole]:~/Test-Harness$ prove t/*.t
t/000-load.t ........... 13/97 # Testing Test::Harness 3.42, Perl 5.028001, /usr/bin/perl5.28.1
t/000-load.t ........... ok     
t/aggregator.t ......... ok     
t/bailout.t ............ ok     
t/base.t ............... ok     
t/callbacks.t .......... ok     
t/console.t ............ ok   
t/env_opts.t ........... ok     
t/errors.t ............. ok     
t/file.t ............... 1/56 ^C
shlomif[black-hole]:~/Test-Harness$ runprove t/*.t 2>&1 | head -10
t/000-load ........... 1..97
ok 1 - use TAP::Parser;
ok 2 - ... and TAP::Parser should have the correct version
ok 3 - use App::Prove;
ok 4 - ... and App::Prove should have the correct version
ok 5 - use App::Prove::State;
ok 6 - ... and App::Prove::State should have the correct version
ok 7 - use App::Prove::State::Result;
ok 8 - ... and App::Prove::State::Result should have the correct version
ok 9 - use App::Prove::State::Result::Test;
shlomif[black-hole]:~/Test-Harness$ 

I just wasted many minutes trying to understand why it doesn;t work on travis.

It is hard to watch for library changes without it. GH notifies about releases, not tags.

s/test/the/

0001-Documentation-typo-fix.patch.txt

test simply checks whether STDIN is a tty.

Such test doesn't check anything and should be removed.

Hi All,

I've been exploring the sitecustomize.pl feature. When I have my sitecustomize.pl installed then t/compat/test-harness-compat.t fails the "switches" test. A bit of bisecting shows me that a sitecustomize.pl file containing the single line of use strict; seems to be all that is necessary to cause the test to fail.

Of course, my sitecustomize.pl starts with

use strict;
use warnings;

and then goes on to use other modules which presumably also use strict;

I believe that the test is simply checking if strict.pm has been loaded and that it expects it to not have been loaded. If so then the test will fail if anything (e.g. a sitecustomize.pl) has caused strict.pm to be loaded.

Does that seem like a correct analysis?

If so, is there anything to do to make the test more robust?

Test::Harness version 3.36_01, Test::More version 1.302177

prove'ing a test which forks a daemon process doesn't exit until the daemon is killed. This is new since updating Test:: libs to all latest versions. Previously the test would exit.

This is a bit of a problem as I use a function in our test suite to restart some daemons if code has changed. This is no longer usable since the first test never exits.

Example: with prove this hangs until the sleep ends but exits immediately when run with perl

use Test::More; system('sleep 5000 &');

Also posted to Test::More: Test-More/test-more#868 (comment)

We have a number of unreleased changes, most importantly some end users are waiting on the release of #18 , @Ovid, can you cut the release? I'm willing to do it too, but I don't have a comaint bit.

We're currently bunding a fairly old version of Test::More (0.72, from 2007, the same that shipped with v5.10.0). I don't think this is particularly helpful, specially considering we're not upgrading it and pretty much everyone has a more recent T::M installed. Test::More has been in core since 5.8.0 (and was shipped with 5.6.2 too), I don't think we'd have serious bootstrapping issues on any perl we support.

With perl 5.8.9 to perl 5.14.4 I'm seeing test failures in t/source_handler.t. This probably affects some other Ye Olde Perles with old versions of some toolchain modules:

Can't exec "t/source_tests/source.sh": Permission denied at /home/david/cpantesting/perl-5.8.9/lib/5.8.9/IPC/Open3.pm line 268.
open3: exec of t/source_tests/source.sh failed at /home/david/cpantesting/perl-5.8.9/.cpan/build/Test-Harness-3.42-DyMKY5/blib/lib/TAP/Parser/Iterator
/Process.pm line 165

#   Failed test '... line 1'
#   at t/source_handler.t line 386.
#          got: undef
#     expected: '1..1'

#   Failed test '... line 2'
#   at t/source_handler.t line 386.
#          got: undef
#     expected: 'ok 1 - source.sh'
Can't exec "t/source_tests/source_args.sh": Permission denied at /home/david/cpantesting/perl-5.8.9/lib/5.8.9/IPC/Open3.pm line 268.
open3: exec of t/source_tests/source_args.sh foo failed at /home/david/cpantesting/perl-5.8.9/.cpan/build/Test-Harness-3.42-DyMKY5/blib/lib/TAP/Parser
/Iterator/Process.pm line 165

#   Failed test '... line 1'
#   at t/source_handler.t line 386.
#          got: undef
#     expected: '1..1'

#   Failed test '... line 2'
#   at t/source_handler.t line 386.
#          got: undef
#     expected: 'ok 1 - source_args.sh foo'
# Looks like you failed 4 tests of 79.
t/source_handler.t ..............
Dubious, test returned 4 (wstat 1024, 0x400)
Failed 4/79 subtests
        (less 4 skipped subtests: 71 okay)

It seems that even though t/source_tests/source_args.sh has the execute bit set in the tarball, something in the old toolchain is removing it:

perl-5.8.9 $ ls -l .cpan/build/Test-Harness-3.42-ud1QgD/t/source_tests/
total 60
-rw-r--r-- 1 david david  75 Jul  7  2013 harness
-rw-r--r-- 1 david david 112 Jul  7  2013 harness_badtap
-rw-r--r-- 1 david david 117 Jul  7  2013 harness_complain
-rw-r--r-- 1 david david 179 Jul  7  2013 harness_directives
-rw-r--r-- 1 david david 235 Jul  7  2013 harness_failure
-rw-r--r-- 1 david david 443 Sep 11  2016 psql.bat
-rw-r--r-- 1 david david 149 Jul  7  2013 source
-rw-r--r-- 1 david david  21 Jul  7  2013 source.1
-rw-r--r-- 1 david david  54 Sep 11  2016 source_args.sh
-rw-r--r-- 1 david david 103 Jul  7  2013 source.bat
-rw-r--r-- 1 david david  70 Jul  7  2013 source.pl
-rw-r--r-- 1 david david  46 Sep 11  2016 source.sh
-rw-r--r-- 1 david david  69 Jul 23  2017 source.t
-rw-r--r-- 1 david david  23 Jul  7  2013 source.tap
-rw-r--r-- 1 david david 131 Jul 23  2017 test.tap

You may want to twiddle the test to explicitly turn the execute bit on in this case.

I seem prove --harness TAP::Harness t/00_compile.t should work. But the command says following with TAP::Harness 3.31:

Unknown arguments to TAP::Harness::new (harness_class) at /home/tokuhirom/.plenv/versions/5.20.0/lib/perl5/site_perl/5.20.0/TAP/Harness/Env.pm line 118.

It works with TAP::Harness 3.30.

I think following change may break this option.

Add harness_class argument to TAP::Harness::Env (Leon Timmermans)

ref.
tokuhirom/Test-Pretty#20