nkabir / shunit2 Goto Github PK

What steps will reproduce the problem?
1. assertNotNull 'error-msg-4' "xxx there's no /x/y"

What is the expected output? What do you see instead?
The assertion above fails despite it should not.

What version of the product are you using? On what operating system?
# tested shunit2 versions: shUnit 2.1.5 and shUnit 2.0.3
# shell: GNU bash, version 3.2.48(21)-release (i686-pc-cygwin)
# uname -s: CYGWIN_NT-5.1
# uname -r: 1.5.25(0.156/4/2)
# uname -v: 2008-06-12 19:34
# uname -m: i686
# uname -p: unknown
# uname -i: unknown
# uname -o: Cygwin


Please provide any additional information below.
See attached file (test_assertNotNull.zip) that contains a test script and
the produced output for each of the tested version of shUnit.

See summary.

What steps will reproduce the problem?
1. Set up a script which is meant to bring in all tests:
----- script -----
#!/bin/bash

# contains a test function, named correctly, simple assertEquals 1 1 statements 
within
source path/to/testMyFunc.sh

# this DOES recognize the test function
# type testMyFunc

# now try to bring in shunit2
source path/to/src/shunit2
----- /script -----

What is the expected output? What do you see instead?
EXPECTED:
testMyFunc

Ran 1 tests.

OK


ACTUAL:

Ran 0 tests.

OK


What version of the product are you using? Can you provide the output from
the lib/versions script included with the source?

$ shunit2-2.1.6/lib/versions
os: Mac OS X version: 10.X.X
shell: /bin/bash version: 3.2.48(1)-release
shell: /bin/dash version: not installed
shell: /bin/ksh version: 2011-02-08
shell: /bin/pdksh version: not installed
shell: /bin/sh version: unknown
shell: /bin/zsh version: 4.3.11


Please provide any additional information below.
I verified that the test function made it into the current scope of the script 
before shunit2 was sourced. I have no idea why this wouldn't function properly. 
Without the ability to source a handful of test functions before sourcing 
shunit2, the oneTimeSetUp, etc. functions are utterly useless. If I am 
restricted to setting tests up within the same file in which they are defined, 
those setups could just be put within the test function itself.  

See summary...

I don't think shunit2 (svn version of July 16 '08) should assume code
outside of itself has set -u (no unbound variables) set. Otherwise the
behavior of the code one is testing will be different inside the test
versus when it is run not by shunit2. And although I think it wonderful for
people to write code using set -u alas most of the shell code that is
written doesn't (yet).

Suppose my shell code is for example:

  while [[ -n $1 ]] ; do
    # Do something with $1
    shift
  done

If this code gets called via shunit2, you will get an error. If it isn't
and set +u is in effect, this code works fine. That said, there are ways
rewriting the code that work using set -u such as:

  while (( $# > 0 )) ; do
    # Do something with $1
    shift
  done

but I don't think shunit2 should be enforcing programming behavior.

A fix here would be to save $- inside shunit2, "set -u"  and restore it
before running. Or remove setting set -u altogether, and add it in as a
unit test of the shunit2 code.

I'd like to see colored output: Failed asserts in red, test case names in 
yellow or blue (so I can easily differentiate them from the output), final 
output (FAILED or OK) in red or green.

I found a bug in the Makefile for 2.1.5, but there's no 2.1 branch to supply a 
patch for.

And the Makefile doesn't exist in trunk/source/2.1/  I think trunk/source/2.1 
functions as trunk.  Is trunk/source/2.0 just the 2.0 branch living in an 
unexpected location?

Should I pipe down and learn to love new repository organization schemes?  :P

Dan

What steps will reproduce the problem?
1. execute mkdir_test.sh

What is the expected output?
All tests pass

What do you see instead?
testMissingDirectoryCreation
testExistingDirectoryCreationFails
ASSERT:expected error message to stderr
testParentDirectoryCreation

Ran 3 tests.


What version of the product are you using? On what operating system?

shuint2-2.1.5

$ bash --version
GNU bash, version 3.00.15(1)-release (x86_64-redhat-linux-gnu)
Copyright (C) 2004 Free Software Foundation, Inc.

Please provide any additional information below.
Problem seems to be in the assertTrue, the else clase which evaulates the 
passed-in condition. Debugging output looks as follows:


testExistingDirectoryCreationFails
[ -d '/tmp/shunit.d19435/some_test_dir' ]
[ -z '' ]
++++ cat /tmp/shunit.d19435/output/stderr
+++ assertNotNull 'expected error message to stderr' 'mkdir: cannot create 
directory `/tmp/shunit.d19435/some_test_dir'\'': File exists'
+++ eval '__shunit_lineno="";' if '[' '"${1:-}"' = '"--lineno"' '];' 
then '[' -n '"$2"' ']' '&&' '__shunit_lineno="[$2]' '";' shift '2;' fi
++++ __shunit_lineno=
++++ '[' 'expected error message to stderr' = --lineno ']'
+++ '[' 2 -gt 2 ']'
+++ _shunit_shouldSkip
+++ '[' 1 -eq 1 ']'
+++ return 1
+++ shunit_message_=
+++ '[' 2 -eq 2 ']'
+++ shunit_message_='expected error message to stderr'
+++ shift
+++ assertTrue 'expected error message to stderr' '[ -n '\''mkdir: cannot 
create directory `/tmp/shunit.d19435/some_test_dir'\'': File exists'\'' ]'
+++ eval '__shunit_lineno="";' if '[' '"${1:-}"' = '"--lineno"' '];' 
then '[' -n '"$2"' ']' '&&' '__shunit_lineno="[$2]' '";' shift '2;' fi
++++ __shunit_lineno=
++++ '[' 'expected error message to stderr' = --lineno ']'
+++ '[' 2 -gt 2 ']'
+++ _shunit_shouldSkip
+++ '[' 1 -eq 1 ']'
+++ return 1
+++ shunit_message_=
+++ '[' 2 -eq 2 ']'
+++ shunit_message_='expected error message to stderr'
+++ shift
+++ shunit_condition_='[ -n '\''mkdir: cannot create directory 
`/tmp/shunit.d19435/some_test_dir'\'': File exists'\'' ]'
++++ expr '[ -n '\''mkdir: cannot create directory 
`/tmp/shunit.d19435/some_test_dir'\'': File exists'\'' ]' : '\([0-9]*\)'
+++ shunit_match_=
+++ shunit_return=0
+++ '[' -z '[ -n '\''mkdir: cannot create directory 
`/tmp/shunit.d19435/some_test_dir'\'': File exists'\'' ]' ']'
+++ '[' '[ -n '\''mkdir: cannot create directory 
`/tmp/shunit.d19435/some_test_dir'\'': File exists'\'' ]' = '' ']'
+++ echo '[' -n ''\''mkdir:' cannot create 
directory '`/tmp/shunit.d19435/some_test_dir'\'':' File 'exists'\''' ']'
[ -n 'mkdir: cannot create directory `/tmp/shunit.d19435/some_test_dir': 
File exists' ]
+++ '[' 1 -ne 0 ']'
+++ shunit_return=1
+++ '[' 1 -eq 0 ']'
+++ _shunit_assertFail 'expected error message to stderr'
+++ _shunit_msg_='expected error message to stderr'
+++ __shunit_testSuccess=1
++++ expr 0 + 1
+++ __shunit_assertsFailed=1
++++ expr 7 + 1
+++ __shunit_assertsTotal=8
+++ echo 'ASSERT:expected error message to stderr'
ASSERT:expected error message to stderr
+++ unset _shunit_msg_
+++ unset shunit_message_ shunit_condition_ shunit_match_
+++ return 1
+++ shunit_return=1
+++ unset shunit_message_
+++ return 1
+++ set +x

There is pair of functions: assertSame, assertNotSame.
But exists only assertEquals.
I think assertNotEquals is missing.

Example code:
----SNIP----
#!/bin/bash

test_1() {
        assertEquals "This test works" "0" "0"
}

test_2() {
        shopt -s nullglob
        assertEquals "This test crashes" "0" "0"
}

. /projekte/install/lib/src/shunit2
----SNAP----

Output:
----SNIP----
test_1
test_2
/projekte/install/lib/src/shunit2: eval: line 105: unexpected EOF while looking 
for matching `"'
/projekte/install/lib/src/shunit2: eval: line 106: syntax error: unexpected end 
of file

Ran 2 tests.

OK
----SNAP----

shunit2 version: 2.1.6

os: IBM AIX 6.1
shell: /bin/bash version: 4.1.7(1)-release
shell: /bin/dash version: not installed
shell: /bin/ksh version: --version: 0403-010 A specified flag is not valid for 
this command.
shell: /bin/pdksh version: not installed
shell: /bin/sh version: unknown
shell: /bin/zsh version: not installed

Some users like to prefix their function calls with the keyword 'function'. 
Make sure shUnit2 works 
with these as well.

The tool would be similar to the shunit_test.sh tool already provided, but 
would aggregate results 
and produce a final report.

What steps will reproduce the problem?
1. Normal test (sourced shunit2), testMyFunc() with only one line:  
${_ASSERT_EQUALS_} "1 = 2" 1 2


What is the expected output? What do you see instead?
Never seen this before so not sure what it would look like exactly. However, 
shunit2 gives me:
testMyFunc
shunit2:ERROR assertEquals() requires two or three arguments; 5 given
shunit2:ERROR 1: 1 2: = 3: 2 4: 1

Ran 1 test.

OK

What version of the product are you using? Can you provide the output from
the lib/versions script included with the source?
- 2.1.6
- versions:
os: Mac OS X version: 10.X.X
shell: /bin/bash version: 3.2.48(1)-release
shell: /bin/dash version: not installed
shell: /bin/ksh version: 2011-02-08
shell: /bin/pdksh version: not installed
shell: /bin/sh version: unknown
shell: /bin/zsh version: 4.3.11


Please provide any additional information below.

oneTimeSetUp is not piped to the output file although onTimeTearDown is.

It is not consistent. 

Given that piping onTimeTearDown creates a subshell and prevent users of 
modifying variables of shunit2 (e.g. suite name) I would propose not to pipe it 
either.

The message states "assertSame() requires one or two arguments; $# given", but 
the code is checking for 2 or 3 args.

Patch attached.

What steps will reproduce the problem?
1. svn co http://shunit2.googlecode.com/svn/trunk/ shunit2-read-only
2. cd shunit2-read-only/source/2.1/src/
3. ./shunit2_test.sh > result.txt

What is the expected output?
  No errors

What do you see instead?
  Two ksh errors

What version of the product are you using?
  Subversion HEAD

Can you provide the output from the lib/versions script included with the 
source?
  See attachment

Please provide any additional information below.
  I *just* installed ksh for this test, I'm the only interactive user, and I've never used it before on this machine, so the installation is completely vanilla.
  $ ksh --version
    version         sh (AT&T Research) 93t+ 2010-03-05
  $ uname -a | sed "s/$USER/jdoe/"
  Linux jdoe 2.6.35-30-generic #56-Ubuntu SMP Mon Jul 11 20:01:08 UTC 2011 x86_64 GNU/Linux

What steps will reproduce the problem?
1. Run a test like this: ${_ASSERT_EQUALS_} "some msg" 'x' 'y'

What is the expected output? What do you see instead?
I got this error:
ASSERT:[13] some msg expected:<x> but was:<y>
but got this instead:
shunit2:ERROR assertEquals() requires two or three arguments; 4 given
shunit2:ERROR 1: some 2: msg 3: x

What version of the product are you using? On what operating system?
2.1.5

Please provide any additional information below.

In bin/docbookPrep.sh, line 24 the path for shlib is set to

${MY_DIR}/../lib/sh/shlib

but the file is located in

${MY_DIR}/../lib/shlib

assertTrue does not check for zero args.  It should use the same error checking 
as assertFalse:


*** shunit2     Thu Sep 15 10:35:01 2011
--- shunit2.patched       Thu Sep 15 10:38:25 2011
***************
*** 315,321 ****
  assertTrue()
  {
    ${_SHUNIT_LINENO_}
!   if [ $# -gt 2 ]; then
      _shunit_error "assertTrue() takes one two arguments; $# given"
      return ${SHUNIT_ERROR}
    fi
--- 315,321 ----
  assertTrue()
  {
    ${_SHUNIT_LINENO_}
!   if [ $# -lt 1 -o $# -gt 2 ]; then
      _shunit_error "assertTrue() takes one two arguments; $# given"
      return ${SHUNIT_ERROR}
    fi



URL: http://shunit2.googlecode.com/svn/trunk/source/2.1/src
Revision: 337

(Minor, I know, but I found it by making a mistake and diagnosing the problem.)

Great program!

It took a while, but I found something in shunit2, version 2.1.2, that
looks a little odd. If the interpreter decides to die, then you don't
get any report, even though shunit2 may have accumulated some useful
statistics.

Here's a little example:

  boom() { x=$1; }

  test_boom()
  {
    assertEquals 1 1
    boom; # No parameter given
    assertEquals 0 $?
  }
  . ./shunit2

When I run this I get:
  #
  # Performing tests
  #
  test_boom
  ./shunit2-bug.sh: line 1: $1: unbound variable

There was an assert that ran and that tested okay. Furthermore I think
if there were other tests their status would not be reported as well.

Attached should be one possible patch to address. With this the output
you get is:

  test_boom
  ./shunit2-bug.sh: line 1: $1: unbound variable
  ASSERT:Unknown failure encountered running a test

  #
  # Test report
  #
  tests passed:     1  50%
  tests failed:     1  50%
  tests skipped:    0   0%
  tests total:      2 100%

The regular expression in _shunit_extractTestFunctions (^[  ]*(function 
)*test[A-Za-z0-9_]* *\(\)) cannot detect ksh style functions, due to it 
expecting the "()" to follow the function name. This causes shunit2 to ignore 
all tests when running against a ksh compatible code-base. The following 
function syntax is possible in ksh:

function foo {
  ...
}

function foo
{
  ...
}

foo() {
  ...
}

foo()
{
  ...
}

Adding a prefix for better test recognition. Prefix is added before test 
function echo.
External control is via TEST_PREFIX=<my_prefix> possible.
Default: "------ TEST :: "

What steps will reproduce the problem?
1. Install shunit2 via homebrew (I'm on OS X 10.8.*): $ brew install shunit2
2. Verify install:
    ../Cellar/shunit2/2.1.6/INSTALL_RECEIPT.json
    ../Cellar/shunit2/2.1.6/bin/shunit2
3. Create simple test:
----- script -----
#!/bin/bash
# tests/myFunc.sh
testMyFunc() {
    assertEquals 1 1 || return
}

shunit2
----- /script -----
4. Run test:  $ ./tests/myFunc.sh


What is the expected output? What do you see instead?
EXPECTED:
testMyFunc

Ran 1 test.

OK

ACTUAL:

Ran 0 tests.

OK


What version of the product are you using? Can you provide the output from
the lib/versions script included with the source?
- Using shunit v2.1.6
- Only binary is included from homebrew install


Please provide any additional information below.
As a test, I downloaded the .tgz file for 2.1.6 and then replaced the shunit2 
call with a source call pointing to the absolute path of the shunit2 binary. 
Doing that provided the expected results. However, I would much prefer using 
the binary version as it is path-independent.

Am I just using the binary incorrectly? Is the homebrew install flawed?

The below script crashes but would works fine if you remove '-e'

    #! /bin/sh -e

    testTrue()
    {
      assertTrue 0
    }

    . shunit2


I'm using shunit2 v2.1.6-1 from ubuntu raring

Running a test with a macro with spaces in the optional message results in 
invalid number of 
arguments errors.

Test:
${_ASSERT_EQUALS_} 'some msg' 'x' 'y'

Results:
ASSERT:_ASSERT_EQUALS_ failure
shunit2:ERROR assertEquals() requires two or three arguments; 4 given

The messages and arguments in macros need to be re-quoted.

Dunno if i'm doint something wierd here, to get that error msg.. anyway...

What steps will reproduce the problem?

On fresh ubuntu 9.10 install
1. wget http://shunit2.googlecode.com/files/shunit2-2.1.5.tgz
2. tar -zxvf shunit2-2.1.5.tgz
3. cd shunit2-2.1.5
4. make

To fix it; edited bin/docbookPrep.sh and changed the lib/shlib path
. "${MY_DIR}/../lib/shlib"

What steps will reproduce the problem?
1. cd shunit2-2.1.5
2. make test

What is the expected output? What do you see instead?
Expected to see tests being run.
Saw:
make: *** No rule to make target `@echo', needed by `test'.  Stop.


What version of the product are you using? On what operating system?
2.1.5 on Cygwin.

Please provide any additional information below.
This should fix it:
--- Makefile.orig       2008-07-04 13:47:58.000000000 +0100
+++ Makefile    2010-03-30 10:23:59.189518300 +0100
@@ -70,7 +70,8 @@
          $(BIN_DIR)/docbookPrep.sh "$(DOCBOOK_SHARE_DIR)"; \
        fi

-test: @echo "executing $(PROG) unit tests"
+test:
+       @echo "executing $(PROG) unit tests"
        ( cd $(TEST_SRC_DIR); ./shunit2_test.sh )

 dist: dist-clean build docs

If a test suite dies early because of an interpreter failure, shunit2
(version from SVN of July 6, '08) will not report those tests which were
not run nor give a count where one can determine that tests were not run
because of a premature failure.

I believe this is easily fixed by counting the number of tests in
suite_addTest and using that in _shunit_generateReport

Also I am not sure shunit2 ever reports that a test is skipped. 

If you want me to work up a patch I'd be happy to do that. What prevents me
from doing this now is that "test" is used to refer to functions that run
asserts as well as a count of the assert statements.

I observed that the following functions are not working properly in the
2.1.5 version of shUnit2:

- failNotEquals
- failSame
- failNotSame

AFAIK, failNotEquals should fail when given parameters are not equal so
passing '1' and '0' should fail, while '1' and '1' should not. However I
always get failure, regardless of the parameters:

ASSERT:fail: not equals expected:<1> but was:<1>
ASSERT:fail: not equals expected:<1> but was:<0>

Same happens with testFailSame. This is what I tried...

failSame "failSame FAILED" "text" "text"
failSame "failSame FAILED" "text" "not same text"

...and this is what I got:

ASSERT:failSame FAILED expected not same
ASSERT:failSame FAILED expected not same


And, at last, same happens with testFailNotSame. I tried this...

failNotSame "failNotSame FAILED" "text" "text"
failNotSame "failNotSame FAILED" "text" "not same text"

...and I got this:
ASSERT:failNotSame FAILED expected:<text> but was:<text>
ASSERT:failNotSame FAILED expected:<text> but was:<not same text>


Perhaps I'm misunderstanding something, but I imagine this could be a bug,
so here you are the report. If not, I'd be very happy to know why, so I
could understand the desired behaviour better, so thanks in advance ;-)

Otherwise, please take a look into the patch I'm attaching (done against
2.1.5 version of shunit2, from googlecode SVN), I'm using it right now and
it seems to work fine
:-)


At last, just to say I've submitted this same bug in the old bugzilla at SF
(http://sourceforge.net/tracker/index.php?func=detail&aid=2062264&group_id=18800
1&atid=923569),
before realizing of googlecode being the current right place to do it, so
excuse the "spam"... 

However, as the bug seems to be there yet, I updated my patch (formerly
made against 2.0.4 vversion) according to the last code from 2.1.5 revision
of shUnit2, and that's what I'm uploading right now. Hope it helps.

Br,

Mario

Hi Kate,

Great to see you've created a unit test framework for shell but I'm
having a hard time finding any examples beyond those in log4sh and
shflags.  I've previously been writing unit tests for Java and using
Mockito for mockingn all external dependencies.  I'm now
trying to use those same techniques when testing my scripts with shunit2
and would like your suggestions for patterns to test them.

I've listed here examples of two tests which I have created along with
my thoughts on how I've written them and would be very interested to
hear how you would go about it.  I've got a suspicion that there
should be an easier way of doing some of these.

For testing the first example (below) I need to have
* a directory which will need to exist
* a call to 'make'

My initial approach
* the test to set SOURCE to point to a sandbox testing area which
contains the required directories, in this case 'buildarea'.  This
works ok.
* create a mocked 'Makefile' (see below) inside '${SOURCE}/buildarea'
which outputs the command line options to a file 'Makefile.output'.
Checking that this file exist confirms that 'make' was run
successfully and further checking of the content of this file
validates that 'make' was given the correct parameters.

This seemed to work ok so I moved onto writing my second test

---------------------------------------- First Example
----------------------------------------
-- Code being tested --

function make_kernel() {
       infoecho "Making kernel..."

       VARIANT_DIRECTORY="${SOURCE}/buildarea"
       cd ${VARIANT_DIRECTORY}

       make O=${OBJECT_DIRECTORY}
       RESULT=$?

       if [[ ${RESULT} != 0 ]]; then
               infoecho "Make kernel, calling make failed. Exiting"
               exit ${RESULT}
       fi
}


-- Test Code --

testMakeKernel()
{
       export SOURCE=${whereAmI}/fakedirectories/mwfake
       export OBJECT_DIRECTORY="1234"

       # Remove the output file which will be generated from the
Makefile
       #
       outputfile="${SOURCE}/buildarea"
       rm "${outputfile}"

       make_kernel

       result=$?
       assertTrue "make failed" "[ $result = 0 ]"

       # Make sure the output file exists
       #
       assertTrue "make was called" "[ -f ${outputfile} ]"
       result=`cat ${outputfile}`

       # Make sure it has the correct parameter inside it
       #
       expecting="O:1234"
       assertTrue "Makefile result: expected '${expecting}', but got
'${result}'" "[ '${result}' = '${expecting}' ]"
}

-- Mock Makefile --

default:
       @echo "O:$(O)" > Makefile.output

---------------------------------------- End Of First Example
----------------------------------------

For testing the second example (below) I need to have
* a call to an external command, in this case 'find'
* know data coming back from find
* a call to another external command 'tar'

Now with this test I would like to
* Mock the find command
* Mock the tar command
* Count command invocations to 'find' and 'tar'

My approach was to create a set of mock functions for tar and find
which would produce the required fake data.  This would entail setting
a variable MOCK_FIND_RESULT="directory/file1.spk.tar.gz directory/
file2.spk.tar.gz" so that when 'find' is called it uses
MOCK_FIND_RESULT to produce the result.  This also seemed to work ok.

function find()
{
       # Return preset mock results
       #
       echo "${MOCK_FIND_RESULT}"

       return ${MOCK_FIND_EXIT_STATUS:-0}
}

I have also done the same thing with tar

function tar()
{
       return ${MOCK_TAR_EXIT_STATUS:-0}
}


Counting the number of invocations has however proved tricky and I'm
still working on it.  Basically I added a counter variable inside the
functions like so

function tar()
{
               TARCOUNT=$(expr ${TARCOUNT} + 1)
       return ${MOCK_TAR_EXIT_STATUS:-0}
}

This looks to work fine until we run our test as the code being tested
has the 'tar' command inside the 'for in find' this produces a pipe,
hence a seperate shell.  The counter variable is now incremented in
this additional shell and therefore has no effect on the test
variables.


-------------------------------------------- Second Example
------------------------------------------
-- Code being tested --
function extract_spk_tarballs() {
       infoecho "extract_spk_tarballs: source is ${SOURCE}"
       for filename in `find "${SOURCE}" -name "*spk*.tar.*"`;
       do
               dir=`dirname "${filename}"`

               infoecho "Extracting ${filename}"
               tar xf "$filename" -C "${dir}"
               RESULT=$?

               if [[ ${RESULT} != 0 ]]; then
                       infoecho "Extract spk tarballs, tar failed.
Exiting"
                       exit ${RESULT}
               fi
       done
}


-- Test Code --
test_extract_spk_tarballs() {
       export SOURCE=${whereAmI}/fakedirectories/mwfake

       # Setup two fake results for spk tarballs
       #
       export MOCK_FIND_RESULT="fakedir/file1.spk.tar.gz fakedir/
file2.spk.tar.gz"

       extract_spk_tarballs
       result=$?
       assertTrue "extract_spk_tarballs failed" "[ $result = 0 ]"

       # Can't guarantee that we can count invocations. Setting a
variable in a pipe won't work
       #
       #validateCallCount "tar" "2" "${TARCOUNT}"
}
---------------------------------------- End Of Second Example
---------------------------------------


-- Additional Thoughs --
Instead of creating methods for each mocked command I could use alias
instead, at least for mocking some of them.  This would mean that the
mock commands would be unique and data specific for each test which
would be much better

               alias find="echo fakedir/file1.spk.tar.gz
fakedir/file2.spk.tar.gz"

I see that you already have a ticket offering JUnit/XML ( #16 ) and another 
suggesting TAP ( #8 ).  Neither of those formats collects all of the data that 
I need.  I suggest a generalized plugin architecture is a better answer than 
coding these specific formats into shunit2.

I wanted shunit2 to send results to my Pandokia test reporting system, so I 
implemented a plugin architecture and a plugin that produces data files in the 
pandokia format.

I propose that you consider this plugin architecture (but not the pandokia 
plugin) for inclusion in a future version of shunit2.  I am attaching:

- a patch to shunit2 that implements a plugin architecture for shunit2 to write 
test reports in other formats; this includes a default plugin (built in to the 
shunit2 script) that preserves the existing behaviour of shunit2.

- an example plugin that makes something that looks vaguely like XML; this 
could be enough clues for someone more familiar with JUnit/XML or TAP to write 
a suitable plugin for those formats.  (I can't implement them myself because I 
don't know anything about them, I don't have a test environment, and I don't 
need them.)

The plugin that produces Pandokia logs does not belong in shunit2, but you can 
see it at 
https://svn.stsci.edu/trac/ssb/etal/browser/pandokia/trunk/commands/shunit2_plug
in_pdk .  It might be useful as an example while you are evaluating the patch.

The general principle of the patch is this:

Various points in the shunit2 execution are functions that can be redefined.  
If you invoke "shunit --plugin name file.sh", it uses ". shunit_plugin_$name" 
to load functions from that file into the shunit2 script.  The various plugin 
functions are either redefined by the plugin or left at the default 
implementation, as appropriate.  Test execution then proceeds as usual.


Some notable features of the code:

The plugin entry points are argv processing, init, execute a test, final 
report, and finish.  If you identify additional entry points, they are easy to 
define as further patches to shunit2.

The global variable _shunit_last_test_status reports the status of the most 
recently completed test.  _shunit_assertPass, _shunit_assertFail, and 
_shunit_assertSkip update the status.  The status is available to the plugin to 
include in a report.  I see that shunit2 tests often contain multiple 
assertions in a single test function.  The status is only allowed to go from 
less serious to more serious during a single test function.  That is, Pass can 
change to Fail if another assertion is executed, but a second assertion can 
never change Fail to Pass.


I principally run tests with "shunit2 filename", not by writing the file to 
contain ". shunit2"; this is the mode that the plugin works best.  I know of 
two problems related to this in the patch as it stands now:

1) Plugins can be loaded by "shunit2 --plugin".  I did not yet figure out a 
good way to make the plugin architecture work universally in 
__SHUNIT_MODE_SOURCED.  The problem I found is that shunit2 is sourced many 
times, so the plugin init and finish are called more than once each.  I think 
there is a solution that involves counting how deeply the shunit processes are 
nested, but I have not worked out the details.

2) I can't seem to run any of the tests as single files, except for 
shunit2_test_standalone.sh;  for example, if I type "shunit2 
shunit2_test_misc.sh", it gives an error instead of running tests.  That same 
error comes out if I try to run shunit2_test.sh with a plugin.  I think that 
the case of "shunit2 file.sh" should work even if file.sh ends with ". 
shunit2"; somehow the second invocation of shunit2 should recognize that it is 
redundant and not do anything.  I think that is a separate patch, though.


I am including a sample test file and output that it produces in various modes:

sample.shunit - a sample set of tests

PDK_DEFAULT.LOG.sample - pandokia log file from "pdkrun --test_run sample 
sample.shunit"; possibly not especially interesting to you, but for me, this 
was the point of doing all this.  The format is human readable and fairly 
obvious.

example.txt - stdout of "shunit2 --plugin example --file sample.xml 
sample.shunit"

sample.xml - xml file produced by "shunit2 --plugin example --file sample.xml 
sample.shunit"

regression.txt - output of "./shunit2_test.sh > regression.txt" shows that the 
original tests still pass


The actual patch is attached as:

shunit2.patch - svn diff of shunit2 comparing 
http://shunit2.googlecode.com/svn/trunk/source/2.1/src/shunit2 rev 337 to my 
changed copy

shunit2_plugin_example - example plugin; not useful itself, but as a basis for 
experimenting


You'll note that this patch includes only code, no documentation.  If you 
accept the patch, I offer to write documentation updates.  If you decline the 
patch, I still have my locally modified copy, but I do not need to document it 
further.

I would appreciate your comments about whether you view this as something that 
you would incorporate into shunit2.  Obviously, with the amount of work I've 
put in to it, I will need to make alternate plans if you are not interested.  
If you have an immediate response, even if it is not definite, please let me 
know.

Regards,

Mark

Broken link in http://code.google.com/p/shunit2/wiki/ProjectInfo

Link http://forestent.com/wiki/ShUnit2:AbsolutePathExample on that page is
broken.

Not an issue but an enhancement. I noticed that the skip count was displayed, 
but not which tests were actually skipped. Since some of my tests only occurs 
in certain situation (i.e. running as root or running as user), I needed to 
display the skip tests to make sure I was not skipping too much.

The attached patch adds a flag named SHUNIT_REPORT_SKIP that, if set, will 
display the skipped test. Default behavior is the original behavior: do nothing.

Hope this help others.

Hello,

I would like to submit you a bunch of changes:
  - adds a trick to prevent tested scripts to prematurely exit,
  - fixes a typo in docbookPrep.sh,
  - translates output report format to jUnit XML format,
  - adds a command to run all test suites in a directory and optionally
write test reports to a specific directory (shunit2_testsuite),
  - adds a command to join all test suite reports in a directory into a
single one report (shunit2_collect).
  - adds a RPM specfile.

The API was not changed at all. And by default, reports are displayed to
stdout except if the SHUNIT_OUTPUTDIR is set. In that situation, the report
is saved to $SHUNIT_OUTPUTDIR/TEST-<test suite name>.xml, eg
/tmp/testsuites/reports/TEST-foo.xml for test suite foo.sh.

Please note, that due to network issues, I was not able to start coding
from the latest version (r294) but instead I started from the latest stable
release ie 2.1.5.

Nonetheless, I let you assess the pertinence of those changes and let you
pick whatever is needed to the project. Please help yourself!

Regards,

Frédéric MOULIN.

Hello,
This is not a bug report, but rather a suggestion or feature request.

Suppose shunit2 were made executable, and suppose its Makefile had an
"install" target which installed it to /usr/local/bin.

Then, it could be invoked from the test script without needing to hard-code
the path:

  . $(which shunit2)

Granted the only reason it would be made executable is so that the "which"
program could find it.  However, if you run it as an executable it simply
reports that no tests were run, which can be considered graceful degradation.

And/or, it could be improved to accept a test-script via the command line,
so you could invoke it like:

  shunit2 my-tests

This change would make it easier to deploy tests that depend on shunit2
without bundling shunit2 with the application.

shunit2-2.1.5$ make
cp -p /mydoc/src/shunit2-2.1.5/src/shell/shunit2 /mydoc/src/shunit2-2.1.5/build
Preparing documentation for parsing
Extracting the ShellDoc
Parsing the extracted ShellDoc
Preparing DocBook structure
/mydoc/src/shunit2-2.1.5/bin/docbookPrep.sh: line 24:
/mydoc/src/shunit2-2.1.5/bin/../lib/sh/shlib: No such file or directory
make: *** [docs-docbook-prep] Error 1

================================

The resolution to the problem is to fix the incorrect path on the line
indicated in docbookPrep.sh.

Getting this:

src/shell/shunit2: /tmp/shunit.LpTQbv/oneTimeSetUp: /bin/sh: bad interpreter: 
Permission denied
src/shell/shunit2: /tmp/shunit.LpTQbv/suite: /bin/sh: bad interpreter: 
Permission denied
src/shell/shunit2: /tmp/shunit.LpTQbv/setUp: /bin/sh: bad interpreter: 
Permission denied

Guessing it's because of this:

/dev/sda2 on /tmp type ext3 
(rw,noexec,noatime,nodiratime,data=ordered,commit=300)

Maybe the temp dir could be made inside the current directory by default?

What steps will reproduce the problem?
$ zsh
$ echo $SHLVL
2
$ . ~/src/shell/shunit2
$ echo $SHLVL
1

What is the expected output? What do you see instead?
SHLVL should be 2.  shunit should not exit because it is being sourced, 
executed.

What version of the product are you using? On what operating system?
shunit2-2.1.5, Linux 2.6.32, zsh 4.3.6.

Please provide any additional information below.

Running the provided test suite ends up in the following:
---SNIP---
#------------------------------------------------------------------------------
# System data
#

# test run info
shells: /bin/sh /bin/bash /bin/dash /bin/ksh /bin/pdksh /bin/zsh
tests:  shunit2_test_asserts.sh shunit2_test_failures.sh shunit2_test_macros.sh 
shunit2_test_misc.sh shunit2_test_standalone.sh

# system info
$ date
Fri Mar 23 09:43:40 NFT 2012

$ uname -mprsv
AIX 1 6 00F6C8B24C00 powerpc


#------------------------------------------------------------------------------
# Running the test suite with /bin/sh
#
shell name: sh
shell version: unknown

--- Executing the 'asserts' test suite ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 8 tests.

OK

--- Executing the 'failures' test suite ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'macros' test suite ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'misc' test suite ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
testIssue7
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions

Ran 6 tests.

FAILED (failures=3)

--- Executing the 'standalone' test suite ---
testStandalone

Ran 1 test.

OK


#------------------------------------------------------------------------------
# Running the test suite with /bin/bash
#
shell name: bash
shell version: 4.1.7(1)-release

--- Executing the 'asserts' test suite ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 8 tests.

OK

--- Executing the 'failures' test suite ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'macros' test suite ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'misc' test suite ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
testIssue7
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions

Ran 6 tests.

FAILED (failures=3)

--- Executing the 'standalone' test suite ---
testStandalone

Ran 1 test.

OK

shunit2_test.sh:WARN unable to run tests with the /bin/dash shell


#------------------------------------------------------------------------------
# Running the test suite with /bin/ksh
#
shell name: ksh
shell version: --version: 0403-010 A specified flag is not valid for this 
command.

--- Executing the 'asserts' test suite ---
testAssertEquals
testAssertNotEquals
testAssertSame
testAssertNotSame
testAssertNull
testAssertNotNull
testAssertTrue
testAssertFalse

Ran 8 tests.

OK

--- Executing the 'failures' test suite ---
testFail
testFailNotEquals
testFailSame

Ran 3 tests.

OK

--- Executing the 'macros' test suite ---
testAssertEquals
testAssertNotEquals
testSame
testNotSame
testNull
testNotNull
testAssertTrue
testAssertFalse
testFail
testFailNotEquals
testFailSame
testFailNotSame

Ran 12 tests.

OK

--- Executing the 'misc' test suite ---
testUnboundVariable
ASSERT:assert message was not generated
ASSERT:test count message was not generated
ASSERT:failure message was not generated
testIssue7
testPrepForSourcing
testEscapeCharInStr
testEscapeCharInStr_specialChars
testExtractTestFunctions

Ran 6 tests.

FAILED (failures=3)

--- Executing the 'standalone' test suite ---
testStandalone

Ran 1 test.

OK

shunit2_test.sh:WARN unable to run tests with the /bin/pdksh shell

shunit2_test.sh:WARN unable to run tests with the /bin/zsh shell
---SNAP---

Output of ./lib/versions:
---SNIP---
os: unrecognized version: unrecognized
shell: /bin/bash version: 4.1.7(1)-release
shell: /bin/dash version: not installed
shell: /bin/ksh version: --version: 0403-010 A specified flag is not valid for 
this command.
shell: /bin/pdksh version: not installed
shell: /bin/sh version: unknown
shell: /bin/zsh version: not installed
---SNAP---

OS: IBM AIX 6.1

shunit2-version: 2.1.6

The 2.1 documentation refers to "stopSkipping()", but that does not exist. The 
name in shunit2 is "endSkipping()".

What steps will reproduce the problem?

1. Run the following shell script:
#!/bin/ksh
test_shunit2() {
        assertEquals "My message." 1 2
}
. ./shunit2

What is the expected output?
#
# Performing tests
#
test_shunit2
ASSERT:My message. expected:<1> but was:<2>

#
# Test report
#
tests passed:     0   0%
tests failed:     1 100%
tests skipped:    0   0%
tests total:      1 100%

What do you see instead?
#
# Performing tests
#
test_shunit2
ASSERT:My message.My message. expected:<1> but was:<2>

#
# Test report
#
tests passed:     0   0%
tests failed:     1 100%
tests skipped:    0   0%
tests total:      1 100%

What version of the product are you using?
shUnit 2.1.4

On what operating system?
AIX (IBM Unix)

Using Korn Shell

As described in the TODO.TXT:

"Make it possible to execute a single test by passing the name of the test on
the command line"

It would be very nice if it would be possible to execute single tests. Often, 
when i'm writing tests, I have to first test the test to know it is working 
properly, before I trust its output. 

Often, I don't know if something will work, and just running the test is the 
easiest thing to do to find out. However I have to run all tests each time, 
there seems to be no option to execute just a single specific test. 

So this feature would be very much appreciated.