vrakeshreddy / mb-unit Goto Github PK

Provide basic functionality:

1. Loading tests.
2. Selecting tests to run in a test tree.
3. Running tests and displaying progress.
4. Displaying test results.
5. Generating reports.

Specifically excluded from this task:

* Extensibility stories.
* Loading/saving projects.
* Accessibility enhancements.

Need to figure out how best to enable Explicit test execution.  The way
MbUnit v2.4 does it isn't very satifying as it assumes that since filters
have been set we should run all explicit tests that match those filters.

It's not clear to me we can do much better than that though...

Basic stories:

1. Implement Assert, CollectionAssert and other simple assertion classes.
2. Support [TestFramework], [Test], [RowTest], and [CombinatorialTest].
3. Support execution logs.
4. Support dynamically generated test steps.
5. Fill in enough of the rest so that Gallio can self-host its unit tests.

We should investigate whether CLS compliance of the Core will help us
accept plugin contributions from a larger audience.  I'm unclear at this
time just how useful CLS compliance will be or how much work it might take
to make the whole API clean.  Under this headline are also other
assumptions about how plugin assemblies get loaded which might cause issues
for plugins written in dynamic languages like IronPython.

Some things might not be very portable but we should at least try to figure
out what those things are.

Originally reported by Marc Stober:
http://www.mertner.com/jira/browse/MBUNIT-66

This feature calls for a variety of actions such as making HTML reports
more dynamic.  Existing uses of Javascript in the HTML report to show/hide
panels have been useful.  That idea could be taken a step further by
providing dynamic sorting/grouping based on DOM manipulations.  Obviously
the report should remain usable without Javascript.  I can imagine a few
such affordances like the one described below would go a long way to
enhancing the usability of large MbUnit reports presented outside of the
MbUnit GUI.

=====

Summaries by Variable:

This is the basic idea for a new feature. Thoughts on exactly how this
could be designed and implemented are welcome!

The idea is letting the user see trends by variable in data-driven testing
(especially combinatorial tests).

For example if we have:
[CombinatorialTest] public void MyTest (int a, int b) {
    Assert.AreNotEqual(b, 0);
}

The test would pass for every value of a, but fail whenever the value of b
is 0. In a complex real-world scenario it's not always easy to see that a
particular value of a variable is responsible for a set of failures, so we
could add a "summary by variable" to the report; for example.

a = 0 RG
a = 1 RG
b = 0 RR <- fully red bar - would draw your attention to this issue
b = 1 GG

(In the above R and G are standing in for red and green areas of the result
bars.) 

Icarus should save the name of the currently open project and of test
selections made in the tree and automatically restore them on next load. 
When working with project files, this can be done by creating a *.user file
next to the project file.  When working with plain old assemblies we can
save the information in the user's profile folder (AppData).

I'm opening this issue to make sure we don't lose this important usability
feature from v2.4 as originally reported here:
http://www.mertner.com/jira/browse/MBUNIT-65

We need to port the Resharper Add-in to Gallio.  I'd like to get this
feature in soon so we can show it off to people when Gallio gets formally
introduced to the world.

A couple of notes:

1. The code for the add-in should be moved to the Gallio source tree under
src/Runners and follow a similar pattern to MbUnit.TDNet.AddIn.  A good
name for the project would be MbUnit.Resharper.AddIn for consistency.

2. Required redistributable Resharper libraries should go into
libs\Resharper.  For the alpha release, we will not address version
conflicts.  Consequently let's build against the R# 3.0.2 libraries just
for now.  The R# team plans to improve the versioning story soon anyways.

3. The Build.msbuild script will need to be updated with a couple of items
to build the new addin and copy the relevant binaries to the bin folder. 
This can all follow the same pattern as for MbUnit.TDNet.AddIn.

Thanks!

ExpectedExceptionAttribute doesn't work as expected either. When marknig a 
test method with the ExpectedExceptionAttribute from nunit, and running it 
with mbunit, the test always succeeds, even when the first and only call 
inside the method is Assert.Fail

XmlAssert.AreEqual() does not contain any assertions. As a result, the 
test passes on documents that the underlying XmlDiff flags as different. 

Something like this (borrowed from the implementation of AreIdentical()) 
should do the trick: 

            DiffResult diffResult = xmlDiff.Compare(); 
            Assert.AreEqual(equal, diffResult.Equal, 
xmlDiff.OptionalDescription);

Icarus, Echo and the build tasks should all be able to load and run
pre-made projects.

Requirements:

1. Must persist selections in a portable way.  (May depend on ensuring that
test id's are actually stable identifiers.)
2. Must persist tool-specific settings per-project in an extensible manner
that does not depend on or interfere with other tools.  Perhaps a per-tool
collection of plain key/value pairs will suffice.
3. Must be relatively clean so that the project file can be edited by hand
(by power users) and version controlled.  Small changes to the project file
in a GUI should not overly disrupt the layout of the file.

As Phil notes ( http://haacked.com/archive/2007/05/10/productive-unit-
testing-with-specialized-assertion-classes-in-mbunit.aspx ), this section 
of the documentation ( 
http://www.mertner.com/confluence/display/MbUnit/Assertions ) is a little 
sparse.

We should be able to use most of the Gallio API from CLS-compliant
languages.  Some things might not be very portable but we should at least
try to figure out what those things are.

Echo shouldn't show contributor names.

Implement the [Agent] attribute described in the Gallio Ideas file for
creating multi-threaded tests.  Also add support for condition variables
and other required synchronization primitives.

This feature will be of great value to Ingenio.

When you run a test with TD.NET by selecting the option "Test With...
Debugger" an error occurs at the end of the execution. I've attached the 2
dialogs that appear. This happens after all the code of the
MbUnitTestRunner has been executed, and since I've seen the same error to
occur when some unhandled exception is thrown, I'm inclined to think that
it may be caused by a bug in Gallio, probably when disposing resources.

Extract a general pattern for testing types based on partially declarative
contracts.  This builds on the notion of [TypeFixture] and other attributes
but extracts more common patterns in a reusable form.

(Note: This is all highly speculative.)

public class Contract<T>
{
    public T Instance;
}

[Contract]
public class CollectionContract<T> : Contract<ICollection<T>>
{
    [Invariant]
    public void CountAlwaysEqualsNumberOfItemsInEnumeration()
    {
        int enumerationCount;
        foreach (T item in Instance)
            enumerationCount += 1;

        Assert.AreEqual(enumerationCount, Instance.Count, "Collection count
should equal number of items in enumeration.");
    }

    // wrappers check pre/post conditions around an operation
    public void Add(T item)
    {
        int oldCount = Instance.Count;

        Instance.Add(item);

        Assert.AreEqual(oldCount + 1, Instance.Count, "Add should increase
collection count by 1.");
    }

    // specific tests
    [Test]
    public void AddThenRemove()
    {
        // etc...
    }
}

[Contract]
[PropertyContract("NonNullableProperty", Nullable=false)]
[PropertyContract("ComplexTypeProperty", Factory="SomeFactory")]
[PropertyContract("IgnoredProperty", Ignored=true)]
[PropertyContract("FancyProperty", Contract=typeof(SomePropertyContract))]
public class POCOContract<T> : Contract<T>
{
}

All assertions should call IncrementAssertCount() (see this thread 
http://groups.google.co.uk/group/MbUnitDev/browse_thread/thread/ff931231b08
0f9e9/4ca0d484fb17c6b1?lnk=gst&q=assertcount&rnum=1#4ca0d484fb17c6b1 ), 
this is not currently done in a consistent manner. The assert tests should 
also be modified to check the assert count before and after each test. 
Also, a lot of assertions call other assertions internally which 
artificially inflates the assertion count - in my opinion if you call one 
assert in your test code the count should increase by one.

1. Tolerate missing tools dependencies by disabling the building of the
affected components if necessary.

2. Update the NSIS script to tolerate missing components by excluding the
relevant installer sections.

3. Insert a check for VS Express Edition and disable installation of VS
Help integration in that version.  At least until we can clear up any
possible VSIP licensing issues.

Should be able to specify whether the Gallio add-in should take over the
NUnit and MbUnit v2 add-ins (by using a lower priority) or leave them alone.

TODO:

1. Implement remaining task options including Filter specification.
2. Report failures correctly.
3. Write unit tests.
4. Write an integration test that actually launches a little MSBuild
project with a sample test project to use as a sanity check.
5. Clean up the XML documentation of the task parameters so we can generate
high quality user docs from it.  Include example task usages.

Writing tests for Plain old .Net Classes is pretty boring.  To get coverage
you have to get/set each property and check for exceptions.  Several people
have addressed this problem before with reflection-based tweaks but they're
a little clumsy.

We should provide something out of the box that's better.  For example, we
can add a new attribute that automatically provides test cases for all
public properties of some type.  For properties that require extra work, we
can define a naming convention so allow various aspects of the generated
tests to be configured.  We might also be able to provide declarative
specifications of property contracts like exception behavior.

eg.

[TestFixture]
[GenerateTestsForProperties(typeof(SomeType))]
[PropertyContract(typeof(SomeType), "Foo",
CommonContracts.ThrowsArgumentNullExceptionIfValueIsNull)]
public class SomeTypeTest
{
    [Test]
    public void PropertyFoo()
    {
        // overrides the auto-generated test for property Foo of SomeType.
    }

    [Factory]
    public SomeValueType CreateSomeValueType()
    {
        // return a dummy instance of a type used in property setter
    }
}

        [Test]
        public void StringFormatEqualsTest()
        {
            string firstString = string.Format("{0}", "test");
            string secondString = string.Format("{0}", "test"); 

            Assert.AreNotEqual(firstString, secondString); //test passes
        } 

        [Test]
        public void StringEqualsTest()
        {
            string firstString = "test";
            string secondString = "test";
            Assert.AreNotEqual(firstString, secondString); //test fails
        }

AreNotEqual is comparing references and AreEqual is comparing values.

The reason AreNotEqual is correctly failing in StringEqualsTest is because
of the way the .NET framework manages the string intern pool. All string
literals come from the string intern pool.

In StringEqualsTest, firstString and secondString are the same reference,
so AreNotEqual correctly fails when comparing references.

In StringFormatEqualsTest, string.Format creates a new reference to each
string - even if the string is already in the string intern pool, so
AreNotEqual incorrectly passes because the two references are different,
even though they contain the same value.

The following change would make StringFormatEqualsTest work with MBUnit as
it forces firstString and secondString to use references from the intern
pool. MBUnit seems to be relying on this in the AreNotEqual test.

string firstString = string.Intern(string.Format("{0}", "test"));
string secondString = string.Intern(string.Format("{0}", "test"));

Assert.AreNotEqual(firstString, secondString); //test fails

I "reused" this property now on my installation so that 

* if false, and the failure count is not zero, then the MbUnit task still 
succeeds 

the modification to the InternalExecute() method is as follows: 

return (result.Counter.FailureCount == 0) | !HaltOnFailure ; 

In fact, this makes a lot of sense. Otherwise, by no means the number of 
failed tests or any other statistics can be gathered from the task. 
Oh, I also added output properties for the ReportCounter counters of the 
ReportResult. (vfr other issue reported MBUNIT-142) 

Steps to reproduce
1. Create more than one unit test assembly, with test sets that complete in
less than one minute.
2. Run MbUnit and pass several of these test assemblies to the command line
version of MbUnit.

What is the expected output? What do you see instead?
Default output with the report name format of mbunit-{0}{1} produces report
files with date and time filenames. We should have one report per assembly.

As ToShortTimeString() is used in the code, two assemblies producing
reports in the same minute have the same filename, so one report is
overwritten with the next set or results. Either
result.Date.ToLongTimeString() should be used (line 72 of ReportBase.cs) or
another parameter (eg. {2}) provided to ensure a unique filename can be
created.

What version of the product are you using? On what operating system?
MbUnit v2.4.1, on Windows XP Pro, SP2. NCover v1.5.8

Please provide any additional information below.
To create a coverage report with NCover running MbUnit from MsBuild, the
format is NCover.Console.exe UnitTest.exe UnitTestArgs TestAssemblies. 

I cannot set the filename to something unique, as the same filename is used
for all assemblies. mbunit-{0}{1} does not work for the reason stated
above. My only option is to make multiple calls to NCover, once for each
test assembly. This then leaves multiple unit test reports to import into
Cruise Control which then doesn't work with the MbUnit xsl. Also, there is
significant overhead in calling NCover repeatedly, rather than one call
with all the test assemblies. Both NCover and MbUnit have startup and
shutdown overheads which takes our unit test time from 12 to 16 minutes.

I tried to make the change myself with the code from svn but there was a
problem (a known issue according to the user forum) which meant the command
line exe would build but not run. 

Let's be the first xUnit framework to support .Net Framework 3.5.
This doesn't look like much work but it would be impressive to have for demos.

As with release of NUnit 2.4.1, they have added the following asserts. 
Should we add support for them too? 
Assert.Greater(long, long) 
Assert.Greater(ulong, ulong) 
Assert.GreaterOrEqual(long, long) 
Assert.GreaterOrEqual(ulong, ulong) 
Assert.Less(long, long) 
Assert.Less(ulong, ulong) 
Assert.LessOrEqual(long, long) 
Assert.LessOrEqual(ulong, ulong)

The DebugMonitor class has a private member called stepNames, which is
never instantiated. This causes a lot of exceptions to be thrown, but they
are catched by the RemoteEventListener class so you only see them when
debugging.

Create the Text\HTML reports and XSLT to do the transforms.

Good starting point is to look at the classes in the MbUnit.Core.Reporting
namespace.  I already built a TextReportFormatter and HtmlReportFormatter
based on XSLT resources in MbUnit.Core.Reporting.Resources.

It is required that the report is created purely in standards based XHTML 
so it can be styled in CSS.

Need to update existing scraps of documentation and put them someplace more
useful.

When test failures occur, Echo should return a different result code.

All of the MbUnit components should have strings extracted so that they can
be fully localized.

When we choose to do this, we can break this task down to cover work on
individual components.

The idea is to have a data source attribute that can obtain an external
value provided by the test runner.

[BindExternal("CorporateDomainName", Description="The domain name of the
corporate site to be tested.  Varies between development and production
environments.", Default="dev.mycompany.com")]
public string corporateDomainName;

What steps will reproduce the problem?
1.
2.
3.

What is the expected output? What do you see instead?


Please use labels and text to provide additional information.

Need to port the installer over from MbUnit v2.4 and adapt it for Gallio. 
It would be nice to have the NSIS binaries in SVN so that anyone can build
an installer right after a checkout.  Building the installer should be
included in the "Release" build target but not in the standard build.

Assert.AreEqual(Double, Double, Double, String, String, Object()) 
has a me attribute which is a little odd, and calls its self repeatedly 
until a stack overflow is raised. 

public static void AreEqual(double expected, double actual, double delta, 
string me, string format, params object[] args) 
{ 
    AreEqual(expected, actual, delta, me, format, args); 
} 

TODO:

1. Implement remaining task options including Filter specification.
2. Report failures correctly.
3. Write unit tests.
4. Write an integration test that actually launches a little NAnt project
with a sample test project to use as a sanity check.
5. Clean up the XML documentation of the task parameters so we can generate
high quality user docs from it.  Include example task usages.

Currently, when Reflector class used to test non-public members, it's 
impossible to test members that defined in base classes. 
For example we have to classes A & B. B is derived from A. A class defines 
integer count. 

public class A 
{ 
  private int count = 5; 
  ... 
} 

public class B : A 
{ 
  ... 
} 

Code below will generate an ReflectionException. 

B b = new B(); 
Reflector.GetField(_b, "count"); 

Todo:

1. Unit tests to prove that it works.  The RunNamespace method looks
suspect to me right now.

2. Fix the filters.  Don't use string-based parsing for this, it produces
anbiguous results.  For example, RunMember should be using an
AndFilter<ITest> containing a TypeFilter<ITest> and MemberFilter<ITest> to
ensure that exactly the right member is selected.  There's no sense
unpacking this information into a potentially ambiguous string only to
rederive a filter specification from it.

3. Add a link to a generated HTML report.

Julian pointed out that AutoRunner in MbUnit v2 was a facility for a
console application to run its own tests (hence "auto").  In Gallio it
performs some of the other functions of v2's AutoRunner but not the one
that really gave it its name.

So we should probably rename AutoRunner in Gallio and introduce a new
AutoRunner more like in MbUnit v2.

From Seth A. Roby on the mailing list Aug 28, 2007.

=====

I have a few DecoratorPatternAttribute subclasses defined, but they don't
work quite the way I thought they did.

If I've got a fixture with a setup, the test itself, and a teardown, the
flow looks like this:
    1. SetUp
    2. MyTest
    3. TearDown

I thought the DecoratorPatternAttribute would provide the Invoker, and that
Invoker would wrap the entire process. Note that I don't want it to wrap
the FIXTURE, just the single test's SetUp/Test/TearDown, as
so:
    1. MyInvoker.Execute begins, calls its Invoker.Execute which leads to 2-4
    2. SetUp
    3. MyTest
    4. TearDown
    5. The internal call to Invoker.Execute from Step 1 ends, and
MyInvoker.Execute completes its work.

But what really happens is that a new Invoker is created for each step.
    1. MyInvoker.Execute begins, calls its Invoker.Execute which leads to 2
ONLY
    2. SetUp
    3. The internal call to Invoker.Execute from Step 1 ends, and
MyInvoker.Execute completes its work.
    4. MyInvoker.Execute begins, calls its Invoker.Execute which leads to 5
ONLY
    5. MyTest
    6. The internal call to Invoker.Execute from Step 4 ends, and
MyInvoker.Execute completes its work.
    7. MyInvoker.Execute begins, calls its Invoker.Execute which leads to 8
ONLY
    8. TearDown
    9. The internal call to Invoker.Execute ends from Step 7, and
MyInvoker.Execute completes its work.

This makes a lot of use cases impossible:
    - I can't set up some external resource in my Attribute, modify it's
behavior in the SetUp, and then use it in the Test, because by the time the
Test rolls around the Attribute has fired again, and the SetUp steps are
overridden by the new Attribute.
    - I can't modify the resource in SetUp and TearDown the resulting
object, because in TearDown it's a new Attribute that doesn't know any of
the changes that occured in SetUp. So if I had a cache of files on disk, I
can't change that location in SetUp, or I'd leak those files.

If the Invoker was the same one every time I could keep track of it that
way, but it's a new instance each time, so no dice.

How would I go about making the "wrap the whole process" decorator I want?
Is it possible using some other class, or do I have to involve myself with
the IRun interface and it's ilk (which I haven't needed to deal with yet)?

Reported on 2.4 but moving up

It would be handy if the MbUnit task for MSBuild would have some extra 
output properties for e.g. total number of tests run, total number of 
successfullty run test, total number of failures, well, infact all 
properties of the ReportCounter class.

implementation of these extra Properties 
also : use of HaltOnFailure property (cfr MBUNIT-143) 

Julian suggested we also release MbUnit as a ZIP file.
I agree I've had cause to want this behavior before when merging MbUnit
into a development tree.

The zip file should probably leave out the TD.Net Addin and the Visual
Studio help docs as they aren't useful standalone.

When testing legacy code, it is sometimes impractical to refactor it to
make it easier to work with.  Sometimes code is so test hostile that one
must resort to mocking out non-virtual or non-public parts of the framework
to be able to make any headway at all.  Tools like TypeMock exist to solve
this problem.

In particular, Rhino.Mocks can't handle mocking non-virtual or non-public
methods because it uses DynamicProxy to generate dynamic subclasses. 
Moreover it can only provide assistance with situations in which the point
of instantiation of the mocked dependency is under the control of the test.

However, because MbUnit load test assemblies itself into an isolated
environment, it is in a good position to provide extra assistance for
runtime dyamic interception of any method call.  An MbUnit plugin could
conceivable extend the MbUnit loader to enable this behavior on demand. 
The same plugin could then expose its interception services as an
alternative to DynamicProxy that Rhino.Mocks or other tools could use.

Here are four possible approaches:

1. Use Mono.Cecil to deconstruct and load assemblies into the dynamic
assembly context of the isolated AppDomain with Reflection.Emit.  Then use
MethodRental to instrument the code on demand as needed.

2. Use Mono.Cecil to deconstruct and fully instrument the assemblies that
require interception.  Then add hooks into the instrumentation on demand.

3. Load the test assemblies into a separate CLR process and install a CLR
profiler hook to force recompilation and instrument the code prior to JIT
compilation on demand.  This option has the downside of being very complex
to implement, disable debugging, and preventing the use of other code
profiling tools like NCover.

4. Wait for Microsoft to provide a real AOP solution...

setting HaltOnFailure to any value has no effect. In fact, from the 
source, I can see its value is not used anywhere. 

What's the purpose of this ? Otherwise, it should removed or at least made 
obsolete (have it display a warning when set) ?

I "reused" this property now on my installation so that 

* if false, and the failure count is not zero, then the MbUnit task still 
succeeds 

the modification to the InternalExecute() method is as follows: 

return (result.Counter.FailureCount == 0) | !HaltOnFailure ; 

In fact, this makes a lot of sense. Otherwise, by no means the number of 
failed tests or any other statistics can be gathered from the task. 
Oh, I also added output properties for the ReportCounter counters of the 
ReportResult. (vfr other issue reported MBUNIT-142) 

Should be able to display the Echo report after test execution straight to
the console.

This is similar to another issue related to displaying plain text reports
in the console.  However in this case we want to launch the shell command
to display the report file itself.  For example, an HTML report should just
launch in the browser because of its .html extension.  A .txt file will
probably open up in notepad or something.

We have this option in MbUnit v2.  We just need to implement it for Echo.

Adopt a cleaner format for help text similar to MSBuild command arguments.
Improve test coverage.
Fix word-wrapping issues in help text.

These projects aren't really necessary for running these tests.  If we go
down this path of tailoring sample test assemblies too much we'll quickly
end up with too many of them to be manageable.

At the moment the TestResources projects are really just broken down by
test framework which is pretty much unavoidable.

To verify that the runner yields an appropriate result code, provide it
with a Filter that selects FailingTest or PassingTest by type from one of
the other TestResources projects.

Thanks!

Currently the installer forces an install for all users.  (That's better
than only half-supporting current user installs.)

To support installs for current user we need to add a custom page to the
NSIS installer to prompt all users / current user and SetShellVarContext.

I'm not sure the TD.Net add-in supports reading test framework
configuration values from HKCU in the registry.  So we might have to work
with Jamie to fix this since for current user installs we obviously can't
write to HKLM.

This is the conglomeration of a few different old feature requests.

1. Sort by name.
2. Sort by declaration order.
3. Group by bound variables.
4. Group by metadata such as importance.