Hi,

More of a question than an Issue, is there anyway to ask the xUnit Runner to run its
tests on the Main UI Thread?

Thanks for any help.

Best Regards
Matt

The UI thread plays an important role in device apps. Certain tests might need to execute on it.

The STAExample can likely be adapted:
https://github.com/xunit/samples.xunit/tree/master/STAExamples

Filtering is more useful if it allows the execution of a subset of tests. Currently to execute a subset of tests one must:

1. Filter
2. Click on each test

The Run All button does what it says, regardless of any filtering.

To address this, I would suggest a Run Filtered button above the Run All button which executes only those tests that are visible.

Hello,

I'm trying to add a Xamarin.Forms test project.

I created a Xamarin.Forms PCL and when trying to install the xunit.runner.devices NuGet package, I get the following error:

Could not install package 'System.Runtime.InteropServices.RuntimeInformation 4.0.0'. You are trying to install this package into a project that targets '.NETPortable,Version=v4.6,Profile=Profile44', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

What am I doing wrong?

Prior to XF 2.3.4, the UWP test runner supporting calling Xamarin.Forms.Forms.Init(e) to enable XF for tests which require it.

However, starting with XF 2.3.4 the following exception occurs when you attempt to initialize Xamarin.Forms in the UWP test runner:
HRESULT E_FAIL has been returned from a call to a COM component

Changing the initialization entry point has no effect.

When running a test under Android when a filter has been entered, the status for the test is never updated. It just kind of hangs indefinitely. Running the same test without a filter (or on iOS with or without a filter) works as expected.

A repro solution can be downloaded from here.

Here's an example showing the repro with a filter entered and the test consequently stuck:

The font is so large, with no word wrap in the UI, that if I have two or more tests that all start with the same word or two, I can't discern between them. No UI I can find will display the full test name.

For example, here is a screenshot that demonstrates no test name is shown. When the test happens to fail I can usually figure out the test name from the stack trace. But when a test passed, or before I run it, I can't tell.

Hi.

I have a solution with an UWP application where I need to add some unit so I have decided to tryout xunit and run it on devices like described here: https://xunit.github.io/docs/getting-started-devices-uwp.html.

In respect to running the xunit test on a device everything works fine, but at the moment I add a reference to the xunit.runner.devices nuget package Visual Studio are not able to find/detect test methods anymore.

Output windows shows:

[13-11-2017 10:31:23 Informational] ------ Discover test started ------
[13-11-2017 10:31:23 Warning] [xUnit.net 00:00:00.0838783] Skipping: UwpTestRunner (Could not find/load any of the following assemblies: xunit.execution.desktop.dll)
[13-11-2017 10:31:23 Informational] ========== Discover test finished: 0 found (0:00:00,1760319) ==========
[13-11-2017 10:33:21 Informational] ------ Discover test started ------
[13-11-2017 10:33:22 Warning] [xUnit.net 00:00:00.0835425] Skipping: UwpTestRunner (Could not find/load any of the following assemblies: xunit.execution.desktop.dll)
[13-11-2017 10:33:22 Informational] ========== Discover test finished: 0 found (0:00:00,1620295) ==========
[13-11-2017 10:34:41 Informational] ------ Discover test started ------
[13-11-2017 10:34:41 Warning] [xUnit.net 00:00:00.0745213] Skipping: UwpTestRunner (Could not find/load any of the following assemblies: xunit.execution.desktop.dll)
[13-11-2017 10:34:41 Informational] ========== Discover test finished: 0 found (0:00:00,1550351) ==========

I can see that other people have seen similar problems also but they seems to be related to older earlier versions of xunit/xunit devices which should have been fixed by now.

Any suggestions to what is wrong in my case:

[Details]
UWP target (min = max = Anniversary Update)
Xunit (2.3.1)
Xunit devices (2.3.0)

It would be cool if there was a "running" status so that one could quickly filter to those tests that are stuck in their execution (quite a common occurrence when writing async tests).

I've created very simple skeleton of solution with PCL unit test library & iOS unit test library as suggested in your How To. I uncommented line for PCL in AppDelegate and added few tests in PCL library - as result nothing was discovered by runner.

I think that once runner get test assembly loaded and test assembly has no tests - all further test assemblies are ignored

http://developer.xamarin.com/guides/cross-platform/xamarin-forms/

Now I can talk about it :). We should do it with that, so that we can have a single mobile runner across all three mobile platforms!

I think I'm encountering what is a bug, but I'm not really sure where it's happening. I'm trying to setup a file writer for test results on Android 4.4 (Nexus 4)

I have the following as my main activity:

using System;
using System.Collections.Generic;
using System.IO;
using System.Linq;
using System.Reflection;

using Android.App;
using Android.OS;

using Xunit.Runners.UI;
using Xunit.Sdk;

using AndroidEnvironment = Android.OS.Environment;
using Environment = System.Environment;

namespace DeviceGatewaySink.Tests.Android
{
    [Activity(Label = "xUnit Android Runner", MainLauncher = true, Theme = "@android:style/Theme.Material.Light")]
    public class MainActivity : RunnerActivity
    {
        protected override void OnCreate(Bundle bundle)
        {
            this.AddTestAssembly(Assembly.GetExecutingAssembly());
            this.AddExecutionAssembly(typeof(ExtensibilityPointFactory).Assembly);
            base.OnCreate(bundle);

            var path =
                AndroidEnvironment.MediaMounted == AndroidEnvironment.ExternalStorageState ?
                    AndroidEnvironment.GetExternalStoragePublicDirectory(AndroidEnvironment.DirectoryDocuments).AbsolutePath
                    : Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments);
            path = Path.Combine(path, string.Format("tests-{0}", DateTime.UtcNow.ToString("O").Replace(":", "-") + ".log"));

            Console.WriteLine("Writing log to: {0}", path);
            this.Writer = File.CreateText(path);
        }
    }
}

However, when I do a test run, it creates a file, but the file is 0 bytes -- no content at all inside the file. I tried overriding onpause and closing the file stream but that didn't do anything different.

Any idea what might be going on here? Any other way to get a file with results to get written out?

Since upgrading to xunit 2.0.0-beta5-build2785 and unit.runner.xamarin 0.99.5-beta5 the flags to AutoStart and TerminateAfterExecution no longer work for iOS or Android, in that the test execution doesn't automatically start, and the app doesn't close automatically.

Manual execution still works.

When running tests under the Android runner, the test counts do not reset back to zero. Thus, running a single test multiple times results in output like this:

Tests run: 1 Passed: 1 Failed: 0 Skipped: 0
Tests run: 2 Passed: 2 Failed: 0 Skipped: 0
Tests run: 3 Passed: 3 Failed: 0 Skipped: 0

We have some test cases that do something like the following:

    public class ExampleTestClass
    {
        public static IEnumerable<object[]> MemberDataObject
        {
            get
            {
                return new[] {
                    new object[] { new TestObjectAlpha() },
                    new object[] { new TestObjectBeta() }
                };
            }
        }

        [Theory, MemberData("MemberDataObject")]
        public void ExampleTestMethodUsingObjectParameters(TestObject pTestObject)
        {
                //Test code here
        }

Where TestObjectAlpha and TestObjectBeta inherit from TestObject.

What ends up happening, is that the test runner GUI only has ONE case listed to run -- ExampleTestMethodUsingObjectParameters(). Additionally, the test result output from the writer provides no parameters and simply lists this test twice, since there were two cases. That means that if one of the tests failed, I'm not provided any information as to which one failed.

If you use literals or strings, the GUI shows each case individually, and the parameters are provided both in the GUI and from the results writer:

    public class ExampleTestClass
    {
        public static IEnumerable<object[]> MemberDataObject
        {
            get
            {
                return new[] {
                    new object[] { "TestCaseAlpha" },
                    new object[] { "TestCaseBeta" }
                };
            }
        }

        [Theory, MemberData("MemberDataObject")]
        public void ExampleTestMethodUsingStringParameters(string pTestCase)
        {
                //Test code here
        }

This results in individual cases in the GUI:

Assembly.Name.ExampleTestClass.ExampleTestMethodUsingStringParameters(pTestCase: "TestCaseAlpha")
Assembly.Name.ExampleTestClass.ExampleTestMethodUsingStringParameters(pTestCase: "TestCaseBeta")

And the results writer also has this information. Since the runner already has to do some kind of reflection, it seems to me that it would make more sense just to output the object name as a string and make it look like so:

Assembly.Name.ExampleTestClass.ExampleTestMethodUsingObjectParameters(pTestObject: TestObjectAlpha())
Assembly.Name.ExampleTestClass.ExampleTestMethodUsingObjectParameters(pTestObject: TestObjectBeta())

Are there some other considerations for why this is not done, like speed? Currently, to get this desired operation, I am forced to do the following instead:

    public class ExampleTestClass
    {
        private static Dictionary<string, TestObject> _testCases = new Dictionary<string, TestObject>
        {
            { "TestObjectAlpha", new TestObjectAlpha() },
            { "TestObjectBeta", new TestObjectBeta() }
        };

        public static IEnumerable<object[]> MemberDataObject
        {
            get
            {
                return new[] {
                    new object[] { "TestObjectAlpha" },
                    new object[] { "TestObjectBeta" }
                };
            }
        }

        [Theory, MemberData("MemberDataObject")]
        public void ExampleTestMethod(string pTestCase)
        {
                //Test code here references _testCases[pTestCase] instead
        }

which produces the following:

Assembly.Name.ExampleTestClass.ExampleTestMethod(pTestCase: "TestObjectAlpha")
Assembly.Name.ExampleTestClass.ExampleTestMethod(pTestCase: "TestObjectBeta")

Silverlight based Windows Phone unit tests cannot be easily automated with the vstest.console.exe runner because the target project in which they are hosted is a standard Windows Phone application contrary to the built-in MSTest Windows Phone (Silverlight) Unit Test project. Is this scenario supported?

Need to read an app.config file and load it into the new xUnitDiscover/Exec settings.

Hi,

Using the filtering box with my project (>700 tests) fails often (but not always) with:

System.ArgumentOutOfRangeException: Argument is out of range.
Parameter name: index
  at Xamarin.Forms.ListProxy.get_Item (Int32 index) [0x00000] in <filename unknown>:0
  at Xamarin.Forms.TemplatedItemsList`2[Xamarin.Forms.ItemsView`1[Xamarin.Forms.Cell],Xamarin.Forms.Cell].get_Item (Int32 index) [0x00000] in <filename unknown>:0
  at Xamarin.Forms.Platform.iOS.ListViewRenderer+ListViewDataSource.GetCellForPath (Foundation.NSIndexPath indexPath) [0x00000] in <filename unknown>:0
  at Xamarin.Forms.Platform.iOS.ListViewRenderer+ListViewDataSource.GetCell (UIKit.UITableView tableView, Foundation.NSIndexPath indexPath) [0x00000] in <filename unknown>:0
  at at (wrapper managed-to-native) ObjCRuntime.Messaging:void_objc_msgSend (intptr,intptr)
  at UIKit.UITableView.EndUpdates () [0x00010] in /Developer/MonoTouch/Source/monotouch/src/build/native/UIKit/UITableView.g.cs:279
  at Xamarin.Forms.Platform.iOS.ListViewRenderer.UpdateItems (System.Collections.Specialized.NotifyCollectionChangedEventArgs e, Int32 section, Boolean resetWhenGrouped) [0x00000] in <filename unknown>:0
  at Xamarin.Forms.Platform.iOS.ListViewRenderer+<>c__DisplayClass5.<OnCollectionChanged>b__4 () [0x00000] in <filename unknown>:0
  at Foundation.NSAsyncActionDispatcher.Apply () [0x00000] in /Developer/MonoTouch/Source/maccore/src/Foundation/NSAction.cs:164
  at at (wrapper managed-to-native) UIKit.UIApplication:UIApplicationMain (int,string[],intptr,intptr)
  at UIKit.UIApplication.Main (System.String[] args, IntPtr principal, IntPtr delegate) [0x00005] in /Developer/MonoTouch/Source/monotouch/src/UIKit/UIApplication.cs:62
  at UIKit.UIApplication.Main (System.String[] args, System.String principalClassName, System.String delegateClassName) [0x0001c] in /Developer/MonoTouch/Source/monotouch/src/UIKit/UIApplication.cs:45

Quite possibly a Xamarin Forms issue, but nonetheless it's blocking me from using the feature. It is also painfully slow, especially when typing the first few characters to filter on.

The Android test runner is failing with this exception when I try to upgrade XForms to something greater than 2.1.x. I have scoured everything to make sure I don't have XForms assemblies of another version. Is no one else getting this?

I'm running this on a Raspberry Pi 3, with Windows 10 IoT Core 10.0.14393.576 "Anniversary update".

My unit test app builds, deploys and runs as expected. I can click "Run Everything" and all the tests run (some fail, this is expected).

However, if I click on the summary bar where it tells me how many tests there are, the runner app simply quits instead of showing the test details. It doesn't matter whether I do this before or after running the tests, the app always just quits.

xunit.runner.devices 2.1.0

Often one just wants to run tests in a certain fixture. The current filter mechanism only allows filtering based on the test name. I think it would be super useful to allow filtering on the fixture name too.

Get rid of the packages.config's

I think this is pretty straightforward and clear, but basically: I would like to get xUnit-style XML Results, because I can run a transform on them to change them to NUnit-style and have our build servers read the XML as part of our CI processes.

As it stands, without just importing this projects code and modifying it myself, I'm forced to write a parser for the text-file output of the current ResultListener. It doesn't look like I can use my own ResultListener, either, which further forces this requirement. I tried importing some of the source code to make a modified DeviceRunner, but ITestRunner is a private interface.

Overall, it seems like this would be a reasonable desire from other people trying to do Continuous Integration as well. I doubt I'm the only one who'd prefer this to be facilitated a little better, but I could easily be wrong.

Tags: Android, native, C++, NDK, Visual Studio, xunit, GDB, power fail/restart, crash

Microsoft Visual Studio Community 2017
Version 15.0.26228.4 D15RTWSVC
Microsoft .NET Framework
Version 4.7.02046

Installed Version: Community

Visual Basic 2017   00369-60000-00001-AA033
Microsoft Visual Basic 2017

Visual C# 2017   00369-60000-00001-AA033
Microsoft Visual C# 2017

Visual C++ 2017   00369-60000-00001-AA033
Microsoft Visual C++ 2017

Common Azure Tools   1.9
Provides common services for use by Azure Mobile Services and Microsoft Azure Tools.

EclipseAndroidImporter   1.0
Import Android projects from Eclipse.

GitHub.VisualStudio   2.2.0.8
A Visual Studio Extension that brings the GitHub Flow into Visual Studio.

JavaScript Language Service   2.0
JavaScript Language Service

KofePackagePackage Extension   1.0
KofePackagePackage Visual Studio Extension Detailed Info

Markdown Editor   1.11.201
A full featured Markdown editor with live preview and syntax highlighting. Supports GitHub flavored Markdown.

Microsoft MI-Based Debugger   1.0
Provides support for connecting Visual Studio to MI compatible debuggers

Microsoft Visual Studio VC Package   1.0
Microsoft Visual Studio VC Package

Mono Debugging for Visual Studio   Mono.Debugging.VisualStudio
Support for debugging Mono processes with Visual Studio.

NuGet Package Manager   4.0.0
NuGet Package Manager in Visual Studio. For more information about NuGet, visit http://docs.nuget.org/.

Visual C++ for Cross Platform Mobile Development (Android)   15.0.26228.00
Visual C++ for Cross Platform Mobile Development (Android)

Visual Studio Spell Check Everywhere   VSSpellCheckEverywhere
An extension that enables spell checking within any Visual Studio file editor or tool window that uses WPF text boxes.
https://GitHub.com/EWSoftware/VSSpellChecker

Visual Studio Spell Checker   VSSpellChecker
An editor extension that checks the spelling of comments, strings, and plain text as you type or interactively with tool windows.
https://GitHub.com/EWSoftware/VSSpellChecker

Xamarin   4.3.0.784 (73f58d6)
Visual Studio extension to enable development for Xamarin.iOS and Xamarin.Android.

Xamarin.Android SDK   7.1.0.41 (9578cdc)
Xamarin.Android Reference Assemblies and MSBuild support.

Xamarin.iOS and Xamarin.Mac SDK   10.4.0.123 (35d1ccd)
Xamarin.iOS and Xamarin.Mac Reference Assemblies and MSBuild support.`

Here are the steps:

• Create an Android tests project
• Go to Project properties > Android Options >Debugging Options: set [Debugger] to [C++]

With this code

 [Activity(Label = "xUnit Android Runner", MainLauncher = true, Theme= "@android:style/Theme.Material.Light")]
    public class MainActivity : RunnerActivity
    {
        protected override void OnCreate(Bundle bundle)
        {
	        AutoStart = false;

			var gea = Assembly.GetExecutingAssembly();
			
			AddTestAssembly(gea);

			// you cannot add more assemblies once calling base
			base.OnCreate(bundle);
        }
    }

I get the following error at startup right after launching the test project:

The target device is x86 Google's emulator.

With AddTestAssembly(..) commented out:

var gea = Assembly.GetExecutingAssembly();
			
//AddTestAssembly(gea);

the project launches. Therefore there is a bug somewhere in AddTestAssembly(...), associated XamarinForms code, and/or the GDB used by VS for Android debugging.

When running the app an error occurs stating that it cannot find xunit.runner.utility.dll to load, and inspecting the app/apk output you can see it is indeed not bundled (implicit linked assemblies are optimised out)

To get around this you can manually include the xunit.runner.utility package from nuget in the project.

Better solution to ensure that mono bundles this assembly in the app would be to add the xunit.runner.utility nugget package as a dependency to xunit.runner.xamarin and not a file reference:

NOTE: this happens even if you include the following line in the appdelegate or mainactivity classes:
AddExecutionAssembly(typeof(ExtensibilityPointFactory).Assembly);

Hey there, @onovotny,

I was wondering if using this nuget package to test Xamarin Native projects (not forms). I was curious if anyone has done this yet. Or if everyone is pretty much using Xamarin Forms with xUnit. I've had some struggles with getting the build to behave consistently. I think a big part of my issue is that I'm referencing the original native projects to my test projects. Is this the correct way of going about it?

I have several follow up questions if I'm on the right track. But if I'm going about his wrong, I'd like to know before going further down the wrong path.

Thank you for your time,
Kyle

I understand that it is easier to maintain runner based on Xamarin.Forms, however it would be useful to have a runner that is not dependent on that library - only Xamarin.Android or Xamarin.iOS.

I have 2 Xamarin.Android app (without Xamarin.Forms) with android support library in newest version (24.2.1). Xamarin.Forms depends on older version of that library (23.3.0) and because of that test project cannot be compiled.
Sometimes you can work around that issue by not using support library in project referenced by test project, but I guess this won't be always possible. So it would be nice to have a runner without Xamarin.Forms, so we can use any version of android support library.

My tests log to ITestOutputHelper but the text from these logs doesn't appear to be accessible in the mobile test runner either for failed or successful tests.

I just created a test [Theory] with [ClassData] as input. When one of the test case fails, xunit.devices still shows the whole test as successful. I tried it on WP8 test runner only. If it is the last Theory that fails, the whole test is failed. Can you reproduce this error?

    [Theory]
    [ClassData(typeof(PropertyInputData))]
    public void ShouldReadWriteNativeTypes(ITestValue testValue)
    {
        // Arrange
        var inputValue = testValue.Value;

        // Act
        var outputValue = ReadWriteValueOfType(inputValue);

        // Assert // HERE you can manipulate the Assertion to reproduce the error
        Assert.Equal(inputValue, outputValue);
    }

    public class PropertyInputData : IEnumerable<object[]>
    {
        private readonly List<object[]> data = new List<object[]>
        {
            new object[] { new TestValue<bool>(true) },
            new object[] { new TestValue<short>(short.MaxValue) },
            new object[] { new TestValue<ushort>(ushort.MaxValue) },
            new object[] { new TestValue<int>(int.MaxValue) },
            new object[] { new TestValue<uint>(uint.MaxValue) },
            new object[] { new TestValue<float>(float.MaxValue) },
        };

        public IEnumerator<object[]> GetEnumerator()
        {
            return this.data.GetEnumerator();
        }

        IEnumerator IEnumerable.GetEnumerator()
        {
            return this.GetEnumerator();
        }
    }

The Duration property is assigned to the TestResultViewModel instance after the TestCaseViewModel creates its success message, so it always reads as zero.

@onovotny I am looking for a solution to enable xunit unit test automation in CI platforms such as the Xamarin Mobile Center. I seems that currently I can only run unit tests via MSBuild or post-build schell scripts (and the tests are not carried out in the devices)... After checking the project a bit, I think currently it is not possible to automate this type of testing? As Mobile Center supports Xamarin.UITest, I wonder if it is possible to create UITest for the xunit Android/iOS test applications and integrate the results directly to the output window.

For instance, OOTB, we can create a single test case, simulate the Tap event on "Run Everything" and then retrieve the result of each test and create a summary in the output. A bit more advance way is to create dynamically multiple test cases using something like:

[Test, TestCaseSource("GetTestResults")]
public void ExportTestResults(TestResult testResult)
{
// pass or not depends on the results displayed by the xUnit Device UI
}

static TestResult [] GetTestResults()
{
    return GetAllTestResultsFound();
}

I haven't really tested this but it seems that this can be implemented as an external plugin to support CI testing for xunit.device.

Hi,

Thanks for this project! I found that I couldn't install the latest RC with WP8.1 support into my solution when running VS2013. I get Install-Package : 'xunit.core' already has a dependency defined for 'xunit.extensibility.core'.

In VS2015 I had no trouble, I wonder if it is related the to change in NuGet dependency version resolution?

Description

Warning Path/xxx.csproj: Warning: Target '_CopyOutOfDateSourceItemsToOutputDirectory', not found in the project (xxx).

Steps to reproduce

Create single page iOS application inside Xamarin.Forms solution. Add package xUnit.net for Devices (stable 2.2.0). Compile project and observe warning.

Expected behavior

No warnings

Actual behavior

Single warning is shown.

Environment

Xamarin Studio for mac.

Installation of the Nuget packages gives an error and rolls back. What am I doing wrong?

I tried also the '2.0.0-build0007' version, same error.

Could not install package 'xunit.runner.utility 2.1.0'. You are trying to install this package into a project that targets 'MonoAndroid,Version=v5.0', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

Full log:

PM> Install-Package xunit.runner.devices
Attempting to resolve dependency 'xunit.core (≥ 2.1.0)'.
Attempting to resolve dependency 'xunit.extensibility.core (= 2.1.0)'.
Attempting to resolve dependency 'xunit.abstractions (= 2.0.0)'.
Attempting to resolve dependency 'xunit.extensibility.execution (= 2.1.0)'.
Attempting to resolve dependency 'xunit.runner.utility (≥ 2.1.0)'.
Attempting to resolve dependency 'Xamarin.Forms (≥ 1.5.1.6471)'.
Attempting to resolve dependency 'Xamarin.Android.Support.v4 (≥ 23.0.1.1)'.
Attempting to resolve dependency 'Xamarin.Android.Support.Design (≥ 23.0.1.1)'.
Attempting to resolve dependency 'Xamarin.Android.Support.v7.AppCompat (≥ 23.0.1.1)'.
Attempting to resolve dependency 'Xamarin.Android.Support.v7.CardView (≥ 23.0.1.1)'.
Attempting to resolve dependency 'Xamarin.Android.Support.v7.MediaRouter (≥ 23.0.1.1)'.
Installing 'xunit.runner.utility 2.1.0'.
Successfully installed 'xunit.runner.utility 2.1.0'.
Installing 'Xamarin.Android.Support.v4 23.0.1.1'.
Successfully installed 'Xamarin.Android.Support.v4 23.0.1.1'.
Installing 'Xamarin.Android.Support.v7.AppCompat 23.0.1.1'.
Successfully installed 'Xamarin.Android.Support.v7.AppCompat 23.0.1.1'.
Installing 'Xamarin.Android.Support.Design 23.0.1.1'.
Successfully installed 'Xamarin.Android.Support.Design 23.0.1.1'.
Installing 'Xamarin.Android.Support.v7.CardView 23.0.1.1'.
Successfully installed 'Xamarin.Android.Support.v7.CardView 23.0.1.1'.
Installing 'Xamarin.Android.Support.v7.MediaRouter 23.0.1.1'.
Successfully installed 'Xamarin.Android.Support.v7.MediaRouter 23.0.1.1'.
Installing 'Xamarin.Forms 1.5.1.6471'.
Successfully installed 'Xamarin.Forms 1.5.1.6471'.
Installing 'xunit.runner.devices 1.3.2'.
Successfully installed 'xunit.runner.devices 1.3.2'.
Adding 'xunit.runner.utility 2.1.0' to NLog.UnitTests.Xamarin.Android.
Uninstalling 'xunit.runner.utility 2.1.0'.
Successfully uninstalled 'xunit.runner.utility 2.1.0'.
Install failed. Rolling back...
Install-Package : Could not install package 'xunit.runner.utility 2.1.0'. You are trying to install this package into a project that targets 'MonoAndroid,Version=v5.0', but the package does not contain any assembly references or content files that are compatible with that framew
ork. For more information, contact the package author.
At line:1 char:1
+ Install-Package xunit.runner.devices
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [Install-Package], InvalidOperationException
    + FullyQualifiedErrorId : NuGetCmdletUnhandledException,NuGet.PowerShell.Commands.InstallPackageCommand

PM> 

I'm using xunit pretty extensively, this project is particularly great for running tests on the devices. However, after yesterday's major update, my breakpoints within unit tests are no longer causing the debugger to stop in visual studio. However, breakpoints do hit, but application execution is suspended -- it's just visual studio doesn't pick up on the breakpoint and I can't resume or step over, or anything like that.

Looking for ideas on how to troubleshoot this. Let me know if this belongs more in the xamarin territory. I figured I'd try here, because breakpoints still work within my application -- just not the test runner.

Visual Studio 2013

Having upgraded an older project with lots of tests to the latest device runner, I'm getting an exception on this line of DeviceRunner.cs. The debugging experience in VS+Xamarin is still terrible, but I managed to get this poorly-formatted stack trace:

"  at System.Threading.Tasks.TaskCompletionSource`1[System.Object].SetResult (System.Object result) [0x0000c] in /Users/builder/data/lanes/2320/1f068b49/source/maccore/_build/Library/Frameworks/Xamarin.iOS.framework/Versions/git/src/mono/external/referencesource/mscorlib/system/threading/Tasks/TaskCompletionSource.cs:322 \n  at Xunit.Runners.DeviceRunner+<>c__DisplayClass17_0.<RunTests>b__0 () [0x000f9] in C:\\BuildAgent\\work\\a3e04f08310ca743\\src\\xunit.runner.viewmodels\\DeviceRunner.cs:254 \n  at Xunit.Runners.ThreadPoolHelper+<>c__DisplayClass0_0.<RunAsync>b__0 (System.Object _) [0x00000] in C:\\BuildAgent\\work\\a3e04f08310ca743\\src\\xunit.runner.ios-unified\\ThreadPoolHelper.cs:14 "

I tried at least narrowing down the test that was causing the problem by writing a BeforeAfterTestAttribute that spat out the test name before and after execution. However, this then causes all tests to pass! So it must be some kind of race condition.

Before I go digging too deep, any ideas what might cause this?

The layout is such that test names are not discernible on small devices. The text just gets cut off somewhere within the namespace name, or class name if you're lucky. Also, rotating the device in an attempt to find out test names seems to reset everything.

Perhaps trim the start of the text?

I followed the instructions here

http://blog.novotny.org/2014/07/10/getting-started-with-xunit-for-xamarin/

I copied the the content of AppDelegate.cs.txt to AppDelegate.cs, but when i run it, the Simulator window stays black. When i look to Application Output window, there is a logged error:

Could not register the assembly 'xunit.runner.ios': System.Reflection.ReflectionTypeLoadException: The classes in the module cannot be loaded.
at (wrapper managed-to-native) System.Reflection.Assembly:GetTypes (System.Reflection.Assembly,bool)
at System.Reflection.Assembly.GetTypes () [0x00000] in /Developer/MonoTouch/Source/mono/mcs/class/corlib/System.Reflection/Assembly.cs:351
at MonoTouch.Registrar.DynamicRegistrar.CollectTypes (System.Reflection.Assembly assembly) [0x00000] in /Developer/MonoTouch/Source/maccore/src/ObjCRuntime/DynamicRegistrar.cs:228
at MonoTouch.Registrar.Registrar.RegisterAssembly (System.Reflection.Assembly assembly) [0x00049] in /Developer/MonoTouch/Source/maccore/src/ObjCRuntime/Registrar.cs:1307

Here is all my version information

xunit version is 2.0.0-beta4-build-2738
xunit.runner.xamarin version is 0.99.3-beta4-build-1017

=== Xamarin Studio ===

Version 5.5 (build 227)
Installation UUID: 671bdca8-d671-4579-9ae2-006dce0e536d
Runtime:
Mono 3.10.0 ((detached/47db868)
GTK+ 2.24.23 (Raleigh theme)

Package version: 310000019

=== Apple Developer Tools ===

Xcode 6.0.1 (6528)
Build 6A317

=== Xamarin.iOS ===

Version: 8.2.0.193 (Indie Edition)
Hash: 959c1e4
Branch:
Build date: 2014-10-03 00:25:37-0400

=== Build Information ===

Release ID: 505000227
Git revision: 7b721eeec7a2fa4c4f4de0ecd2aed4dc25edac95
Build date: 2014-10-02 15:53:38-04
Xamarin addins: 99ed56b428b31eba1efaace4d82188d6f334e6ca

=== Operating System ===

Mac OS X 10.9.5
Darwin lfwdev08.local 13.4.0 Darwin Kernel Version 13.4.0
Sun Aug 17 19:50:11 PDT 2014
root:xnu-2422.115.4~1/RELEASE_X86_64 x86_64

Hi,

It seems that xunit.runner.devices 2.x cannot be installed on netstandard projects? I have tried different profiles (I have added the imports) and based on the supported packages listed on nuget it should work.

Thanks

Just downloading and opening the xUnit.devices.sln solution file on the latest VS Mac yields the following error on xunit.runner.devices:

Load Failed: Project does not support framework '_,Version=10.0'

Playing around with the target frameworks in the csproj just changes the version number (if monoandroid80 comes first, it's version 8.0, and if xamarinios10 comes first it's version 1.0).

To avoid the XY problem, the reason I'm trying to compile is so that I can step through to see why my Xamarin Android runner discovers no tests (see #64), while the iOS one from the same testing sources does (as does .net core). So if I get an answer to that, I won't need to compile anymore but this is still an interesting piece of info I think.

When running tests under iOS, sometimes the test count does not reflect the actual number of tests executed. For example, when running a single test multiple times, one can see output like this:

Tests run: 1 Passed: 1 Failed: 0 Skipped: 0
Tests run: 1 Passed: 1 Failed: 0 Skipped: 0
Tests run: 0 Passed: 0 Failed: 0 Skipped: 0
Tests run: 1 Passed: 1 Failed: 0 Skipped: 0

VS2017 v15.4.5
xunit.runner.devices v2.3.2
xunit v2.3.1

Followed https://xunit.github.io/docs/getting-started-devices-uwp.html and works successfully on x86 and ARM.
Upon adding a reference to my UWP application, I get an error:

3>error PRI175 : 0x80073b0f - Processing Resources failed with error : Duplicate Entry.
3>GENERATEPROJECTPRIFILE : error PRI277: 0xdef00532 - Conflicting values for resource ''

Do I have to move all logic I want tested to a class library?

NuGet 3.3 added support for content files. We need to add support for this here.

Need to switch to the unified API in order to support 64-bit.

After upgrading to 1.2.3-build0010 from the CI, I eventually discovered my Android test project stopped building with errors such as those mentioned in this xamarin forum post:

11>\Trunk\MobileUI\Android\Resources\Resource.designer.cs(329,137,329,165): error CS0117: 'Resource.Color' does not contain a definition for 'abc_color_highlight_material'
11>\Trunk\MobileUI\Android\Resources\Resource.designer.cs(529,146,529,177): error CS0117: 'Resource.Drawable' does not contain a definition for 'abc_control_background_material'
11>\Trunk\MobileUI\Android\Resources\Resource.designer.cs(846,133,846,157): error CS0117: 'Resource.Style' does not contain a definition for 'Base_V23_Theme_AppCompat'
11>\Trunk\MobileUI\Android\Resources\Resource.designer.cs(847,139,847,169): error CS0117: 'Resource.Style' does not contain a definition for 'Base_V23_Theme_AppCompat_Light'

The advice in the post is to upgrade to SDK 23. While I can certainly do that, and am working to do so now, it requires I install the SDK 23 and takes a while. Did this devices runner package need to depend on a newer Xamarin.Forms, or would it be OK to refer to the one previously used?

Basic functionality works -- tests are discoverable and runnable. There is a lot of room for improvements in each of the runner apps

When I updated xam forms and the android support libs to the latest versions (23.2), nuget will go through and update, but then the dependency on this package will then initiate a downgrade of devices.xunit and xamarin forms.

If I update devices.xunit back to 2.0, then it will downgrade other libs.

It gets stuck in a weird cycle. Not sure if it's a nuget bug, but one way to fix it would be to release a nuget update bumping the version numbers in the nuspec, which you have done recently.

Consider changing the UX so that when I tap a test name it does not re-execute the test, but rather switches to a backstage view that shows the full test name (no matter how long), shows any relevant test output and failure text, and offers a re-execute button.

For tests that have not yet executed when I tap on them, executing them (before or instead of showing backstage) feels right.

This problem seems specific to Android. iOS projects are using 2.3.3 without issue for the same set of tests.

Not sure what is going on or how to get more information but using the latest stable version of xunit (2.3.1) and xunit.runner.devices 2.2.0 when the unit test app starts I see an activity indicator and then a list of test assemblies and tests inside.

I couldn't attempt the xunit 2.3.1 / runner 2.3.0 combination because it seems that they import different versions of xunit.execution.dotnet which multiply defines types.

Changing nothing except upgrading to xunit.runner.devices 2.3.2 or 2.3.3 causes the UI to show no test assemblies. Are there any changes needed to the way that AddExecutionAssembly or AddTestAssembly are used?