josetr / vsixtesting Goto Github PK

View Code? Open in Web Editor NEW

29.0 9.0 10.0 222 KB

A powerful library to create integration tests for Visual Studio Extensions (VSIX).

Home Page: https://www.nuget.org/packages/VsixTesting.Xunit/

License: Other

C# 100.00%

vsixtesting's Introduction

VsixTesting

VsixTesting allows you to easily test your Visual Studio Extensions.

It also supports Visual Studio 2019 / 2022.

Getting Started

The fastest way to get started is cloning the VsixTestingSamples repo and playing with it.

git clone https://github.com/josetr/VsixTestingSamples

Create the test project

TestProject.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <TargetFramework>net461</TargetFramework>
  </PropertyGroup>

  <ItemGroup>
    <PackageReference Include="xunit" Version="2.3.1" />
    <PackageReference Include="xunit.runner.visualstudio" Version="2.3.1" />

    <PackageReference Include="VsixTesting.Xunit" Version="0.1.65" /> 

    <!-- Optional package containing shell types used in this sample (VS2017 ~ VS2022) -->    
    <PackageReference Include="Microsoft.VisualStudio.Shell.15.0" Version="15.0.26201" /> 

    <!-- Optional package containing shell types used in this sample (VS2012 ~ VS2019) -->    
    <!-- <PackageReference Include="VSSDK.Shell.11" Version="11.0.4" /> -->

    <!-- Optional project reference to your VSIX Project -->
    <!-- <ProjectReference Include="..\MyVsixProject\MyVsixProject.csproj" /> -->
    <!--
       VsixTesting contains an MSBuild target that scans all project references
       and if they generate a .vsix package, it will copy them 
       to the output folder where the test assembly is located, which
       will cause VsixTesting to install such packages because all packages
       located next to the test assembly are installed by default.
    -->        
  </ItemGroup>
</Project>

TestClass.cs

using Microsoft.VisualStudio.Shell;
using Microsoft.VisualStudio.Shell.Interop;
using Xunit;

namespace Tests
{  
    public class TestClass
    {
        [VsFact]
        void FactTest()
            =>  Assert.NotNull(Package.GetGlobalService(typeof(SVsWebBrowsingService)));

        [VsTheory]
        [InlineData(123)]
        void TheoryTest(int n)
        {
            Assert.NotNull(Package.GetGlobalService(typeof(SVsWebBrowsingService)));
            Assert.Equal(123, n);
        }
    }
}

Configuring how tests run

All the settings are located in ITestSettings and are implemented by 3 attributes:

[VsTestSettings] for classes/collections/assemblies
[VsFact] and [VsTheory] for methods

Intellisense should be used to read the documentation for each property that can be set.

License

This repository is licensed with the Apache, Version 2.0 license.

vsixtesting's People

Contributors

Stargazers

Watchers

Forkers

gitter-badger sharwell jcansdale komsa-ag lvshuiyiqing elishateng mcmatty yannduran emrekultursay

vsixtesting's Issues

[Question] How to programmatically dismiss a modal dialog?

Hi I am trying to add a test that triggers our custom item template wizard, which opens a WPF form to collect user input for template instantiation. How could I programmatically control inputs and click the "Finish" button to dismiss the modal dialog to resume the action? Currently my experimental instance just got stuck waiting for user input at the following line in my test

        [VsFact(Version = "[15.0-16.0)", UIThread = true)]
        async Task Test()
        {
            // open solution and set up stuff ...
            // stuck at following line waiting for user to interact with the WPF dialog opened by our custom wizard
            projectItems.AddFromTemplate(solution.GetProjectItemTemplate(templateName, "VC"),
                    itemName)
        }

Skip installation if the extension is already installed globally

Hi, first thank you for this awesome library. This makes it so much easier for us to set up integration test for our Visual Studio extension.

In our testing setup, we have already installed the plugin globally for other tests with admin elevation (since we need to install outside of the extension). In such cases tests are failing with the following error:

Error Message:
 System.Exception : System.Exception: Extension 'MyPlugin 1.0' version must be higher than the globally installed extension 'MyPlugin 1.0'.
   at VsixTesting.Installer.Installer.Install(String extensionPath, Nullable`1 allUsers)
   at VsixTesting.Installer.Installer.<>c__DisplayClass5_1.<Install>b__0(String path)
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
   at VsixTesting.Installer.Installer.Install(String applicationPath, String rootSuffix, IEnumerable`1 extensionPaths, Nullable`1 allUsers)
   at VsixTesting.Installer.Program.<>c__DisplayClass1_0.<Run>b__0()
   at VsixTesting.Installer.Program.Run(String applicationPath, String rootSuffix, String[] args)
Stack Trace:
   at async Task<(int Result, string Output)> Vs.VisualStudioUtil.RunInstallerAsync(VsHive hive, IEnumerable<string> args)
   at async Task<(int InstallCount, string Output)> Vs.VisualStudioUtil.InstallExtensionsAsync(VsHive hive, IEnumerable<string> extensions)
   at void VsixTesting.VsInstance+<>c__DisplayClass15_0+<<Prepare>b__0>d.MoveNext()
   at void VsixTesting.IDiagnosticsExtensions+<>c__DisplayClass1_0+<<RunAsync>b__0>d.MoveNext()
   at void VsixTesting.XunitX.Internal.DiagnosticTestRunner<T>+<>c__DisplayClass5_0+<<RunAsync>b__0>d.MoveNext()
   at async Task Xunit.Sdk.ExecutionTimer.AggregateAsync(Func<Task> asyncAction)
   at async Task<decimal> VsixTesting.XunitX.Internal.DiagnosticTestRunner<T>.RunAsync(IMessageBus messageBus, CancellationTokenSource cancellationTokenSource)
   at async Task VsixTesting.VsInstance.Prepare(VsHive hive, IEnumerable<string> extensionsToInstall, bool resetSettings, IDiagnostics diagnostics, bool installInvoker)
   at async Task VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.Init(CancellationTokenSource cts)
   at async Task<RunSummary> VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.RunAsync(CancellationTokenSource cts)

It would be great if we can just make the test to proceed in such case since the extension is already installed.

Can't find VsixTesting.dll when the path contains a space

Hi,

When trying to run a test I immediately get a FileNotFoundException about VsixTesting.dll.
I used dotPeek to check for the references and found that some DLLs are missing (marked in blue), which might hint to the source of the problem:

I tried running the tests both from VS22 and VS19, targeting different versions for the experimental instance.

ThreadHelper.JoinableTaskFactory is null when running test on Visual Studio 2017

If I run the following test:

using Xunit;
using Xunit.Abstractions;
using EnvDTE;
using Microsoft.VisualStudio.Shell;
using Task = System.Threading.Tasks.Task;

namespace VsixTestingTests
{
    public class VsTests
    {
        readonly ITestOutputHelper output;

        public VsTests(ITestOutputHelper output)
        {
            this.output = output;
        }

        [VsFact]
        public async Task VersionAsync()
        {
            await ThreadHelper.JoinableTaskFactory.SwitchToMainThreadAsync();
            var dte = (DTE)ServiceProvider.GlobalProvider.GetService(typeof(DTE));
            output.WriteLine(dte.Version);
        }
    }
}

It passes on Visual Studio 2015, but ThreadHelper.JoinableTaskFactory appears to be null on Visual Studio 2017:

Output from VsixTestingTests.VsTests.VersionAsync - 2015 Exp:
  14.0
Test 'VsixTestingTests.VsTests.VersionAsync - 2017 Exp' failed: System.NullReferenceException : Object reference not set to an instance of an object.
	at Microsoft.VisualStudio.Shell.ThreadHelper.get_JoinableTaskContext()
	at Microsoft.VisualStudio.Shell.ThreadHelper.get_JoinableTaskFactory()
	Class1.cs(21,0): at VsixTestingTests.VsTests.<VersionAsync>d__2.MoveNext()
	--- End of stack trace from previous location where exception was thrown ---
	at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
	--- End of stack trace from previous location where exception was thrown ---
	at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
	--- End of stack trace from previous location where exception was thrown ---
	at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
	at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)

1 passed, 1 failed, 0 skipped, took 34.48 seconds (xUnit.net 2.4.0 build 4049).

If I change to [VsFact(UIThread = true)], the test passes on both versions.

Any idea what's going on?

Maybe @AArnott or @jnm2 would have some ideas?

Execute tests using the same Visual Studio SKU as parent

I tend to have a bunch of Visual Studio 2017 instances installed. Depending on which version I updated last, VsixTesting might decide to execute tests inside Visual Studio 2017 Team Explorer, which isn't necessarily appropriate.

When starting a Visual Studio from an extensibility project, the version launched will be the same as the parent instance. It would be nice if VsixTesting could model this behavior.

This could be done by looking at the parent process and preferring an instance based on this, rather than the highest version number (or whatever the current scheme is).

Here is some code that will find the parent process, which seems to work well:

using System;
using System.ComponentModel;
using System.Diagnostics;
using System.Runtime.InteropServices;

/// <summary>
/// A utility class to determine find the parent process.
/// </summary>
/// <remarks>
/// Shamelessly copied from https://stackoverflow.com/a/3346055/121348 (posted by Simon Mourier).
/// </remarks>
[StructLayout(LayoutKind.Sequential)]
public struct ParentProcessUtilities
{
    // These members must match PROCESS_BASIC_INFORMATION
    internal IntPtr Reserved1;
    internal IntPtr PebBaseAddress;
    internal IntPtr Reserved20;
    internal IntPtr Reserved21;
    internal IntPtr UniqueProcessId;
    internal IntPtr InheritedFromUniqueProcessId;

    [DllImport("ntdll.dll")]
    private static extern int NtQueryInformationProcess(IntPtr processHandle, int processInformationClass, ref ParentProcessUtilities processInformation, int processInformationLength, out int returnLength);

    /// <summary>
    /// Gets the parent process of the current process.
    /// </summary>
    /// <returns>An instance of the Process class.</returns>
    public static Process GetParentProcess()
    {
        return GetParentProcess(Process.GetCurrentProcess().Handle);
    }

    /// <summary>
    /// Gets the parent process of specified process.
    /// </summary>
    /// <param name="id">The process id.</param>
    /// <returns>An instance of the Process class.</returns>
    public static Process GetParentProcess(int id)
    {
        Process process = Process.GetProcessById(id);
        return GetParentProcess(process.Handle);
    }

    /// <summary>
    /// Gets the parent process of a specified process.
    /// </summary>
    /// <param name="handle">The process handle.</param>
    /// <returns>An instance of the Process class.</returns>
    public static Process GetParentProcess(IntPtr handle)
    {
        ParentProcessUtilities pbi = default(ParentProcessUtilities);
        int returnLength;
        int status = NtQueryInformationProcess(handle, 0, ref pbi, Marshal.SizeOf(pbi), out returnLength);
        if (status != 0)
        {
            throw new Win32Exception(status);
        }

        try
        {
            return Process.GetProcessById(pbi.InheritedFromUniqueProcessId.ToInt32());
        }
        catch (ArgumentException)
        {
            // not found
            return null;
        }
    }
}

Add option to launch Visual Studio from VsixTesting.ExtensionInstaller

We've recently been struggling to keep the GitHub for Visual Studio extension working on both Visual Studio 2017 and 2015. The team now primarily uses 2017 and the extension is built with 2017. Installing, running and debugging on 2015 has become a pain.

I was wondering how feasible it would be to use various components of VsixTesting to create a painless F5 experience for installing and running extensions in different versions?

Add an option to launch Visual Studio from VsixTesting.ExtensionInstaller after installing
Use the debug propagation trick, so that if VsixTesting.ExtensionInstaller is started with the debugger attached, it would automatically attach the debugger from the parent devenv process to the child process

We could then create a launchSettings.json entry for the Visual Studio versions we want to target. It might be something like this:

    "Start VS 2015": {
      "commandName": "Project",
      "commandLineArgs": "/ApplicationPath \"C:\\Program Files (x86)\\Microsoft Visual Studio 14.0\\Common7\\IDE\\devenv.exe\" /RootSuffix Exp
            /InstallAndStart \"$(SolutionDir)build\\$(Configuration)\\GitHub.VisualStudio.vsix\""
    },

Do you think that would work?

Write a proper documentation of how to use this tool for VSIX unit tests

Cannot skip tests if no support version of Visual Studio is available

Steps to reproduce:

Write a test that only works on Visual Studio 2012
Run the test suite on a machine that does not have Visual Studio 2012 installed

Expected results:

The test which requires Visual Studio 2012 can be skipped. There are multiple options for skipping the test:

Automatic: The test is skipped by the framework when the prerequisite cannot be found
Manual: The test can be skipped by passing a command line argument to the test runner

Actual results:

There is no way to run the tests without at least one error being reported.

ExtensionManagerService changed, causing RuntimeBinderException

System.Exception : Microsoft.CSharp.RuntimeBinder.RuntimeBinderException: No overload for method 'UpdateLastExtensionsChange' takes '0' arguments
       at CallSite.Target(Closure , CallSite , Object )
       at System.Dynamic.UpdateDelegates.UpdateAndExecuteVoid1[T0](CallSite site, T0 arg0)
       at VsixTesting.Installer.ExtensionManagerService.UpdateLastExtensionsChange()
       at VsixTesting.Installer.Installer.Install(String extensionPath, Nullable`1 allUsers)
       at VsixTesting.Installer.Installer.<>c__DisplayClass5_1.<Install>b__0(String path)
       at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
       at VsixTesting.Installer.Installer.Install(String applicationPath, String rootSuffix, IEnumerable`1 extensionPaths, Nullable`1 allUsers)
       at VsixTesting.Installer.Program.<>c__DisplayClass1_0.<Run>b__0()
       at VsixTesting.Installer.Program.Run(String applicationPath, String rootSuffix, String[] args)
  Stack Trace: 
    VisualStudioUtil.RunInstallerAsync(VsHive hive, IEnumerable<string> args) line 49
    VisualStudioUtil.InstallExtensionsAsync(VsHive hive, IEnumerable<string> extensions) line 18
    VsInstance+<>c__DisplayClass15_0+<<Prepare>b__0>d.MoveNext() line 54
    IDiagnosticsExtensions+<>c__DisplayClass1_0+<<RunAsync>b__0>d.MoveNext() line 34
    DiagnosticTestRunner<T>+<>c__DisplayClass5_0+<<RunAsync>b__0>d.MoveNext() line 182
    ExecutionTimer.AggregateAsync(Func<Task> asyncAction) line 48
    DiagnosticTestRunner<T>.RunAsync(IMessageBus messageBus, CancellationTokenSource cancellationTokenSource) line 180
    VsInstance.Prepare(VsHive hive, IEnumerable<string> extensionsToInstall, bool resetSettings, IDiagnostics diagnostics, bool installInvoker) line 41
    VsInstanceTestAssemblyRunner.Init(CancellationTokenSource cts) line 110
    VsInstanceTestAssemblyRunner.RunAsync(CancellationTokenSource cts) line 79

Apparently the method ExtensionManagerService.UpdateLastExtensionsChange() in Microsoft.VisualStudio.ExtensionManager.Implementation.dll (and therein used ones) changed and now take a bool named updateClient.
This results in the exception above, when running VsFact-TestMethods.

Visual Studio Professional 2019 v16.7.2
Microsoft.VisualStudio.ExtensionManager.Implementation.dll reports 'Date Modified' as 2020-08-13

Error on VS 2022: Method not found: 'System.Object EnvDTE._DTE.GetObject(System.String)'

When I try to run my tests on VS 2022 (17.1 Preview 1), I get the following error.

[xUnit.net 00:00:00.00] xUnit.net VSTest Adapter v2.4.3+1b45f5407b (32-bit Desktop .NET 4.0.30319.42000)
[xUnit.net 00:00:00.42]   Starting:    VisualStudio.Data.Sqlite.IntegrationTests
[xUnit.net 00:00:08.64]     Microsoft.VisualStudio.Data.Sqlite.DataSourceTests.Is_registered - 2022 Exp [FAIL]
[xUnit.net 00:00:08.64]       System.MissingMethodException : Method not found: 'System.Object EnvDTE._DTE.GetObject(System.String)'.
[xUnit.net 00:00:08.64]       Stack Trace:
[xUnit.net 00:00:08.64]            at void VsixTesting.VsInstance+<>c__DisplayClass16_0+<<Launch>b__0>d.MoveNext()
[xUnit.net 00:00:08.64]            at void System.Runtime.CompilerServices.AsyncTaskMethodBuilder<TResult>.Start<TStateMachine>(ref TStateMachine stateMachine)
[xUnit.net 00:00:08.64]            at Task<VsInstance> VsixTesting.VsInstance.Launch(VsHive hive, TimeSpan timeout, IDiagnostics diagnostics)+(IOutput output) => { }
[xUnit.net 00:00:08.64]         D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/Diagnostics.cs(38,0): at Task<T> VsixTesting.XunitX.Internal.BasicDiagnostics.RunAsync<T>(string displayName, Func<IOutput, Task<T>> work)
[xUnit.net 00:00:08.64]         D:/a/VsixTesting/VsixTesting/src/VsixTesting/VsInstance.cs(80,0): at Task<VsInstance> VsixTesting.VsInstance.Launch(VsHive hive, TimeSpan timeout, IDiagnostics diagnostics)
[xUnit.net 00:00:08.65]         D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/VsInstanceTestAssemblyRunner.cs(111,0): at async Task VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.Init(CancellationTokenSource cts)
[xUnit.net 00:00:08.65]         D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/VsInstanceTestAssemblyRunner.cs(79,0): at async Task<RunSummary> VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.RunAsync(CancellationTokenSource cts)
[xUnit.net 00:00:08.65]       Output:
[xUnit.net 00:00:08.65]         1. Preparing Instance
[xUnit.net 00:00:08.65]         Extension VisualStudio.Data.Sqlite 1.0 is up to date.
[xUnit.net 00:00:08.65]         Extension VsixTesting.Invoker 1.0.0 is up to date.
[xUnit.net 00:00:08.65]         Extension VsixTesting.Invoker 1.0.0 is up to date.
[xUnit.net 00:00:08.65]         
[xUnit.net 00:00:08.65]         2. Launching Instance
[xUnit.net 00:00:08.65]   Finished:    VisualStudio.Data.Sqlite.IntegrationTests

The sample tests work just fine which leads me to believe it has something to do with including this dependency:

<PackageReference Include="Microsoft.VisualStudio.Data.Framework" Version="17.0.31902.203" />

I get the same error with both the NuGet.org version 0.1.66 of VsixTesting.Xunit and MyGet version 0.1.71-g32e57811c9.

A million thanks for this awesome library! 🙏Let me know if you need anything else to help investigate.

Nerdbank.Streams error on Visual Studio 2022 after update to 17.7.4

VsixTesting stopped working somewhere between VS versions 17.5 - 17.7. Currently, the latest Visual Studio version is 17.7.4, and the VsixTesting.Instances.VisualStudio.VisualStudio[2022Exp] test (using the sample with version upgraded to 0.1.75) fails with the following error:

Message: 
System.Exception : System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.IO.FileNotFoundException: Could not load file or assembly 'Nerdbank.Streams, Version=2.9.0.0, Culture=neutral, PublicKeyToken=cac503e1823ce71c' or one of its dependencies. The system cannot find the file specified.
   at Microsoft.VisualStudio.ExtensionEngine.Impl.Settings.MetadataStore.ExtensionMetadataStore.ReadExtensionData(IEngineHost engineHost)
   at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ExtensionSettingsCache.LoadExtensionMetadata()
   at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ExtensionSettingsCache..ctor(ISettingsManager settingsManager, IEngineHost engineHost, IExtensionEngine engine)
   at Microsoft.VisualStudio.ExtensionManager.Impl.Settings.InstalledExtensionSettingsList..ctor(IEngineHost engineHost, IExtensionEngine engine, IExtensionSettingsCache extensionSettingsCache)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionEngineImpl..ctor(IEngineHost engineHost, EngineMode mode, IEnumerable`1 enabledExtensions, CancellationToken disposalToken, IExtensionSettingsCache extensionSettingsCache)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService..ctor(ISettingsManager settingsManager, CultureInfo preferredCulture, ExtensionManagerServiceMode mode, IEnumerable`1 enabledExtensions, EngineHostId engineHostId, CancellationToken disposalToken, Boolean delayServiceHub, ILogger logger, IExtensionSettingsCache extensionSettingsCache)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService..ctor(SettingsManager settings, ExtensionManagerServiceMode mode, ILogger logger)
   --- End of inner exception stack trace ---
   at System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
   at System.Reflection.RuntimeConstructorInfo.Invoke(BindingFlags invokeAttr, Binder binder, Object[] parameters, CultureInfo culture)
   at VsixTesting.Installer.ExtensionManagerService.Create(ExternalSettingsManager externalSettingsManager)
   at VsixTesting.Installer.Installer.Install(String applicationPath, String rootSuffix, IEnumerable`1 extensionPaths, Nullable`1 allUsers)
   at VsixTesting.Installer.Program.Run(String applicationPath, String rootSuffix, Version vsProductVersion, String[] args)

  Stack Trace: 
<RunInstallerAsync>d__17.MoveNext() line 51
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
<InstallExtensionsAsync>d__15.MoveNext() line 18
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
<<Prepare>b__0>d.MoveNext() line 55
--- End of stack trace from previous location where exception was thrown ---
<16 more frames...>
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
<Prepare>d__15.MoveNext() line 41
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
<<LaunchAndDebug>b__0>d.MoveNext() line 73
--- End of stack trace from previous location where exception was thrown ---
ExceptionDispatchInfo.Throw()
TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
<<RunOnStaThreadAsync>b__0>d.MoveNext() line 43

  (result has additional output)

The library Nerdbank.Streams is available at C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\PublicAssemblies\Nerdbank.Streams.2.x\Nerdbank.Streams.dll, but it's not picked up. From the error, it looks like it's trying to load version 2.9.0, but that file in Visual Studio version has been updated from 2.8.0 to 2.10.0, which might be the reason for the issue. Maybe it's an assembly redirect problem (not sure on VsixTesting or Visual Studio)?

Improved ability to copy VSIX to output directory

It's my understanding that copying the VSIX to the integration test output directory is required for correct deployment in test scenarios. Originally I did this:

  <ItemGroup>
    <ProjectReference Include="..\Tvl.VisualStudio.MouseFastScroll\Tvl.VisualStudio.MouseFastScroll.csproj" />
  </ItemGroup>

  <ItemGroup>
    <None Include="..\Tvl.VisualStudio.MouseFastScroll\bin\$(Configuration)\net45\Tvl.VisualStudio.MouseFastScroll.vsix" CopyToOutputDirectory="PreserveNewest" />
  </ItemGroup>

I believe the experience would be improved if I could write this instead:

  <ItemGroup>
    <ProjectReference Include="..\Tvl.VisualStudio.MouseFastScroll\Tvl.VisualStudio.MouseFastScroll.csproj" CopyVsix="true" />
  </ItemGroup>

We could provide this functionality if the following was added to a .targets file in the NuGet package for VsixTesting.Xunit:

  <Target Name="PrepareVsixProjectReferences"
          BeforeTargets="ResolveProjectReferences"
          DependsOnTargets="PrepareProjectReferences">
    <MSBuild
        Projects="@(_MSBuildProjectReferenceExistent)"
        Targets="CreateVsixContainer;VSIXContainerProjectOutputGroup"
        BuildInParallel="$(BuildInParallel)"
        Properties="%(_MSBuildProjectReferenceExistent.SetConfiguration); %(_MSBuildProjectReferenceExistent.SetPlatform); %(_MSBuildProjectReferenceExistent.SetTargetFramework); CreateVsixContainer=true"
        Condition="'%(_MSBuildProjectReferenceExistent.CopyVsix)' == 'true'"
        ContinueOnError="!$(BuildingProject)"
        RemoveProperties="%(_MSBuildProjectReferenceExistent.GlobalPropertiesToRemove)">

      <Output TaskParameter="TargetOutputs" ItemName="_ProjectReferenceVsixOutputs" />
    </MSBuild>

    <ItemGroup>
      <None Include="@(_ProjectReferenceVsixOutputs)" Condition="Exists('%(Identity)')" CopyToOutputDirectory="PreserveNewest" />
    </ItemGroup>
  </Target>

Run tests on specific .sln

Hello,

Can you provide a way to set what SLN is opened by the devenv.exe? Visual Studio is not initializing an asyncpckage until a solution is opened, thus I cannot run tests on my custom commands.

I saw that you use the following VsixTesting.Invoker/Properties/launchSettings.json, maybe you can add an attribute which can set the path to the solution to open on launch.

Command line arguments example: D:/TestProjects/CustomAllocator.sln /RootSuffix Exp

Consider running on the UI thread by default

Currently the library starts tests on background threads by default. The behavior can be changed for an assembly by the following:

[assembly: VsTestSettings(UIThread = true)]

I would propose running tests on the UI thread by default instead, using one of the following:

Change the default for UIThread to true, or
Remove the UIThread property, and force the value was true

Overall, I believe this change better represents the requirements of code users are planning to execute as integration tests.

Many VS APIs require are affinitized to the main thread, so they can be used directly in tests
Integration tests typically cannot run concurrently due to state collisions within the IDE environment, so affinitizing them to the UI thread is not constraining
While xunit in general is used for unit testing which does not require a main thread context, integration tests typically run in a more restricted environment. I would argue that it's acceptable to use a default which differs from other xunit frameworks because the host environment itself is "special".

This change would impact users in the following ways:

The IAsyncLifetime interface can be used to implement asynchronous initialization and cleanup code, since the UI thread will be blocked for the execution of constructors and Dispose
Integration test methods should be asynchronous if they need to avoid blocking the UI thread
Integration tests which need to run on the background thread can use await TaskScheduler.Default to switch to a background thread

Make VsixTesting.Installer wait for input if there's an exception while installing

I'm attempting to install for Visual Studio 2017 - Community Preview, but VsixTesting.Installer is failing with an exception. The exception flashes for a fraction for a second before vanishing. There doesn't seem to be an obvious way to prevent this.

Could you either add a /wait option or make it automatically wait if an exception is thrown? 🙏

Can't build it

Hi, I clone this project and try to build it in Visual Studio 2019. But I found there is nuget package Pdb2Pdb can't find. Could you please update the project ?

Newtonsoft.Json dependency fails after VS 2022 17.2 update

The VsixTesting.Installer.x64.exe tries to load Newtonsoft.Json version 9.0, but with Visual Studio 2022, the minimum version of Newtonsoft.Json is 13.0 (https://devblogs.microsoft.com/visualstudio/using-newtonsoft-json-in-a-visual-studio-extension/)

This incorrect dependency started causing test failures with a recent Visual Studio 2022 17.2 update. With that update, the Newtonsoft.Json.dll (9.0.0) file located at Common\Ide\PrivateAssemblies\ was deleted, and this dependency is no longer (accidentally) satisfied.

  Error Message:
   System.Exception : System.IO.FileNotFoundException: Could not load file or assembly 'Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed' or one of its dependencies. The system cannot find the file specified.
File name: 'Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed'
   at Microsoft.VisualStudio.Setup.Serialization.PackageSerializer..ctor(IServiceProvider services)
   at Microsoft.VisualStudio.Setup.Installer.VsixInstaller.GetVsixType(Stream vsixStream)
   at Microsoft.VisualStudio.ExtensionManager.InstallableExtensionImpl.ReadVSIXManifestFromPackage(Stream stream, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.InstallableExtensionImpl..ctor(String path, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionEngineImpl.CreateInstallableExtension(String extensionPath, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.Microsoft.VisualStudio.ExtensionManager.IVsExtensionManager.CreateInstallableExtension(String extensionPath)
   at VsixTesting.Installer.Installer.Install(String extensionPath, Nullable`1 allUsers)
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
   at VsixTesting.Installer.Installer.Install(String applicationPath, String rootSuffix, IEnumerable`1 extensionPaths, Nullable`1 allUsers)
   at VsixTesting.Installer.Program.Run(String applicationPath, String rootSuffix, Version vsProductVersion, String[] args)

Assembly manager loaded from:  C:\Windows\Microsoft.NET\Framework64\v4.0.30319\clr.dll
Running under executable  c:\src\temp\5ou1evhk.avg\VsixTesting.Installer.x64.exe
--- A detailed error log follows.

=== Pre-bind state information ===
LOG: DisplayName = Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
 (Fully-specified)
LOG: Appbase = file:///c:/src/temp/5ou1evhk.avg
LOG: Initial PrivatePath = NULL
Calling assembly : Microsoft.VisualStudio.Setup.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: No application configuration file found.
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config.
LOG: Post-policy reference: Newtonsoft.Json, Version=9.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
LOG: Attempting download of new URL file:///c:/src/temp/5ou1evhk.avg/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///c:/src/temp/5ou1evhk.avg/Newtonsoft.Json/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///c:/src/temp/5ou1evhk.avg/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///c:/src/temp/5ou1evhk.avg/Newtonsoft.Json/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json/Newtonsoft.Json.EXE.

  Stack Trace:
     at async Task<(int Result, string Output)> Vs.VisualStudioUtil.RunInstallerAsync(VsHive hive, IEnumerable<string> args) in D:/a/VsixTesting/VsixTesting/src/VsixTesting/Vs/VisualStudioUtil.Installer.cs:line 51
   at async Task<(int InstallCount, string Output)> Vs.VisualStudioUtil.InstallExtensionsAsync(VsHive hive, IEnumerable<string> extensions) in D:/a/VsixTesting/VsixTesting/src/VsixTesting/Vs/VisualStudioUtil.Installer.cs:line 18
   at void VsixTesting.VsInstance+<>c__DisplayClass15_0+<<Prepare>b__0>d.MoveNext() in D:/a/VsixTesting/VsixTesting/src/VsixTesting/VsInstance.cs:line 53
   at void VsixTesting.IDiagnosticsExtensions+<>c__DisplayClass1_0+<<RunAsync>b__0>d.MoveNext() in D:/a/VsixTesting/VsixTesting/src/VsixTesting/IDiagnostics.cs:line 34
   at void VsixTesting.XunitX.Internal.DiagnosticTestRunner<T>+<>c__DisplayClass5_0+<<RunAsync>b__0>d.MoveNext() in D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/Diagnostics.cs:line 182
   at async Task Xunit.Sdk.ExecutionTimer.AggregateAsync(Func<Task> asyncAction) in C:/Dev/xunit/xunit/src/xunit.execution/Sdk/Frameworks/ExecutionTimer.cs:line 48
   at async Task<decimal> VsixTesting.XunitX.Internal.DiagnosticTestRunner<T>.RunAsync(IMessageBus messageBus, CancellationTokenSource cancellationTokenSource) in D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/Diagnostics.cs:line 185
   at async Task VsixTesting.VsInstance.Prepare(VsHive hive, IEnumerable<string> extensionsToInstall, bool resetSettings, IDiagnostics diagnostics, bool installInvoker) in D:/a/VsixTesting/VsixTesting/src/VsixTesting/VsInstance.cs:line 41
   at async Task VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.Init(CancellationTokenSource cts) in D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/VsInstanceTestAssemblyRunner.cs:line 111
   at async Task<RunSummary> VsixTesting.XunitX.Internal.VsInstanceTestAssemblyRunner.RunAsync(CancellationTokenSource cts) in D:/a/VsixTesting/VsixTesting/src/VsixTesting.Xunit/Internal/VsInstanceTestAssemblyRunner.cs:line 80

Consider avoiding traits with spaces

Currently the default automatically-assigned traits have spaces in both the key and the value.

VsixTesting/src/VsixTesting.Xunit/Internal/VsTestCaseBase.cs

Line 61 in 3cc0e35

Traits["Visual Studio"] = new List<string>(new[] { InstanceId });

The use of this form results in cases where command line arguments involving the traits are not obvious.

RootSuffix = "" fails with --RootSuffix is required.

If I mark a test with [VsTestSettings(RootSuffix = "")], it fails with the following exception:

Test 'Tests.TestClass.TheoryTest(n: 123) - 2015 ' failed: System.Exception : System.ArgumentException: Argument --RootSuffix is required.
   at VsixTesting.ExtensionInstaller.CommandLineParser.One(String[] args, String name)
   at VsixTesting.ExtensionInstaller.Program.Main(String[] args)

The same test passes with, [VsTestSettings(RootSuffix = "\"\"")].

VsixTesting doesn't work with NCrunch

It appears VsixTesting doesn't currently work with NCrunch. 😢

It seems the issue is the TestFramework attribute:

[assembly: TestFramework("Xunit.VsTestFramework", "VsixTesting.Xunit")]

Even regular [Fact] tests aren't detected when a test project includes this attribute (see ASimpleFact above).

@remcomulder have you come across this issue before? Is there any way to get this working? 🙏

Newtonsoft.Json dependency fails on VS 2022 17.4

I am currently on VS 2022 17.4.2 and running the tests fails.
I.e. cloning https://github.com/josetr/VsixTestingSamples, opening one of the two sample solutions, building everything and attempting to run a VS 2022 test fails.
Test explorer says:

Message: 
System.Exception : System.IO.FileNotFoundException: Could not load file or assembly "Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed" or one of its dependencies. The system cannot find the file specified.
File name: "Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed"
   at Microsoft.VisualStudio.Setup.Serialization.PackageSerializer..ctor(IServiceProvider services)
   at Microsoft.VisualStudio.Setup.Installer.VsixInstaller.GetVsixType(Stream vsixStream)
   at Microsoft.VisualStudio.ExtensionManager.InstallableExtensionImpl.ReadVSIXManifestFromPackage(Stream stream, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.InstallableExtensionImpl..ctor(String path, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionEngineImpl.CreateInstallableExtension(String extensionPath, CultureInfo preferredCulture)
   at Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.Microsoft.VisualStudio.ExtensionManager.IVsExtensionManager.CreateInstallableExtension(String extensionPath)
   at VsixTesting.Installer.Installer.Install(String extensionPath, Nullable`1 allUsers)
   at System.Linq.Enumerable.Count[TSource](IEnumerable`1 source, Func`2 predicate)
   at VsixTesting.Installer.Installer.Install(String applicationPath, String rootSuffix, IEnumerable`1 extensionPaths, Nullable`1 allUsers)
   at VsixTesting.Installer.Program.Run(String applicationPath, String rootSuffix, Version vsProductVersion, String[] args)

=== Pre-bind state information ===
LOG: DisplayName = Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
 (Fully-specified)
LOG: Appbase = file:///C:/Users/<User>/AppData/Local/Temp/ig3g2ple.kis
LOG: Initial PrivatePath = NULL
Calling assembly : Microsoft.VisualStudio.Setup.Common, Version=1.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a.
===
LOG: This bind starts in LoadFrom load context.
WRN: Native image will not be probed in LoadFrom context. Native image will only be probed in default load context, like with Assembly.Load().
LOG: No application configuration file found.
LOG: Using host configuration file:
LOG: Using machine configuration file from C:\Windows\Microsoft.NET\Framework64\v4.0.30319\config\machine.config wird verwendet.
LOG: Post-policy reference: Newtonsoft.Json, Version=13.0.0.0, Culture=neutral, PublicKeyToken=30ad4fe6b2a6aeed
LOG: Attempting download of new URL file:///C:/Users/<User>/AppData/Local/Temp/ig3g2ple.kis/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Users/<User>/AppData/Local/Temp/ig3g2ple.kis/Newtonsoft.Json/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Users/<User>/AppData/Local/Temp/ig3g2ple.kis/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///C:/Users/<User>/AppData/Local/Temp/ig3g2ple.kis/Newtonsoft.Json/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json/Newtonsoft.Json.DLL.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json.EXE.
LOG: Attempting download of new URL file:///C:/Program Files/Microsoft Visual Studio/2022/Community/Common7/IDE/PrivateAssemblies/Newtonsoft.Json/Newtonsoft.Json.EXE.


  Stack Trace: 
VisualStudioUtil.RunInstallerAsync(VsHive hive, IEnumerable<string> args) line 51
VisualStudioUtil.InstallExtensionsAsync(VsHive hive, IEnumerable<string> extensions) line 18
VsInstance+<>c__DisplayClass15_0+<<Prepare>b__0>d.MoveNext() line 55
IDiagnosticsExtensions+<>c__DisplayClass1_0+<<RunAsync>b__0>d.MoveNext() line 34
VsInstance.Prepare(VsHive hive, IEnumerable<string> extensionsToInstall, bool resetSettings, IDiagnostics diagnostics, bool installInvoker) line 41
VsInstanceTestAssemblyRunner.Init(CancellationTokenSource cts) line 111
VsInstanceTestAssemblyRunner.RunAsync(CancellationTokenSource cts) line 80

  Standard Output: 
1. Preparing Instance

(Some parts were originally in German; I translated them to English.)

I can see that I have a C:\Program Files\Microsoft Visual Studio\2022\Community\Common7\IDE\PrivateAssemblies\Newtonsoft.Json.13.0.1.0\, but no 13.0.0.0.

I guess this is more or less the same issue as with the 17.2 update that was reported in issue #35?

Version conflict with System.Collections.Immutable

Currently the VsixTesting.Xunit package depends indirectly on System.Collections.Immutable version 1.4.0. The first version of this package published with Visual Studio is 1.1.36. This mismatch causes warnings during the build if an extension needs to reference System.Collections.Immutable.

📝 The dependency in the test library comes via the reference to Ben.Demystifier. @benaadams in there a version of this package which depends only on System.Collections.Immutable 1.1.36?

Execution of test hangs forever (VS 2022 17.4)

Attempting to run even a simple test with Visual Studio 2022 17.4.4 never finishes. For example, getting https://github.com/josetr/VsixTestingSamples, updating the VsixTesting.Xunit version in the VsClassLibrary.Tests.csproj file to the latest 0.1.75, building everything in VsClassLibrary.sln and attempting to execute the test TestMethod17 causes the test to start running but to never finish. Looking in the Task Manager, VsixTesting.Installer.x64.exe gets created but just waits with 0% CPU load.

When debugging the VsixTesting.Installer.x64.exe, a System.MissingMethodException occurs, stating that the method "!0 Microsoft.VisualStudio.Threading.AsyncLazy`1.GetValue()" was not found. The exception then gets caught later and a message is attempted to be written to the console, but this for some reason hangs forever.

Is it broken only for me, or is someone else also observing the same issue?

From what I could gather:

In the main thread, the function VsixTesting.Installer.x64.exe!VsixTesting.Installer.Installer.Install(string extensionPath, bool? allUsers) calls extensionManager.CreateInstallableExtension(), see source code. Call stack:

ntdll.dll!NtWaitForSingleObject�()	Unknown
KernelBase.dll!WaitForSingleObjectEx�()	Unknown
clrjit.dll!Compiler::lvaInitTypeRef() Line 251	C++
clrjit.dll!Compiler::compCompileHelper(CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd, CORINFO_METHOD_INFO * methodInfo, void * * methodCodePtr, unsigned long * methodCodeSize, JitFlags * compileFlags, CorInfoInstantiationVerification) Line 6161	C++
clrjit.dll!Compiler::compCompile(CORINFO_METHOD_STRUCT_ * methodHnd, CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd, CORINFO_METHOD_INFO * methodInfo, void * * methodCodePtr, unsigned long * methodCodeSize, JitFlags * compileFlags) Line 5644	C++
clrjit.dll!jitNativeCode(CORINFO_METHOD_STRUCT_ * methodHnd, CORINFO_MODULE_STRUCT_ * classPtr, ICorJitInfo * compHnd, CORINFO_METHOD_INFO * methodInfo, void * * methodCodePtr, unsigned long * methodCodeSize, JitFlags * compileFlags, void * inlineInfoPtr) Line 6953	C++
clrjit.dll!CILJit::compileMethod(ICorJitInfo * compHnd, CORINFO_METHOD_INFO * methodInfo, unsigned int flags, unsigned char * * entryAddress, unsigned long * nativeSizeOfCode) Line 312	C++
[Managed to Native Transition]	
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.InstallableExtensionImpl.InstallableExtensionImpl(string path, System.Globalization.CultureInfo preferredCulture) Line 166	C#
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ExtensionEngineImpl.CreateInstallableExtension(string extensionPath, System.Globalization.CultureInfo preferredCulture) Line 789	C#
Microsoft.VisualStudio.ExtensionManager.Implementation.dll!Microsoft.VisualStudio.ExtensionManager.ExtensionManagerService.Microsoft.VisualStudio.ExtensionManager.IVsExtensionManager.CreateInstallableExtension(string extensionPath)	Unknown
VsixTesting.Installer.x64.exe!VsixTesting.Installer.Installer.Install(string extensionPath, bool? allUsers) Line 60	C#
System.Core.dll!System.Linq.Enumerable.Count<string>(System.Collections.Generic.IEnumerable<string> source, System.Func<string, bool> predicate)	Unknown
VsixTesting.Installer.x64.exe!VsixTesting.Installer.Installer.Install(string applicationPath, string rootSuffix, System.Collections.Generic.IEnumerable<string> extensionPaths, bool? allUsers) Line 42	C#
VsixTesting.Installer.x64.exe!VsixTesting.Installer.Program.Run(string applicationPath, string rootSuffix, System.Version vsProductVersion, string[] args) Line 112	C#
[AppDomain (VsixTesting.Installer.x64.exe�, #1) -> AppDomain (Installer 17.4.33213.308�, #2)]	
VsixTesting.Installer.x64.exe!VsixTesting.Installer.Program.Main(string[] args)	Unknown

I am bit confused about the clrjit.dll stuff at the top of the stack.

There are 4 other threads running, all of them having the same callstack:

Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ScanModule.ShouldRecurseDirectory(ref Microsoft.IO.Enumeration.FileSystemEntry entry) Line 215	C#
Microsoft.IO.Redist.dll!Microsoft.IO.Enumeration.FileSystemEnumerator<string>.MoveNext() Line 246	C#
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ScanModule.FindExtensionManifests(Microsoft.IO.DirectoryInfo rootDirectory, int depthOverride) Line 186	C#
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ScanModule.ScanExtensions(Microsoft.VisualStudio.ExtensionManager.ExtensionLocations location, Microsoft.IO.DirectoryInfo directory, bool isSdkLocation, int depthOverride) Line 295	C#
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ScanModule.ScanPathForLocation(Microsoft.VisualStudio.ExtensionManager.ExtensionLocations location, string path) Line 315	C#
Microsoft.VisualStudio.ExtensionEngine.dll!Microsoft.VisualStudio.ExtensionManager.ScanModule.ScanInstalledExtensionsInLocation(Microsoft.VisualStudio.ExtensionManager.ExtensionLocations location, bool safeMode) Line 287	C#
mscorlib.dll!System.Threading.Tasks.Task<Microsoft.VisualStudio.ExtensionManager.Impl.Settings.ScanResult>.InnerInvoke()	Unknown
mscorlib.dll!System.Threading.Tasks.Task.Execute()	Unknown

In one of theses threads, the System.MissingMethodException occurs.

Decompilation of Microsoft.VisualStudio.ExtensionManager.ScanModule.ShouldRecurseDirectory() shows, that the code is calling ExtensionManifestExcludedDirectories.Contains(item). Here, ExtensionManifestExcludedDirectories is a member of Microsoft.VisualStudio.ExtensionManager.ScanModule and defined as

private readonly AsyncLazy<HashSet<string>> ExtensionManifestExcludedDirectoriesLazy;
private HashSet<string> ExtensionManifestExcludedDirectories => ExtensionManifestExcludedDirectoriesLazy.GetValue();

I.e. it calls AsyncLazy.GetValue(). However, that method does not seem to exist. Decompilation of Microsoft.VisualStudio.Threading.AsyncLazy does not show it. I guess this is causing the exception.

Searching around, one can find several reports about missing AsyncLazy.GetValue() methods. But they all have different call stacks. The problem apparently was always some corrupt installation, picking assemblies that were no longer up-to-date. Specifically, I checked whether Microsoft.VisualStudio.ExtensionEngine.dll is up-to-date (also in the GAC), but this seems to be the case. I suspect that Microsoft in some update removed the AsyncLazy.GetValue() method, but did not adapt the Microsoft.VisualStudio.ExtensionManager.ScanModule. If that is true, the IVsExtensionManager.CreateInstallableExtension method might be broken?