Templates in test names do not expand variables when skipped about pester HOT 8 OPEN

@fflaten while it may be by design, from a user perspective I would expect names to always be expanded no matter what state the test ends up in.

from pester.

I am sure there are good reasons for the current implementation, but I did notice that when skipping tests using Set-ItResult like this

$sb = {
    Describe 'Test names with templates using ForEach' {
        It 'Displays array values when Set-ItResult -Skipped <_>' -ForEach @('foo', 'bar', 'test') {
            Set-ItResult -Skipped
            $_ | Should -Not -BeNullOrEmpty
        }

        It 'Displays hashtable values when Set-ItResult -Skipped <Name> <Value>' -ForEach @(
            @{ Name = 'foo'; Value = 'test 1' }
            @{ Name = 'bar'; Value = 'test 2' }
            @{ Name = 'foobar'; Value = 'test 3' }
        ) {
            Set-ItResult -Skipped
            $Name | Should -Not -BeNullOrEmpty
            $Value | Should -Not -BeNullOrEmpty
        }
    }
}

$c = New-PesterConfiguration
$c.Run.Container = New-PesterContainer -ScriptBlock $sb
$c.Output.Verbosity = 'Detailed'
Invoke-Pester -Configuration $c

template names are expanded successfully

Pester v5.5.0

Starting discovery in 1 files.
Discovery found 6 tests in 80ms.
Running tests.
Describing Test names with templates using ForEach
  [!] Displays array values when Set-ItResult -Skipped foo is skipped 45ms (41ms|4ms)
  [!] Displays array values when Set-ItResult -Skipped bar is skipped 4ms (2ms|2ms)
  [!] Displays array values when Set-ItResult -Skipped test is skipped 5ms (2ms|3ms)
  [!] Displays hashtable values when Set-ItResult -Skipped foo test 1 is skipped 45ms (43ms|2ms)       
  [!] Displays hashtable values when Set-ItResult -Skipped bar test 2 is skipped 4ms (2ms|2ms)
  [!] Displays hashtable values when Set-ItResult -Skipped foobar test 3 is skipped 14ms (12ms|2ms)    
Tests completed in 288ms
Tests Passed: 0, Failed: 0, Skipped: 6 NotRun: 0

I guess that is because the skipping is performed from inside the execution context. Is there a performance concern here, or could the same not be done when a test is skipped by using the It -Skip parameter?

from pester.

We cannot know what will happen inside of the test execution. Setting the test result using the set-itresult command is a special type of test failure. It doesn’t really have to happen inside of the test body. It can happen in any method that’s been called, and even by any custom exception if it has the right shape. For that reason, we cannot do static analysis, which would anyway be more expensive than just expanding the test name and runing the test.

The second aspect of this is that we have to run the test anyway, because we have to run the code that preceeds the set-it result command.

from pester.

As csandfeld alluded to, using Set-ItResult is a workaround. It's not as clean, but works.

Instead of -Skip on the It:

Describe 'Test names with templates using ForEach' {
  BeforeDiscovery {
    $skipCondition = ...
  }
  It 'Should display hashtable values, even when skipping <Name> <Value>' -Skip:$skipCondition -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ) {
    $Name | Should -Not -BeNullOrEmpty
    $Value | Should -Not -BeNullOrEmpty
  }
}

[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)
[!] Should display hashtable values, even when skipping <Name> <Value> 1ms (0ms|1ms)

Use Set-ItResult inside the body of the test:

Describe 'Test names with templates using ForEach' {
  BeforeAll {
    $skipCondition = ...
  }
  It 'Should display hashtable values when skipping using Set-ItResult <Name> <Value>' -ForEach @(
    @{ Name='foo'; Value='test 1' }
    @{ Name='bar'; Value='test 2' }
    @{ Name='foobar'; Value='test 3' }
  ) {
    if ($skipCondition) {
      Set-ItResult -Skipped -Because 'Skipping this test'
    }
    else {
      $Name | Should -Not -BeNullOrEmpty
      $Value | Should -Not -BeNullOrEmpty
    }
  }r
}

[!] Should display hashtable values when skipping using Set-ItResult foo test 1 is skipped, because Skipping this test 6ms (2ms|4ms)
[!] Should display hashtable values when skipping using Set-ItResult bar test 2 is skipped, because Skipping this test 4ms (2ms|2ms)
[!] Should display hashtable values when skipping using Set-ItResult foobar test 3 is skipped, because Skipping this test 4ms (2ms|2ms)

The catch is that the variable(s) used in the skip condition inside the test body need to be in BeforeAll instead of of BeforeDiscovery. In my particular case, I would need to duplicate that logic as multiple tests rely on the skip condition. Some can use -Skip and some would use Set-ItResult.

from pester.

In my particular case, I would need to duplicate that logic as multiple tests rely on the skip condition.

You can pass the skipcondition-value from Discovery to Run-phase through -ForEach. Add it as a key in your testcases, or as ... -ForEach @{ skipcondition = $skipCondition } on a parent Context/Describe. Will be available as $skipCondition in the test.

from pester.

Thanks for the detailed report. The current behavior is by design.

Pester expands variables in the name between test setup and execution to support variables set in BeforeAll/BeforeEach. Skipping a test will skip all of this.

Related issue #2220

from pester.

This would be hard for us to do and not end up with empty values for all the template names, offering about as much value as not expanding them at all. This is because the names are expanded by evaluating the string in the current execution contex of the test. So any variable or simple subexpression will be expanded correctly.

from pester.

Thank you for the insights @nohwnd, appreciate you taking the time to explain.

from pester.