Test failure: CheckPasswordSignInReturnsLockedOutWhenLockedOut about identity HOT 16 CLOSED

It's somewhat possible, but it was the entire purpose of having TState was to be able to pass structs without boxing. Seems like an safe enough change for 3.0 milestone.

cc @Eilon

from identity.

Oh, if this if for 3.0 then no problem at all. :-)

from identity.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

Please use this workflow to address this flaky test issue, including checking applicable checkboxes and filling in the applicable "TODO" entries:

• Is this actually a flaky test?

  • No, this is a regular test failure, fix the test/product (TODO: Link to commit/PR)
  • Yes, proceed below...

• Is this test failure caused by product code flakiness? (Either this product, or another product this test depends on.)

  • File a bug against the product (TODO: Link to other bug)
  • Is it possible to change the test to avoid the flakiness?
    • Yes? Go to the "Change the test!" section.
    • No?
      • Disable the test (TODO: Link to PR/commit)
      • Wait for other bug to be resolved
      • Wait for us to get build that has the fix
      • Re-enable our test (TODO: Link to PR/commit)
      • Close this bug

• Is it that the test itself is flaky? This includes external transient problems (e.g. remote server problems, file system race condition, etc.)

  • Is there is a way to change our test to avoid this flakiness?
    • Yes? Change the test!
      • Change the test to avoid the flakiness, for example by using a different test strategy, or by adding retries w/ timeouts (TODO: Link to PR/commit)
      • Run the test 100 times locally as a sanity check.
      • Close this bug
    • No?
      • Is there any logging or extra information that we could add to make this more diagnosable when it happens again?
        • Yes?
          • Add the logging (TODO: Link to PR/commit)
        • No?
          • Delete the test because flaky tests are not useful (TODO: Link to PR/commit)

from identity.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 1 failures with about the same error on master at 1:50:55 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 1:50:55 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 1:50:55 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 1:51:12 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

from identity.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 2 failures with about the same error on master at 1:50:55 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 1:50:54 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 1:50:54 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:29:07 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:29:05 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 3:38:37 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:29:54 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:52:01 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:59:15 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 4:57:20 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:33 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:33 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 2:07:13 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:26 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 2:07:16 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:30 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:31 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 12:21:31 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 2:07:10 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

from identity.

@HaoK please investigate.

from identity.

Weird this isn't failing for me locally in master, but its definitely failing on all ci builds right now, @ajcvickers has anything changed in EF recently? The submodules updated before these tests started failing were:

EntityFrameworkCore => 647aed13836f1f4f0d3c3d7f6a9a0165826e3e6a
Logging => cc90113f450182d472ae155767ff388fc92698e8
Mvc => 13cf754425c6120ac0b062dd51da2b89ac161dc2
Razor => 4d44639a51643223d5eb4ca317e201845c79d0c3

from identity.

Okay so something has regressed that has caused the logging mocks to fail @BrennanConroy has something changed in logging recently?

This mock logger code hasn't changed since 1.0 probably, but it appears to no longer get called from a log warning:

   // Actual log code which is indeed getting hit:
  Logger.LogWarning(3, "User {userId} is currently locked out.", await UserManager.GetUserIdAsync(user));

  // Mock code which is no longer getting invoked which is recording calls to a string
        public static Mock<ILogger<T>> MockILogger<T>(StringBuilder logStore = null) where T : class
        {
            logStore = logStore ?? LogMessage;
            var logger = new Mock<ILogger<T>>();
            logger.Setup(x => x.Log(It.IsAny<LogLevel>(), It.IsAny<EventId>(), It.IsAny<object>(),
                It.IsAny<Exception>(), It.IsAny<Func<object, Exception, string>>()))
                .Callback((LogLevel logLevel, EventId eventId, object state, Exception exception, Func<object, Exception, string> formatter) =>
                {
                    if (formatter == null)
                    {
                        logStore.Append(state.ToString());
                    }
                    else
                    {
                        logStore.Append(formatter(state, exception));
                    }
                    logStore.Append(" ");
                });
            logger.Setup(x => x.BeginScope(It.IsAny<object>())).Callback((object state) =>
                {
                    logStore.Append(state.ToString());
                    logStore.Append(" ");
                });
            logger.Setup(x => x.IsEnabled(LogLevel.Debug)).Returns(true);
            logger.Setup(x => x.IsEnabled(LogLevel.Warning)).Returns(true);

            return logger;
        }

from identity.

We started using structs as TState in ILogger.Log() and logger.Setup(x => x.Log(It.IsAny<LogLevel>(), It.IsAny<EventId>(), It.IsAny<object>(), doesn't get invoked when T is struct.

I've fixed this for kestrel:
aspnet/KestrelHttpServer#3066

from identity.

@pakrym Isn't this a breaking change?

from identity.

In what way?

from identity.

@ajcvickers are you ok if I just remove this mocking entirely? There is limited value in verifying these log messages, we already have tests verifying that the proper results are being returned, and we are only checking it in 5 tests anyways

from identity.

There is a great alternative to mocking - LoggedTest base class that not only provides TestSink with the list of all messages logged but also forwards them to xunit test output.

For example this is how kestrel asserts log messages:
https://github.com/aspnet/KestrelHttpServer/blob/d29e410b035a3fe7db0d4625ee37cd6cb76e40d3/test/Kestrel.Transport.FunctionalTests/RequestTests.cs#L684

from identity.

@HaoK I'm fine with that. I was just concerned the production code consuming logs might be assuming that TState is not a struct and breaking in the same way. If that's not a worry, then no problem.

from identity.

This comment was made automatically. If there is a problem contact ryanbrandenburg.

There were 2 failures with about the same error on master at 3:03:03 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 3:24:32 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 3:32:40 PM:

• CheckPasswordSignInReturnsLockedOutWhenLockedOut
• PasswordSignInReturnsLockedOutWhenLockedOut

There were 2 failures with about the same error on master at 3:29:02 PM:

• PasswordSignInReturnsLockedOutWhenLockedOut
• CheckPasswordSignInReturnsLockedOutWhenLockedOut

from identity.

Was fixed by 1fde1cc

from identity.