Crash when used in test target with asynchronous tests about swift-snapshot-testing HOT 5 CLOSED

Dang, that's super frustrating 😕

A repro would be great. We haven't encountered this yet, but if we can narrow it down to a bug we can fix or file with Apple, that'd be great 😄

from swift-snapshot-testing.

Ok, I have finally managed to narrow down the cause to a single test function and have created a sample project which will repro the issue every time: https://github.com/mluisbrown/SnapshotBug

The README for the repo explains everything, but I'll copy some of it here so the information is all in one place:

When you run the tests (⌘U) they run fine but end in a crash in UIApplicationMain:

Thread 1: EXC_BAD_INSTRUCTION (code=EXC_I386_INVOP, subcode=0x0)

I have enabled Zombie Objects diagnostics for the test target and with that enabled you also get an error in the console similar to:

*** -[CALayer setNeedsLayout]: message sent to deallocated instance 0x60000327d100

You can make the bug happen earlier (before the tests end) if you uncomment the autoreleasepool block in SnapshotBugTests.swift

This will cause a slightly different Zombie error to appear in the console:

*** -[UIWindowLayer isHidden]: message sent to deallocated instance 0x600000d4cd60

This bug is somehow related with the async nature of the test and the fact that SnapshotTesting also uses XCWaiter. This kind of test works fine when using ios-snapshot-test-case which is not async.

Somehow the combination of the two async code paths is probably causing the run loop to release objects which otherwise would not be released.

The root cause of the crashes is async tests similar to the once in the sample repo. Other types of async tests, eg with Nimble's toEventually don't cause the problem, but they can surface the crash earlier in the test suite.

from swift-snapshot-testing.

Just to clarify, if you do your async testing using .asyncPullback this issue does not exist. It's an issue if you use SnapshotTesting together with your own async code.

So, I wouldn't categorize it as a bug necessarily, but it's something users should be aware of.

from swift-snapshot-testing.

I'm experiencing the same issue very frequently now and was able to catch it using the address sanitizer SUMMARY: AddressSanitizer: heap-use-after-free (libclang_rt.asan_iossim_dynamic.dylib:x86_64+0x1a547) in wrap_memcpy.

Please let me know if there's more information that I can provide to help to isolate and fix this issue.

I also noticed it happens in a view controller snapshot test in which the view controller displays a MKMapView, does not happen in any other of our snapshot tests.

from swift-snapshot-testing.

I'm going to convert this to a discussion, as @mluisbrown notes, it's something folks may want to be aware of. If there are future related issues found that are due to bugs in SnapshotTesting that we can fix, though, please do open an issue!

from swift-snapshot-testing.