Device simulator dependence on recursive description snapshots about swift-snapshot-testing HOT 7 CLOSED

Ah cool good to know! If it's only frames, then using the device sizing+traits we do for image snapshots probably would fix this! We could look into beefing up recursiveDescription to take the same options that .image does.

Another approach is for you and your team to settle on a single canonical simulator to use for recording screenshots. This is what we did at Kickstarter and other places, and seems to be a common approach.

Thanks for the info!

from swift-snapshot-testing.

Hey @kaqu! We haven't run into that yet cause we typically run our snapshot tests from a framework target instead of an app target. If you can do the same, then that would fix it. Otherwise it sounds like we need to encode some simulator properties into the file names of the snapshots for people that want to run tests on multiple simulators. We can look into that soon!

from swift-snapshot-testing.

Oh, also, that .iPhoneX stuff won't be too helpful here because that only overrides traits in the view/controller. I think you will still see some difference.

If you are willing to share, could you show us some of the differences you are getting on different simulators? I'm curious what it looks like. Thanks!

from swift-snapshot-testing.

Thanks for quick response 🙂 I always run tests on the same device, but my colleague found that when doing PR and running tests on a different device (currently selected in Xcode), so It would be nice to keep that in sync between developers without keeping in mind to test outside of Xcode or with certain settings.

Here is a diff I've got from one of my views:

@@ −1,11 +1,11 @@
-<UIView; frame = (0 0; 375 812); autoresize = W+H; layer = <CALayer>>
-   | <NavigationBar; frame = (0 0; 375 109); layer = <CALayer>>
-   |    | <UIStackView; frame = (20 44; 335 60); layer = <CATransformLayer>>
+<UIView; frame = (0 0; 414 896); autoresize = W+H; layer = <CALayer>>
+   | <NavigationBar; frame = (0 0; 414 109); layer = <CALayer>>
+   |    | <UIStackView; frame = (20 44; 374 60); layer = <CATransformLayer>>
    |    |    | <UIStackView; frame = (0 30; 0 0); layer = <CATransformLayer>>
-   |    |    | <Label; baseClass = UILabel; frame = (102.667 19.3333; 129.667 21.6667); text = 'TEXT'; userInteractionEnabled = NO; layer = <_UILabelLayer>>
-   |    |    | <UIStackView; frame = (335 30; 0 0); layer = <CATransformLayer>>
-   |    | <UIImageView; frame = (0 104; 375 5); opaque = NO; userInteractionEnabled = NO; layer = <CALayer>>
-   | <UIView; frame = (0 109; 375 703); autoresize = W+H; layer = <CALayer>>
-   |    | <Label; baseClass = UILabel; frame = (0 20; 375 0); hidden = YES; userInteractionEnabled = NO; layer = <_UILabelLayer>>
-   |    | <ImageView; baseClass = UIImageView; frame = (140.667 214.667; 94 94); opaque = NO; userInteractionEnabled = NO; layer = <CALayer>>
-   |    | <Label; baseClass = UILabel; frame = (0 328.667; 375 19.3333); text = 'TEXT'; userInteractionEnabled = NO; layer = <_UILabelLayer>>
+   |    |    | <Label; baseClass = UILabel; frame = (122.333 19.3333; 129.667 21.6667); text = 'TEXT'; userInteractionEnabled = NO; layer = <_UILabelLayer>>
+   |    |    | <UIStackView; frame = (374 30; 0 0); layer = <CATransformLayer>>
+   |    | <UIImageView; frame = (0 104; 414 5); opaque = NO; userInteractionEnabled = NO; layer = <CALayer>>
+   | <UIView; frame = (0 109; 414 787); autoresize = W+H; layer = <CALayer>>
+   |    | <Label; baseClass = UILabel; frame = (0 20; 414 0); hidden = YES; userInteractionEnabled = NO; layer = <_UILabelLayer>>
+   |    | <ImageView; baseClass = UIImageView; frame = (160 256.667; 94 94); opaque = NO; userInteractionEnabled = NO; layer = <CALayer>>
+   |    | <Label; baseClass = UILabel; frame = (0 370.667; 414 19.3333); text = 'TEXT'; userInteractionEnabled = NO; layer = <_UILabelLayer>>

Recorded on default iPhoneX and failed on default iPhone Xs Max simulator. Looks like only frames changed.

from swift-snapshot-testing.

It will be nice to have tests independent from developer environment. You can achieve this using specialized image snapshots, it will produce always the same results. Making that possible for recursive description might increase its usage (it is awesome!) 🙂 I am wondering if this is the case for any other snapshot type, but it might be only view specific (I don't know if that is the case for NSView since I have used it only on iOS).

from swift-snapshot-testing.

I have noticed something that maybe related to this issue, I still have limited understanding of when the layout constraints update took place, and when the view is being snapshot, which could affect the result of the snapshot output.

I noticed that if I declare a constraint that's dependent on a view.bounds property at runtime, this will cause the snapshot output to be dependent on which Simulator with the corresponding screen size that we're running on.

For example, the following code will only produce a snapshot output that took the value of view.bounds.width in reference to the simulator it's running on, instead of the ViewImageConfig of the snapshot image. As a result the width is smaller when it's displayed on a larger view configuration.

override func viewDidLoad() {
    super.viewDidLoad()

    let offset: CGFloat = 20.0
    textField.widthAnchor.constraint(equalTo: view.bounds.width - 2*offset).isActive = true
    
    // This text field is off screen initially and it doesn't have a right anchor, 
    // but instead constrained by its width anchor
    textField.leftAnchor.constraint(equalTo: view.leftAnchor, constant: view.bounds.width + offset).isActive = true
}

I realized this is an unconventional layout declaration, so let me visualize it:

1. The initial layout

2. On user action, text field will be animated to the left, and then layout will be updated

I'm not sure if I should have find a more stable declaration of width anchor of the text field, or whether should I dig deeper into the Snapshot testing process, to resolve this.

Also, I have tried to improve the layout constraints declaration, to rely on the widthAnchor of the view of a view controller. Which surprisingly works at runtime. I thought I have to set a widthAnchor explicitly to a value before using it. But, even in this case, snapshot still produced unexpected output if I update the initial layout to a runtime value of a view.bounds, and then take snapshot on different ViewImageConfig.

from swift-snapshot-testing.

Also, I have tried to improve the layout constraints declaration, to rely on the widthAnchor of the view of a view controller. Which surprisingly works at runtime. I thought I have to set a widthAnchor explicitly to a value before using it. But, even in this case, snapshot still produced unexpected output if I update the initial layout to a runtime value of a view.bounds, and then take snapshot on different ViewImageConfig.

I discovered that to maintain consistency on various image snapshot of layout update on taking snapshot (updating means to modify the constant value of NSLayoutConstraint), a different instance of each view controller is needed.

For example, instead of this:

assertSnapshot(matching: vc, as: .image(on: .iPhoneSe))
assertSnapshot(matching: vc, as: .image(on: .iPhone8))
assertSnapshot(matching: vc, as: .image(on: .iPhoneX))

A separate instance need to be used:

func viewController() -> UIViewController {
    return UIViewController()
}

assertSnapshot(matching: viewController(), as: .image(on: .iPhoneSe))
assertSnapshot(matching: viewController(), as: .image(on: .iPhone8))
assertSnapshot(matching: viewController(), as: .image(on: .iPhoneX))

from swift-snapshot-testing.