Code Monkey home page Code Monkey logo

swift-corelibs-xctest's Introduction

XCTest

The XCTest library is designed to provide a common framework for writing unit tests in Swift, for Swift packages and applications.

This version of XCTest implements the majority of unit testing APIs included in XCTest from Xcode 7 and later. Its goal is to enable your project's tests to run on all the platforms Swift supports without having to rewrite them.

Using XCTest

Your tests are organized into a simple hierarchy. Each XCTestCase subclass has a set of test methods, each of which should test one part of your code.

For general information about using XCTest, see:

Using XCTest with Swift Package Manager

The Swift Package Manager integrates directly with XCTest to provide a streamlined experience for unit testing SwiftPM packages. If you are using XCTest within a SwiftPM package, unit test files are located within the package's Tests subdirectory, and you can build and run the full test suite in one step by running swift test.

For more information about using XCTest with SwiftPM, see its documentation.

Standalone Command Line Usage

When used by itself, without SwiftPM, this version of XCTest does not use the external xctest CLI test runner included with Xcode to run tests. Instead, you must create your own executable which links libXCTest.so, and in your main.swift, invoke the XCTMain function with an array of the tests from all XCTestCase subclasses that you wish to run, wrapped by the testCase helper function. For example:

XCTMain([
    testCase(TestNSString.allTests),
    testCase(TestNSArray.allTests),
    testCase(TestNSDictionary.allTests),
])

The XCTMain function does not return, and will cause your test executable to exit with either 0 for success or 1 for failure. Certain command line arguments can be used to modify the test runner behavior:

  • A particular test or test case can be selected to execute. For example:
$ ./FooTests Tests.FooTestCase/testFoo                            # Run a single test method
$ ./FooTests Tests.FooTestCase                                    # Run all the tests in FooTestCase
$ ./FooTests Tests.FooTestCase/testFoo,Tests.FooTestCase/testBar  # Run multiple test methods
  • Tests can be listed, instead of executed.
$ ./FooTests --list-tests
Listing 4 tests in FooTests.xctest:

Tests.FooTestCase/testFoo
Tests.FooTestCase/testBar
Tests.BarTestCase/test123

$ ./FooTests --dump-tests-json
{"tests":[{"tests":[{"tests":[{"name":"testFoo"},{"name":"testBar"}],"name":"Tests.FooTestCase"},{"tests":[{"name":"test123"}],"name":"Tests.BarTestCase"}],"name":"Tests.xctest"}],"name":"All tests"}

Contributing to XCTest

To contribute, you'll need to be able to build this project and and run its test suite. The easiest way to do so is via the Swift build script.

First, follow the instructions in the Swift README to build Swift from source. Confirm you're able to build the Swift project using utils/build-script -R.

Once you are able to build the Swift project, build XCTest and run its tests:

$ cd swift-corelibs-xctest
$ ../swift/utils/build-script --preset corelibs-xctest

This project is only guaranteed to build with the very latest commit on the Swift and swift-corelibs-foundation main branches. You may update to the latest commits using the Swift utils/update-checkout script:

$ ../swift/utils/update-checkout

Using Xcode

To browse files in this project using Xcode, use XCTest.xcworkspace. You may build the project using the SwiftXCTest scheme. Run the SwiftXCTestFunctionalTests scheme to run the tests.

However, in order to successfully build the project in Xcode, you must use an Xcode toolchain with an extremely recent version of Swift. The Swift website provides Xcode toolchains to download, as well as instructions on how to use Xcode with those toolchains. Swift development moves fairly quickly, and so even a week-old toolchain may no longer work.

If none of the toolchains available to download are recent enough to build XCTest, you may build your own toolchain by using the utils/build-toolchain script in the Swift repository.

Keep in mind that the build script invocation in "Contributing to XCTest" above will always work, regardless of which Swift toolchains you have installed. The Xcode workspace exists simply for the convenience of contributors. It is not necessary to successfully build this project in Xcode in order to contribute.

swift-corelibs-xctest's People

Contributors

briancroom avatar compnerd avatar csknns avatar ddunbar avatar douggregor avatar drodriguez avatar eschaton avatar finagolfin avatar gribozavr avatar grynspan avatar ianpartridge avatar ikesyo avatar johnno1962 avatar jrose-apple avatar kateinoigakukun avatar larryonoff avatar marcusrossel avatar maxdesiatov avatar michael-lehew avatar modocache avatar parkera avatar phausler avatar rudkx avatar seabaylea avatar seanrolszewski avatar shahmishal avatar slavapestov avatar spevans avatar stmontgomery avatar tkremenek avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

swift-corelibs-xctest's Issues

[SR-906] Allow XCTestCase.continueAfterFailure to be set

Previous ID SR-906
Radar None
Original Reporter @modocache
Type Improvement
Additional Detail from JIRA
Votes 2
Component/s XCTest
Labels Improvement
Assignee None
Priority Medium

md5: 8a68ba9a31eeee0ca919922b7bddbeee

Issue Description:

XCTestCase.continueAfterFailure currently always returns true, no matter what:

public var continueAfterFailure: Bool {
get {
return true
}
set {
// TODO: When using the Objective-C runtime, XCTest is able to throw an exception from an assert and then catch it at the frame above the test method. This enables the framework to effectively stop all execution in the current test. There is no such facility in Swift. Until we figure out how to get a compatible behavior, we have decided to hard-code the value of 'true' for continue after failure.
}
}

However, as of #40 there is no reason for this attribute to always return true--if a user sets it to false, it should currently work (grep the codebase for "continueAfterFailure" to see for yourself).

1. Remove the custom setter/getter from continueAfterFailure in order to allow it to be set to true/false by the user.
2. Add functional tests that confirm the test suite stops when continueAfterFailure is false. This might be tricky--a false value for continueAfterFailure causes the test to throw a fatalError, and none of the functional tests currently test this sort of behavior. Still, I don't think this will require any massive changes to the testing architecture in the repo. Start by adding a simple test case that uses continueAfterFailure and check the output. It should be similar to other functional test output.

[SR-7615] swift-corelibs-xctest missing `XCTWaiter`

Previous ID SR-7615
Radar rdar://problem/41022833
Original Reporter aamct (JIRA User)
Type Bug
Status Resolved
Resolution Done
Environment
  • Ubuntu 16.04

  • Swift 4.1

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @stmontgomery
Priority Medium

md5: b7e47078a4f8c5a0559964725b530169

is duplicated by:

  • SR-6249 Asynchronous testing missing features
  • SR-9010 Linux XCTest missing "wait(for expectations:)" helpers

Issue Description:

Currently, swift-corelibs-xctest does not have `XCTWaiter`.

[SR-4573] Date format inconsistent between XCTest on Darwin and Linux

Previous ID SR-4573
Radar None
Original Reporter @alblue
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @alblue
Priority Medium

md5: 36909d36fd2030528ab81fc2b786c37b

Issue Description:

On Darwin, tests get displayed as:

Test Suite 'All tests' started at 2017-04-12 16:50:11.630

On Linux, tests get displayed as:

Test Suite 'All tests' started at 16:06:28.533

The year/month/day is missing from the Linux version

formatter.dateFormat = "HH:mm:ss.SSS"

[SR-907] Add convenience XCTestExpectation constructors for NSNotification

Previous ID SR-907
Radar None
Original Reporter @modocache
Type Improvement
Status Closed
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee apps.yoon (JIRA)
Priority Medium

md5: 2bc0310ca30f22236b57ae9be6504fd5

Issue Description:

Apple XCTest offers several convenient methods for constructing XCTestExpectations:

@interface XCTestCase : XCTest
// ...
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath expectedValue:(nullable id)expectedValue;
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath handler:(nullable XCKeyValueObservingExpectationHandler)handler;
- (XCTestExpectation *)expectationForNotification:(NSString *)notificationName object:(nullable id)objectToObserve handler:(nullable XCNotificationExpectationHandler)handler;
- (XCTestExpectation *)expectationForPredicate:(NSPredicate *)predicate evaluatedWithObject:(id)object handler:(nullable XCPredicateExpectationHandler)handler;
// ...
@end

Key-value observing doesn't exist in native Swift, but NSNotification and NSPredicate do (thanks to swift-corelibs-foundation). Add the following convenience constructors:

1. expectationForNotification:object:handler:
2. expectationForPredicate:evaluatedWithObject:handler:

Also, note that you'll need to add the following closure typealiases as well:

typedef BOOL (^XCNotificationExpectationHandler)(NSNotification *notification);
typedef BOOL (^XCPredicateExpectationHandler)();

[SR-460] API incompatibility of XCTestCase setUp() and tearDown() methods

Previous ID SR-460
Radar None
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: d4d2b61860ed289302e1fd98c142f4e9

Issue Description:

The following trivial test case, taken nearly verbatim from Xcode's "Unit Test Case Class" file template, builds and runs using Apple XCTest, but fails to compile using `swift-corelibs-xctest`:

import XCTest

class Test: XCTestCase {
    var allTests: [(String, () -> Void)] { return [("testExample", testExample)] }

    override func setUp() {
        super.setUp()
    }

    override func tearDown() {
        super.tearDown()
    }

    func testExample() {
        XCTAssertEqual(1, 1)
    }
}

The errors produced are:

/home/osboxes/swift/LetsTryXCTest/Sources/Test.swift:6:19: error: method does not override any method from its superclass
    override func setUp() {
    ~~~~~~~~      ^
/home/osboxes/swift/LetsTryXCTest/Sources/Test.swift:10:19: error: method does not override any method from its superclass
    override func tearDown() {
    ~~~~~~~~      ^
/home/osboxes/swift/LetsTryXCTest/Sources/Test.swift:7:9: error: 'super' members cannot be referenced in a root class
        super.setUp()
        ^
/home/osboxes/swift/LetsTryXCTest/Sources/Test.swift:11:9: error: 'super' members cannot be referenced in a root class
        super.tearDown()
        ^

The reason for this is that when building with Apple XCTest, the `setUp` and `tearDown` methods are defined on the `XCTest` base class and thus must include the `override` modifier when used in client code, whereas `swift-corelibs-xctest` defines them in the `XCTestCase` protocol (or should, anyway. See [this PR](#26

While I am generally a big fan of the idea of defining `XCTestCase` as a protocol instead of a class that must be derived from, I don't think there is any way to maintain source-level compatibility for users of the framework without converting `XCTestCase` to be a class, thus allowing these test lifecycle methods to be overridden.

[SR-5643] Adopt Swift 4 API adjustments in XCTest

Previous ID SR-5643
Radar rdar://problem/33778877
Original Reporter @glessard
Type Bug
Status Closed
Resolution Done
Environment

ubuntu 16.04, swift-4.0-DEVELOPMENT-SNAPSHOT-2017-08-04-a

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: d9de2b31a27499cd80ba9741088859ea

Issue Description:

XCTestCase.defaultPerformanceMetrics() has not been updated yet in the swift-corelibs-xctest project, making it inconsistent with Xcode's spelling, where it has become a class var.

Similarly, the old XCTPerformanceMetric_WallClockTime has become XCTPerformanceMetric.wallClockTime in Xcode, but is unchanged in swift-corelibs-xctest.

There are some other API adjustments in the XCTest API, such as moving `UInt` types to `Int`, that also should be adopted in swift-corelibs-xctest.

[SR-1944] [corelibs-xctest] Replace xctest-checker with FileCheck

Previous ID SR-1944
Radar None
Original Reporter @modocache
Type Improvement
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement
Assignee None
Priority Medium

md5: 961cab0c1de40e4880f026d33ac12047

Issue Description:

xctest-checker was originally added in #20 as a replacement for FileCheck. At that time, it was easy to build swift-corelibs-xctest without first building apple/swift from source. As we continued to develop xctest-checker into a more fully-featured FileCheck replacement, @briancroom wondered whether it would make sense to switch to FileCheck completely: #94 (comment) I argued against this because it would mean completely coupling the corelibs-xctest build to the overall apple/swift build.

However, as its dependencies on other projects such as swift-corelibs-foundation have expanded, swift-corelibs-xctest has been encouraging contributors to use the Swift build script to build and test the project for a long while now:

apple/swift-corelibs-xctest $ ../swift/utils/build-script --preset corelibs-xctest

The Swift build script has access to the LLVM bin directory, and as such it knows where to find FileCheck.

Is it time to replace xctest-checker with FileCheck? I think the answer is yes. We can migrate using the following steps:

  1. XCTest's build_script.py test subcommand should take an optional path to a FileCheck executable to use when testing. At first, the option will do nothing.

  2. The apple/swift build script should be modified to pass the path to FileCheck to the build script.

  3. XCTest's lit.cfg should provide a %{FileCheck} substitution to the tests.

  4. Tests should migrate over from using the %{xctest_checker} substitution to the %{FileCheck} substitution.

  5. Once all tests have been migrated, xctest-checker should be deleted.

[SR-2507] Can't test optimized builds

Previous ID SR-2507
Radar None
Original Reporter @drewcrawford
Type Bug
Environment

Linux x64
swift-3.0-PREVIEW-6

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, Linux, OptimizedOnly
Assignee None
Priority Medium

md5: e75c7ef8d15f885bc7339674089eb704

Issue Description:

$ cat test.swift
import XCTest

class Foo: XCTestCase {
}

$ swift test.swift
$ swift -O test.swift
LLVM ERROR: Program used external function '_TFC6XCTest10XCTestCasem17_performanceMeterGSqCS_16PerformanceMeter_' which could not be resolved!

[SR-2433] [XCTest] PR CI is malfunctioning on OS X

Previous ID SR-2433
Radar None
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @modocache
Priority Medium

md5: 80b3b7aa4bab8da24c68dab0e0e28e68

Issue Description:

As discussed here, CI builds triggered in PRs aren't currently sending back a passing status even if the XCTest functional tests fail.

[SR-1165] Swift 3 importer creates duplicate XCTest.run() methods

Previous ID SR-1165
Radar rdar://25780530
Original Reporter @modocache
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, ClangImporter, SDKOverlay, StarterBug, XCTest
Assignee @modocache
Priority Medium

md5: 86b338119b5fe40134e34b05d44c80d9

Issue Description:

I used swift-api-dump.py to generate XCTest.framework headers:

$ export SWIFT_BIN_DIR="/path/to/build/Ninja-ReleaseAssert/swift-macosx-x86_64/bin"
$ $SWIFT_BIN_DIR/swift-api-dump.py -s macosx -m XCTest \
    --framework-dir /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Frameworks/

Using a recent commit of Swift (apple/swift@fc86f6e86ef5405756e0052e27dae1830e3d33bc), the importer generates two XCTestCase.run() definitions:

class XCTest : NSObject {
  // ...
  @available(*, deprecated)
  func run() -> XCTestRun
  func run()
}

These two should be disambiguated by being given different names. Alternatively, the deprecated run() method could be removed--although I'd expect you'd need to confirm with the Apple XCTest team that this is OK.

[SR-1609] Support recording baseline for performance regression tests

Previous ID SR-1609
Radar None
Original Reporter @briancroom
Type Bug
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: 89bb688c4da749aeb43653263685237d

Issue Description:

We added basic support for XCTest's performance testing APIs in https://bugs.swift.org/browse/SR-1355, however one crucial component of that is still missing, namely recording baseline results of performance tests and triggering test failures when significant regressions are detected relative to those baselines.

I see a few pieces to this:

  • Persisting baseline data to disk. Xcode uses a Plist format for this. It would probably make sense to use a compatible format here.

  • Comparing test results to a previously recorded baseline and triggering a failure if the performance has regressed. Xcode's supposedly triggers a failure if Average - BaselineAverage is greater than 0.1 seconds and is more than 10% higher than BaselineAverage itself.

  • Providing a mechanism (probably command-line flags?) instructing the runner where to find and create baseline data, and for causing the baseline data to be updated.

[SR-53] XCTAssertThrows in XCTestCase

Previous ID SR-53
Radar None
Original Reporter awph (JIRA User)
Type New Feature
Status Closed
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels New Feature
Assignee None
Priority Medium

md5: d270d045d4b293ed2a829c44aada07b5

Issue Description:

The Objc version of XCTest has some macros to assert functions that should throw an exception. Basically it just creates a try-catch and if the catch part isn't reach, the XCTest will register a failure.
The Swift version of the framework doesn't have a method to do this. I've my own extension to handle this kind of situation, but it's very limited and not clean.

I would appreciate to have your feedback on the idea to add this feature, and also how you think it would be interesting to add it.

I attached an example that I've done to understand this situation. This is just a draft with some quick made stuff to try how to handle this situation. ๐Ÿ˜‰

[SR-4374] XCTest breaks when name of test class and module are same

Previous ID SR-4374
Radar rdar://problem/31272389
Original Reporter @aciidb0mb3r
Type Bug
Status Resolved
Resolution Done
Environment

macOS

Additional Detail from JIRA
Votes 0
Component/s Compiler, XCTest
Labels Bug
Assignee @eeckstein
Priority Medium

md5: faec7f397bf8f410093b93b78914ca45

Issue Description:

The swift package init command create a simple package with one library and one test target. The test target has one test case subclass with same name as the test target. In this case, the test target name seems to be mangled when running the test and XCTest's selected test option doesn't work.

$ export TOOLCHAINS=swift
$ xcrun --find swift
/Library/Developer/Toolchains/swift-DEVELOPMENT-SNAPSHOT-2017-03-26-a.xctoolchain/usr/bin/swift
$ mkdir Hello && cd Hello
$ swift package init
$ swift test
Compile Swift Module 'Hello' (1 sources)
Compile Swift Module 'HelloTests' (1 sources)
Linking ./.build/debug/HelloPackageTests.xctest/Contents/MacOS/HelloPackageTests
Test Suite 'All tests' started at 2017-03-27 18:33:53.841
Test Suite 'HelloPackageTests.xctest' started at 2017-03-27 18:33:53.841
Test Suite '*_TtC10HelloTestsAA*' started at 2017-03-27 18:33:53.841
Test Case '-[*_TtC10HelloTestsAA* testExample]' started.
Test Case '-[*_TtC10HelloTestsAA* testExample]' passed (0.074 seconds).
Test Suite '*_TtC10HelloTestsAA*' passed at 2017-03-27 18:33:53.916.
     Executed 1 test, with 0 failures (0 unexpected) in 0.074 (0.074) seconds
Test Suite 'HelloPackageTests.xctest' passed at 2017-03-27 18:33:53.916.
     Executed 1 test, with 0 failures (0 unexpected) in 0.074 (0.075) seconds
Test Suite 'All tests' passed at 2017-03-27 18:33:53.916.
     Executed 1 test, with 0 failures (0 unexpected) in 0.074 (0.075) seconds

$ swift test -s HelloTests.HelloTests
Test Suite 'Selected tests' started at 2017-03-27 18:34:33.328
Test Suite 'HelloPackageTests.xctest' started at 2017-03-27 18:34:33.328
Test Suite 'HelloPackageTests.xctest' passed at 2017-03-27 18:34:33.328.
     Executed *0 tests*, with 0 failures (0 unexpected) in 0.000 (0.000) seconds
Test Suite 'Selected tests' passed at 2017-03-27 18:34:33.328.
     Executed *0 tests*, with 0 failures (0 unexpected) in 0.000 (0.000) seconds

Change the HelloTests class to any other name in HelloTests.swift and everything will start working as expected.

[SR-6182] Unable to compile Corelibs XCTest when using a non-standard install prefix

Previous ID SR-6182
Radar rdar://problem/35059647
Original Reporter @aciidb0mb3r
Type Bug
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: fa734b5729da24df6af4e66984e47321

Issue Description:

./swift/utils/build-script --release --llbuild --swiftpm --xctest --foundation --libdispatch  --build-subdir linux -- --install-swift --install-llbuild --install-swiftpm --install-xctest --install-prefix=/tmp/swift '--swift-install-components=autolink-driver;compiler;clang-builtin-headers;stdlib;swift-remote-mirror;sdk-overlay;license;sourcekit-inproc' --install-foundation --install-libdispatch
+ popd
~/swift.org
xctest: using gold linker
+ /home/ubuntu/swift.org/swift-corelibs-xctest/build_script.py --swiftc=/home/ubuntu/swift.org/build/linux/swift-linux-x86_64/bin/swiftc --build-dir=/home/ubuntu/swift.org/build/linux/xctest-linux-x86_64 --foundation-build-dir=/home/ubuntu/swift.org/build/linux/foundation-linux-x86_64/Foundation --libdispatch-src-dir=/home/ubuntu/swift.org/swift-corelibs-libdispatch --libdispatch-build-dir=/home/ubuntu/swift.org/build/linux/libdispatch-linux-x86_64 --release
xctest-build: mkdir -p /home/ubuntu/swift.org/build/linux/xctest-linux-x86_64
xctest-build: /home/ubuntu/swift.org/build/linux/swift-linux-x86_64/bin/swiftc -Xcc -fblocks -c -O -emit-object -emit-module -module-name XCTest -module-link-name XCTest -parse-as-library -emit-module-path /home/ubuntu/swift.org/build/linux/xctest-linux-x86_64/XCTest.swiftmodule -force-single-frontend-invocation -swift-version 4 -I /home/ubuntu/swift.org/build/linux/foundation-linux-x86_64/Foundation -I /home/ubuntu/swift.org/build/linux/foundation-linux-x86_64/Foundation/usr/lib/swift -I /home/ubuntu/swift.org/build/linux/libdispatch-linux-x86_64/src -I /home/ubuntu/swift.org/swift-corelibs-libdispatch  /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestSuiteRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestObservation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestSuite.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestErrors.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCase.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTAssert.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestMain.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCaseRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestObservationCenter.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCAbstractTest.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCase+Performance.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+Asynchronous.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+NotificationExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+PredicateExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCWaitCompletionHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCNotificationExpectationHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCPredicateExpectationHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/WallClockTimeMetric.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/PrintObserver.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/TestListing.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCTestInternalObservation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/TestFiltering.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/ObjectWrapper.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/PerformanceMeter.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCPredicateExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/ArgumentParser.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCTestCaseSuite.swift -o /home/ubuntu/swift.org/build/linux/xctest-linux-x86_64/XCTest.o
/home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestMain.swift:20:23: error: missing required module 'CoreFoundation'
    @_exported import Foundation
                      ^
Traceback (most recent call last):
  File "/home/ubuntu/swift.org/swift-corelibs-xctest/build_script.py", line 569, in <module>
    main()
  File "/home/ubuntu/swift.org/swift-corelibs-xctest/build_script.py", line 565, in main
    parsed_args.func(parsed_args)
  File "/home/ubuntu/swift.org/swift-corelibs-xctest/build_script.py", line 201, in build
    source_paths=" ".join(sourcePaths)))
  File "/home/ubuntu/swift.org/swift-corelibs-xctest/build_script.py", line 31, in run
    subprocess.check_call(command, shell=True)
  File "/usr/lib/python2.7/subprocess.py", line 541, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '/home/ubuntu/swift.org/build/linux/swift-linux-x86_64/bin/swiftc -Xcc -fblocks -c -O -emit-object -emit-module -module-name XCTest -module-link-name XCTest -parse-as-library -emit-module-path /home/ubuntu/swift.org/build/linux/xctest-linux-x86_64/XCTest.swiftmodule -force-single-frontend-invocation -swift-version 4 -I /home/ubuntu/swift.org/build/linux/foundation-linux-x86_64/Foundation -I /home/ubuntu/swift.org/build/linux/foundation-linux-x86_64/Foundation/usr/lib/swift -I /home/ubuntu/swift.org/build/linux/libdispatch-linux-x86_64/src -I /home/ubuntu/swift.org/swift-corelibs-libdispatch  /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestSuiteRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestObservation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestSuite.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestErrors.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCase.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTAssert.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestMain.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCaseRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestObservationCenter.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCAbstractTest.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestCase+Performance.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/XCTestRun.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+Asynchronous.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+NotificationExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCTestCase+PredicateExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCWaitCompletionHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCNotificationExpectationHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Public/Asynchronous/XCPredicateExpectationHandler.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/WallClockTimeMetric.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/PrintObserver.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/TestListing.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCTestInternalObservation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/TestFiltering.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/ObjectWrapper.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/PerformanceMeter.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCPredicateExpectation.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/ArgumentParser.swift /home/ubuntu/swift.org/swift-corelibs-xctest/Sources/XCTest/Private/XCTestCaseSuite.swift -o /home/ubuntu/swift.org/build/linux/xctest-linux-x86_64/XCTest.o' returned non-zero exit status 1
./swift/utils/build-script: fatal error: command terminated with a non-zero exit status 1, aborting

[SR-1129] Linker step fails when subclassing a class with a 'public private(set) var'

Previous ID SR-1129
Radar None
Original Reporter @modocache
Type Bug
Status Closed
Resolution Done
Environment
  • Ubuntu 15.10, Linux x86_64

  • Swift commit 4c57791 (Sat Apr 2 10:46:18 2016 -0700)

Additional Detail from JIRA
Votes 0
Component/s Compiler, XCTest
Labels Bug, Linux
Assignee @eeckstein
Priority Medium

md5: 72f6e819d92013f961797857f3f1a7cb

relates to:

  • SR-272 Linker error: Undefined reference to _TFC10Foundation8NSObjectg9_cfTypeIDSu
  • SR-1901 Linker error "hidden symbol" on subclassing Thread

Issue Description:

Pull request apple/swift#2039 adds tests demonstrating the problem. Namely, the following test fails on Linux:

// test/Parse/Inputs/public_var_private_setter.swift
public class BaseClass {
       public internal(set) var variable: Int = 0
}
// test/Parse/use_public_var_private_setter.swift
// RUN: %target-build-swift -emit-module -emit-library %S/Inputs/public_var_private_setter.swift
// RUN: %target-build-swift -I . -L . -lpublic_var_private_setter %s -o use_public_var_private_setter

// On Linux, the linker step of this test fails with "Bad value", specifically:
// "hidden symbol `_TFC25public_var_private_setter9BaseClasscfT_S0_' isn't defined".
// XFAIL: linux

import public_var_private_setter

class Class: BaseClass {}

This is the underlying issue behind https://bugs.swift.org/browse/SR-272, and behind this workaround in swift-corelibs-xctest:

/// - Note: FIXME: This property should be readonly, but currently has to be publicly settable due to a
/// toolchain bug on Linux. To ensure compatibility of tests between
/// swift-corelibs-xctest and Apple XCTest, this property should not be modified.
public var name: String

[SR-752] print does not print in some XCTest situations

Previous ID SR-752
Radar None
Original Reporter @drewcrawford
Type Bug
Status Closed
Resolution Won't Do

Attachment: Download

Environment

Linux x64
swift-DEVELOPMENT-SNAPSHOT-2016-02-08-a

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @modocache
Priority Medium

md5: d46b18027b75685e1ff1ab234eddcb5f

Issue Description:

Okay, let me fix this trainwreck of a bug report. In the attached sample project, we have a function that prints:

public func hello() {
    print("Hello world.")
}

We have a test that calls this function, and then crashes:

class FooTests: XCTestCase {
    func testFoo() {
        hello()
        fatalError("'Hello world' will not be printed.")
    }
}

On OSX / Darwin XCTest, we see the print statement (expected behavior):

Test Suite 'All tests' started at 2016-02-24 02:25:10.840
Test Suite 'XCTestRun.xctest' started at 2016-02-24 02:25:10.841
Test Suite 'FooTests' started at 2016-02-24 02:25:10.841
Test Case '-[footests.FooTests testFoo]' started.
Hello world.
fatal error: 'Hello world' will not be printed.: file src/FooTests.swift, line 6

But on Linux (corelibs-xctest), we do not:

Compiling Swift Module 'lib' (1 sources)
Linking Library:  .atllbuild/products/lib.a
Compiling Swift Module 'footests' (2 sources)
Linking executable .atllbuild/products/footests
Test Case 'FooTests.testFoo' started.
fatal error: 'Hello world' will not be printed.: file src/FooTests.swift, line 6

It is not as simple as "print isn't buffered on Linux", because a trivial example of print-then-fatalError prints as expected. So there is something to do with XCTest specifically involved here.

Implementation notes

SwiftPM does not support XCTest in the Feb 8th snapshot, so instead this sample project uses atbuild. Apologies in advance for foisting an unfamiliar build system on you. You can download it here. On OSX, to reproduce it is simply atbuild in the working directory. For Linux, I have replicated my entire environment in Docker, and you can just run docker build . on any system with Docker installed to see the Linux behavior.

On Darwin, atbuild uses Darwin XCTest, builds .xctest bundles, and runs tests with xcrun xctest /path/to/bundle.xctest. On Linux, it builds executables linked with corelibs-xctest and runs them via the system POSIX call.

[SR-2272] Crash while accessing CommandLine.arguments on macOS XCTest test cases

Previous ID SR-2272
Radar None
Original Reporter paulofaria (JIRA User)
Type Bug
Status Closed
Resolution Done
Environment

macOS

Additional Detail from JIRA
Votes 0
Component/s Standard Library, XCTest
Labels Bug, RunTimeCrash
Assignee None
Priority Medium

md5: a3168b64f3cb1c06e8605f267e75f494

Issue Description:

If you try to access CommandLine.arguments or CommandLine.unsafeArgv inside an XCTest test case you get nice crash:

fatal error: unexpectedly found nil while unwrapping an Optional value
Current stack trace:
0 libswiftCore.dylib 0x000000010d6e0370 swift_reportError + 125
1 libswiftCore.dylib 0x000000010d6fcf90 _swift_stdlib_reportFatalError + 61
2 libswiftCore.dylib 0x000000010d6a9fe0 partial apply for thunk + 63
3 libswiftCore.dylib 0x000000010d503c50 specialized StaticString.withUTF8Buffer<A> (invoke : (UnsafeBufferPointer<UInt8>) -> A) -> A + 351
4 libswiftCore.dylib 0x000000010d672320 partial apply for (_fatalErrorMessage(StaticString, StaticString, StaticString, UInt, flags : UInt32) -> ()).(closure #2) + 158
5 libswiftCore.dylib 0x000000010d6a9fe0 partial apply for thunk + 63
6 libswiftCore.dylib 0x000000010d6afe20 partial apply for thunk + 14
7 libswiftCore.dylib 0x000000010d503c50 specialized StaticString.withUTF8Buffer<A> (invoke : (UnsafeBufferPointer<UInt8>) -> A) -> A + 351
8 libswiftCore.dylib 0x000000010d632b40 specialized _fatalErrorMessage(StaticString, StaticString, StaticString, UInt, flags : UInt32) -> () + 143
9 libswiftCore.dylib 0x000000010d671a00 specialized static Process._computeArguments() -> [String] + 1071
10 libswiftCore.dylib 0x000000010d624850 static Process.arguments.getter + 149
11 QuarkTests 0x000000010cc612a0 loadConfiguration(configurationFile : String) throws -> StructuredData + 755
12 QuarkTests 0x000000010cc606d0 configure<A where ...> (configurationFile : String, server : ConfigurableServer.Type, configure : (A) throws -> ResponderRepresentable) -> () + 169
13 QuarkTests 0x000000010cc5bc70 QuarkTests.testConfiguration() throws -> () + 495
14 QuarkTests 0x000000010cc5c7e0 @objc QuarkTests.testConfiguration() throws -> () + 50
15 CoreFoundation 0x00007fff85dde9e0 invoking_ + 140
16 CoreFoundation 0x00007fff85dde7e0 -[NSInvocation invoke] + 286
17 XCTest 0x0000000109c62d7e __24-[XCTestCase invokeTest]_block_invoke_2 + 481
18 XCTest 0x0000000109c9bc54 -[XCTestContext performInScope:] + 190
19 XCTest 0x0000000109c62c6c -[XCTestCase invokeTest] + 255
20 XCTest 0x0000000109c6340e -[XCTestCase performTest:] + 457
21 XCTest 0x0000000109c608ac -[XCTestSuite performTest:] + 447
22 XCTest 0x0000000109c608ac -[XCTestSuite performTest:] + 447
23 XCTest 0x0000000109c608ac -[XCTestSuite performTest:] + 447
24 XCTest 0x0000000109c74bc1 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 602
25 XCTest 0x0000000109c9cf0f _XCTestMain + 1050
26 xctest 0x0000000109c4374b <unavailable> + 0
27 libdyld.dylib 0x00007fff9772f5ac start + 1

For some reason _NSGetArgv is returning NULL here:

https://github.com/apple/swift/blob/fe932663a8b9119636d8e61711df733f3c9f703e/stdlib/public/stubs/CommandLine.cpp#L58

That behaviour only happens on Linux.
I protected from the crash by doing

var commandLineArguments: [String] {
    return CommandLine.argc > 0 ? CommandLine.arguments : []
}

Maybe the same logic could be applied on the macOS shims. If this is the right direction I can send a PR with the fix. Another option would be to inspect exactly why XCTest is providing no command line arguments. But this would be much harder to do. Going the XCTest route or not I think crash is no good, so it should be protected somehow.

I think the correct behaviour in cases of undefined arguments:

CommandLine.argc = 0
CommandLine.unsafeArgv = nil
CommandLine.arguments = []

@belkadan

[SR-3615] Corelibs XCTest does not re-export Foundation

Previous ID SR-3615
Radar rdar://problem/29929469
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: 6a7740aee2b80266d4aa43519adad590

Issue Description:

An internal engineer has reported that he keeps stubbing toes on a frustrating inconsistency between macOS and Linux:
= On macOS, you can use Foundation data types (such as `Data`) in an XCTest unit test without `import Foundation`
= On Linux, using Foundation data types in unit tests does require `import Foundation`

To provide consistency, Corelibs XCTest should re-export Foundation.

[SR-2332] XCTest test failure: Asynchronous/Predicates/Expectations/main.swift

Previous ID SR-2332
Radar None
Original Reporter @parkera
Type Bug
Status Closed
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: 5cdc557600037f41dc79c21276ad40be

Issue Description:

These tests have been failing for some time, so I disabled them in order to unblock other work. We need to investigate what the root cause of the failure is.

[SR-1589] Add `testCase` variant taking non-throwing functions

Previous ID SR-1589
Radar None
Original Reporter @ddunbar
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @ddunbar
Priority Medium

md5: c858676023ccc3bab722fe07f764cb24

Issue Description:

I would like to write Linux tests as:

class FooTests: XCTestCase {
  func testFoo() {}
  // Notice use of type inference to avoid boilerplate.
  static var allTests = {
    ("testFoo", testFoo)
  }()
}

but type inference will fail with something like:

swiftpm/Tests/Basic/XCTestManifests.swift:17:28: error: cannot convert value of type '[(String, JSONTests -> () -> ())]' to expected argument type '[(String, (XCTestCase) -> () throws -> Void)\
]'
        testCase(JSONTests.allTests),
                 ~~~~~~~~~~^~~~~~~~

It would be nice if Linux XCTest would accept throwing and non-throwing variants of the testCase function.

[SR-1210] Crash in libobjc when class's first stored property's type is not Objective-C compatible

Previous ID SR-1210
Radar rdar://25330746
Original Reporter jscampbell05 (JIRA User)
Type Bug
Status Resolved
Resolution Duplicate
Additional Detail from JIRA
Votes 0
Component/s Compiler
Labels Bug, Runtime, XCTest
Assignee None
Priority Medium

md5: b213bcd9c5df78252fb449d45383a46e

duplicates:

  • SR-1055 Crash in libobjc when class's first stored property's type is not Objective-C compatible

Issue Description:

It seems that there is a case where https://bugs.swift.org/browse/SR-1055 can still occur.

In the final version of Swift 2.2 and Xcode 7.3 I still have a crash if the first property of a class is a enum.

In my class it finished with some lazy properties which held a class, moving these before the enum property fixed the issue. This crash occurred whilst running my Unit Tests and the XCTest framework was searching for subclasses in the runtime in order to detect the tests to run.

[SR-308] [XCTAssert] API incompatibility due to usage of StaticString

Previous ID SR-308
Radar None
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: 04b054794f20358bffdcb2afab563c6d

Issue Description:

Xcode's XCTest has the following signature for XCTAssert:

public func XCTAssert(@autoclosure expression: () -> BooleanType, _ message: String = default, file: String = default, line: UInt = default)

while swift-corelibs-xctest has:

public func XCTAssert(@autoclosure expression: () -> BooleanType, _ message: String = "", file: StaticString = __FILE__, line: UInt = __LINE__)

Note the different types for the `file` parameter. Due to this inconsistency, custom assertion functions which capture file+line from their caller and then delegate to XCTAssert* don't compile on both platforms without resorting to a conditionally-compiled typealias. e.g.:

#if os(Linux)
typealias FileString = StaticString
#else
typealias FileString = String
#endif

func assertAwesomeness(string: String, file: FileString = __FILE__, line: UInt = __LINE__) {
  XCTAssertEqual(string, "Awesome", file: file, line: line)
}

Ideally, the typealias wouldn't be necessary.

[SR-5262] XCTest should have a way to validate assertions (e.g. `assert()`)

Previous ID SR-5262
Radar None
Original Reporter @phausler
Type New Feature
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels New Feature
Assignee None
Priority Medium

md5: 45492463837a5115b54a5e4cdffbc273

Issue Description:

Often times when testing code it is helpful to validate assertion cases that have preconditions or fatalErrors. A method that would aide in this endeavor would be something along the lines of XCTAssertIsFatal where a block of code is passed in that is expected to trigger a fatalError.

Normally this would mean that the code would crash. However if a signal handler for SIGILL is installed we can validate if an assertion is tripped.

See https://github.com/phausler/swift-corelibs-xctest/tree/assert_fatal for a rough draft at handling fatal assertions (note this only works in a non attached debugger mode and has only been written for Darwin, however a linux port should be trivial)

[SR-981] Can't build swift-corelibs-xctest on OS X

Previous ID SR-981
Radar None
Original Reporter jonallured (JIRA User)
Type Improvement
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement
Assignee @modocache
Priority Medium

md5: 4ddc1b17a97237b45b44714d210fa96f

Issue Description:

I was able to get some toolchains installed and then use them when launching XCode, but then when I go to actually build and run the SwiftXCTestFunctionalTests scheme, I get this error:

error: can't exec '/Users/jon/code/swift-corelibs-xctest/../llvm/utils/lit/lit.py' (No such file or directory)

And it's true, that file doesn't exist, but I'm not sure what to do about it.

Thanks!
Jon

[SR-5986] `swift test` induces an error "undefined reference to 'objc_retainAutoreleasedReturnValue'" when calls `CFStringConvertEncodingToIANACharSetName(_: )` on Linux

Previous ID SR-5986
Radar None
Original Reporter @YOCKOW
Type Bug
Status Resolved
Resolution Done
Environment
  • OS: Ubuntu 16.04

  • Swift 4.0

Additional Detail from JIRA
Votes 1
Component/s Compiler, XCTest
Labels Bug, CoreFoundation
Assignee None
Priority Medium

md5: 60bfed75965db16b3167706f3c79fc42

relates to:

  • SR-5271 Don't emit objc_retainAutoreleasedReturnValue() on non-ObjC targets

Issue Description:

[How to reproduce the error] (shell on Linux)

$ mkdir cf-swift
$ cd cf-swift
$ swift package init --type library
$ cat <<EOF >> Sources/cf-swift/cf_swift.swift
> import CoreFoundation
> import Foundation
> struct Dummy {
>   static let string: String? = {
>     let cfEncoding : CFStringEncoding = CFStringConvertNSStringEncodingToEncoding(String.Encoding.utf8.rawValue)
>     guard let _ = CFStringConvertEncodingToIANACharSetName(cfEncoding) else { return nil }
>     return nil
>   }()
> }
> EOF
$ swift build
$ swift test

[Result]
`swift build` ends without any trouble.
`swift test` views error messages below:

Compile Swift Module 'cf_swiftTests' (1 sources)
Compile Swift Module 'cf_swiftPackageTests' (1 sources)
Linking ./.build/x86_64-unknown-linux/debug/cf-swiftPackageTests.xctest
/*/*/cf-swift/Sources/cf-swift/cf_swift.swift:9: error: undefined reference to 'objc_retainAutoreleasedReturnValue'
clang: error: linker command failed with exit code 1 (use -v to see invocation)
<unknown>:0: error: link command failed with exit code 1 (use -v to see invocation)
error: terminated(1): /opt/Swift-4.0/bin/swift-build-tool -f /*/*/cf-swift/.build/debug.yaml test

[SR-5247] Add the new spellings for the *WithAccuracy assertion functions

Previous ID SR-5247
Radar rdar://problem/32828908
Original Reporter @briancroom
Type Improvement
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement
Assignee @briancroom
Priority Medium

md5: 0c195bc6b88550abf01cdc1a06ea5065

Issue Description:

This is to bring Corelibs XCTest in line with changes made in the main XCTest Swift overlay for Xcode 9.

XCTAssertEqualWithAccuracy and XCTAssertNotEqualWithAccuracy are now deprecated in favor of XCTAssertEqual(::accuracy:file:line๐Ÿ™‚ and XCTAssertNotEqual(::accuracy:file:line๐Ÿ™‚.

This fixes an inconsistency where XCTAssertEqualWithAccuracy had accuracy: as a named parameter, but XCTAssertNotEqualWithAccuracy had it unnamed.

[SR-1215] Provide both Swift 2.2 and Swift 3 APIs for corelibs-xctest?

Previous ID SR-1215
Radar None
Original Reporter @modocache
Type New Feature
Status Closed
Resolution Won't Do
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels New Feature
Assignee @modocache
Priority Medium

md5: c552ac518ced1233d8124027dfeee90b

Issue Description:

The current APIs match Swift 3, but not Swift 2.2. For early adopters, this causes some pain: Kitura/Kitura#349 (comment)

Perhaps we should provide both APIs for the time being? Or could we check the current version of the Swift compiler being used, and provide either 2.2-/3.0-compatible APIs?

[SR-519] XCTest reports the total execution of tests incorrectly

Previous ID SR-519
Radar None
Original Reporter @phausler
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @modocache
Priority Medium

md5: b4622ce5b5968ca8923394d0f8cf9a24

Issue Description:

When running all of the Foundation tests it reports on mac os x:
Total executed 337 tests, with 0 failures (0 unexpected) in 0.022 (7.436) seconds
and linux reports:
Total executed 0 tests, with 0 failures (0 unexpected) in 0.0 (11.739) seconds

[SR-1063] Have XCTestObservationCenter conform to new Swift 3 naming rules

Previous ID SR-1063
Radar None
Original Reporter @modocache
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, StarterBug
Assignee shingt (JIRA)
Priority Medium

md5: da16a265f7fa73b0cb4ebaf710ed7ada

Issue Description:

swift-corelibs-xctest's XCTestObservationCenter defines a method sharedTestObservationCenter. However, that method in Apple XCTest is now named simply shared.

swift-corelibs-xctest's XCTestObservationCenter should define shared as well, since our goal is to have an identical API. Change the method name and update the tests.

(I wonder if there are other discrepancies brought about by the Swift 3 naming rules? We should investigate.)

[SR-2434] [XCTest] Functional tests fail to run due to stdlib versioning problem

Previous ID SR-2434
Radar None
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: 1d18cf777b86c9c663642f1835487a8b

Issue Description:

When run on OS X, the binaries that XCTest uses to test itself aren't loading properly, with the following error being output:

dyld: Library not loaded: @rpath/libswiftCore.dylib
  Referenced from: /Users/buildnode/jenkins/workspace/swift-corelibs-xctest-PR-osx/Ninja-DebugAssert/xctest-macosx-x86_64/Debug/SwiftXCTest.framework/Versions/A/Frameworks/libswiftDarwin.dylib
  Reason: Incompatible library version: libswiftDarwin.dylib requires version 1.0.0 or later, but libswiftCore.dylib provides version 0.0.0

The underlying cause of this is unclear to me, but it can be observed on PR CI builds starting with #83, continuing to the latest ones. #82 exhibit this problem. (Note that malfunctioning builds are still showing up as passing due to SR-2433.)

[SR-334] 0 tests execute in "-nan (0.0) seconds"

Previous ID SR-334
Radar None
Original Reporter @modocache
Type Bug
Status Closed
Resolution Done
Environment

Swift version 2.2-dev (LLVM 46be9ff861, Clang 4deb154edc, Swift 778f82939c)
Target: x86_64-unknown-linux-gnu

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @briancroom
Priority Medium

md5: f9653c64a8b8964f4777d5f3f4fbec56

Issue Description:

The following program:

import XCTest
XCTMain([])

Results in the following output:

Total executed 0 tests, with 0 failures (0 unexpected) in -nan (0.0) seconds

Instead, the output should be the same as when 1 or more tests in executed in under 0.1 seconds:

Total executed 0 tests, with 0 failures (0 unexpected) in 0.0 (0.0) seconds

[SR-32] Find way to make allTests required / audited on OS X

Previous ID SR-32
Radar None
Original Reporter @ddunbar
Type Bug
Status Closed
Resolution Duplicate
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: 093d90aa3adc1cdd0aa21c38ed26efb0

is duplicated by:

  • SR-710 Generate XCTestCaseProvider entries on Linux

Issue Description:

We currently have to add allTests entries for each test to work around the lack of auto-discovery on Linux. This is a source of error when adding new tests and developing on OS X as it is easy to forget.

We should see if we can use the method discovery features on OS X to verify that the allTests method has been implemented and contains all of the available tests. This would ensure that it isn't possible to add new tests and forget to update the list.

[SR-8258] Implement XCTAttachment API in corelibs-xctest

Previous ID SR-8258
Radar None
Original Reporter @aaronsky
Type New Feature
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels New Feature
Assignee None
Priority Medium

md5: 91afb2975bf54ea57474a2d05893a54e

Issue Description:

I was trying to migrate some integration testing code so it could be Linux compatible. I was surprised to discover that the open source corelibs-xctest does not implement XCTAttachment nor XCTActivity. I'm sure there's some sort of technical or procedural issue that I'm not aware of that would explain this omission, but it surprised me nonetheless.

I've worked around it in a rudimentary way for my purposes, so this isn't blocking, but I'd be interested in seeing some of these less commonly-used XCTest features come to Linux.

[SR-1355] [XCTest] Support XCTest performance testing API

Previous ID SR-1355
Radar None
Original Reporter @ddunbar
Type New Feature
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels New Feature
Assignee @briancroom
Priority Medium

md5: 14030175159ae9057e04dccb414023e0

relates to:

  • SR-1354 [SwiftPM] Improve support for performance tests

Issue Description:

swift-corelibs-xctest should support the performance testing APIs.

Related to:
https://bugs.swift.org/browse/SR-1354

[SR-1625] gold linker requires LD_LIBRARY_PATH, whereas bfd did not. Why?

Previous ID SR-1625
Radar None
Original Reporter @modocache
Type Bug
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee @ddunbar
Priority Medium

md5: 8834d6092b4b0cde78eaeff0ec184aa4

relates to:

  • SR-1631 swift-integration-tests failing, unable to find libFoundation.so

Issue Description:

See discussion in #118 For some reason, swiftlang/swift#2609 required an extra LD_LIBRARY_PATH to be specified when running the XCTest test suite. Why is that? And can we do this in a "nicer" way?

[SR-1164] XCTestCase testRunClass should be of type XCTestRun.Type in Swift

Previous ID SR-1164
Radar rdar://25568830
Original Reporter @modocache
Type Improvement
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, ClangImporter, SDKOverlay
Assignee None
Priority Medium

md5: 8adee259f35799e321fb96cfd32cf201

Issue Description:

See: #86 (comment)

XCTest defines XCTestCase.testRunClass as an AnyClass?, and checks at runtime that this is a subclass of XCTestRun. It would be nicer if this was exported as XCTestRun.Type or XCTestRun.Type?.

Radar:

[SR-1417] XCTAssertEqual confusingly mentions Optionals on failure

Previous ID SR-1417
Radar None
Original Reporter @nsalmoria
Type Bug
Status Resolved
Resolution Done
Environment

Swift Development Snapshot 2016-05-03 (a), Xcode 7.3.1

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, SDKOverlay, StarterBug, XCTest
Assignee @nsalmoria
Priority Medium

md5: 2a54820de78ceab6bbb23475c5f7f798

Issue Description:

When using XCTAssertEqual() to test that two values are equal, the error message in case of failure always assumes that the two values are Optional.

For example, this line:

XCTAssertEqual(1, 2)

prints:

XCTAssertEqual failed: ("Optional(1)") is not equal to ("Optional(2)")

This is confusing since the values passed to XCTAsserEqual are not Optionals, and adds visual noise to the error report.

It would be expected that the error should simply be

XCTAssertEqual failed: ("1") is not equal to ("2")

[SR-6183] Use CMake for building Corelibs XCTest

Previous ID SR-6183
Radar None
Original Reporter @briancroom
Type Task
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Task, Linux
Assignee None
Priority Medium

md5: 55b6c9013e4d23dd5c8eb5cbe99605d3

Issue Description:

While the long-term goal is still to make SwiftPM the primary way to build Corelibs XCTest, in the meantime, it would be great for the project to get incremental builds instead of always rebuilding the entire library as happens currently. It seems that adopting CMake would be the most pragmatic way to achieve this, and would be in keeping with other Swift.org projects. (e.g. libdispatch which recently switched over to CMake.)

[SR-6249] Asynchronous testing missing features

Previous ID SR-6249
Radar None
Original Reporter @Bouke
Type Bug
Status Resolved
Resolution Duplicate
Environment

Ubuntu 14.04
swift-4.0-RELEASE

Additional Detail from JIRA
Votes 1
Component/s XCTest
Labels Bug
Assignee @stmontgomery
Priority Medium

md5: a019a076a8d7b6b1c92f11d2c6962ab1

duplicates:

  • SR-7615 swift-corelibs-xctest missing XCTWaiter

Issue Description:

The following API / features on XCTestExpectation are missing:

class XCTestExpectation {
    var expectationDescription: String { get set }
    var expectedFulfillmentCount: Int { get set }
    var assertForOverFulfill: Bool { get set }
    var isInverted: Bool { get set }
}

The following API / features are missing on XCTest:

class XCTest {
    func wait(for: [XCTestExpectation], timeout: TimeInterval)
    func wait(for: [XCTestExpectation], timeout: TimeInterval, enforceOrder: Bool)
}

There might be more, but my test cases broke on the linux build because of these.

[SR-1145] XCTestSuite doc comments describe behavior exclusive to Apple XCTest

Previous ID SR-1145
Radar None
Original Reporter @briancroom
Type Bug
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, Documentation, StarterBug
Assignee None
Priority Medium

md5: 0ed03ce823738ee5c09d488b11993363

Issue Description:

The current doc comments on `XCTestSuite` are copied verbatim from Apple XCTest headers. They talk about methods and behavior that isn't available (at least yet) on our XCTestSuite implementation. We should rewrite this documentation to be more applicable to the way that the class is used in swift-corelibs-xctest.

Especially in need of attention are the APIs merged in this PR, which added many new classes (XCTestRun etc.) with documentation that doesn't make sense in our context.

A little additional context from @modocache: #89 (comment)

[SR-3701] Performance regression when calling XCTAssert* functions

Previous ID SR-3701
Radar rdar://problem/29644454
Original Reporter @briancroom
Type Bug
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug, Performance
Assignee @briancroom
Priority Medium

md5: 493a2f415843f92671a619059c65ba8b

Issue Description:

@ddunbar has reported that an internal test suite is showing a pretty significant performance regression in a test that calls XCTAssert* functions in a tight loop. The regression has been traced to this commit and appears to be largely due to NSDictionary->Dictionary bridging costs in the _XCTRunThrowableBlockBridge function.

[SR-1048] XCTest build_script.py should build debug/release depending on Swift build script params

Previous ID SR-1048
Radar None
Original Reporter @modocache
Type Improvement
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee jamal (JIRA)
Priority Medium

md5: 27328da1d032e726efe058bc7dc22b33

Issue Description:

swift-corelibs-xctest's build_script.py is able to build the project in either --debug or --release mode:

build_parser.add_argument(
"--release",
help="builds for release",
action="store_const",
dest="build_style",
const="release",
default="debug")
build_parser.add_argument(
"--debug",
help="builds for debug (the default)",
action="store_const",
dest="build_style",
const="debug",
default="debug")

The Swift build script, utils/build-script, also takes parameters to determine the build mode:
https://github.com/apple/swift/blob/e8eba770d58ce512046951b0647941194987525e/utils/build-script#L358-L367

Swift has many build modes: Debug, Release, Release+Assert, etc. This build mode is never reflected in the XCTest build mode, however: https://github.com/apple/swift/blob/e8eba770d58ce512046951b0647941194987525e/utils/build-script-impl#L1937-L1942

It should be! XCTest should be built in release mode (passed the --release flag) when Swift is being built in release mode. It should be built in debug mode when Swift is built in debug mode.

(It might also be worth looking into the other Swift build modes, such as when assertions are turned on, and determine whether we should reflect that in how XCTest is built. But that's outside the scope of this task.)

[SR-1872] Use runloop source in XCTestCase.waitForExpectations(withTimeout:file:line:handler:)

Previous ID SR-1872
Radar None
Original Reporter @modocache
Type Improvement
Status In Progress
Resolution
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee pranavShenoy95 (JIRA)
Priority Medium

md5: 2f80d90c1c1340fe3c193450551b82f7

Issue Description:

See this FIXME:

"Instead of polling the expectations to check whether they've been fulfilled, it would be more efficient to use a runloop source that can be signaled to wake up when an expectation is fulfilled."

Prerequisites:

  • A development environment capable of building Swift from source.

Helpful skills:
Familiarity with Swift, XCTest, and Foundation's NSRunLoop.

Starting point:

  1. Build Swift from source by following the instructions in the Swift README.

  2. Build and run the swift-corelibs-xctest tests by following the instructions in the swift-corelibs-xctest README. Confirm that all the tests pass.

  3. Modify the source code in XCTestCase+Asynchronous.swift to use a runloop source. The Apple documentation on run loops might be helpful.

  4. Run the swift-corelibs-xctest tests again. Confirm they all still pass. If they do, send a pull request!

[SR-5954] Assertion failure in void _XCTFailureHandler when running unit tests on Swift 4

Previous ID SR-5954
Radar None
Original Reporter ylin (JIRA User)
Type Bug
Environment

Xcode 9 from App Store, Swift 4.0

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: 3490a89fea916f1631607373b1b22a51

Issue Description:

I have Xcode 9 downloaded from the App Store. Here is how to reproduce the error:

1. Clone https://github.com/IBM-Swift/Configuration
2. Run unit tests using a Swift 4 toolchain (change `.swift-version` if you are using `swiftenv`): `swift test`
3. Get assertion failures in the logs, like this:

Test Suite 'All tests' started at 2017-09-21 10:53:21.238
Test Suite 'ConfigurationPackageTests.xctest' started at 2017-09-21 10:53:21.238
Test Suite 'ConfigurationManagerTest' started at 2017-09-21 10:53:21.238
2017-09-21 10:53:21.266 xctest[7061:67158] *** Assertion failure in void _XCTFailureHandler(XCTestCase * _Nonnull, BOOL, const char * _Nonnull, NSUInteger, NSString * _Nonnull, NSString * _Nullable, ...)(), /Library/Caches/com.apple.xbs/Sources/XCTest/XCTest-13201/Sources/XCTestFramework/Core/XCTestAssertionsImpl.m:41
<unknown>:0: error: ConfigurationManagerTest : Parameter "test" must not be nil.
(
    0   CoreFoundation                      0x00007fff84fdc57b __exceptionPreprocess + 171
    1   libobjc.A.dylib                     0x00007fff9a23d1da objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff84fe1132 +[NSException raise:format:arguments:] + 98
    3   Foundation                          0x00007fff86ab0da6 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 166
    4   XCTest                              0x000000010d9ce687 _XCTFailureHandler + 583
    5   XCTest                              0x000000010d9ce923 _XCTPreformattedFailureHandler + 81
    6   libswiftXCTest.dylib                0x00000001115bf019 _T06XCTest7XCTFailySS_s12StaticStringV4fileSu4linetFTf4gXxn_n + 473
    7   libswiftXCTest.dylib                0x00000001115b9b66 _T06XCTest7XCTFailySS_s12StaticStringV4fileSu4linetF + 22
    8   ConfigurationPackageTests           0x000000010e16e504 _T018ConfigurationTests0A11ManagerTestC5setUpyyFZ + 1396
    9   ConfigurationPackageTests           0x000000010e16e568 _T018ConfigurationTests0A11ManagerTestC5setUpyyFZTo + 24
    10  XCTest                              0x000000010d9c1376 __27-[XCTestSuite performTest:]_block_invoke + 58
    11  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    12  XCTest                              0x000000010d9c100b -[XCTestSuite performTest:] + 239
    13  XCTest                              0x000000010d9c14a7 __27-[XCTestSuite performTest:]_block_invoke + 363
    14  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    15  XCTest                              0x000000010d9c100b -[XCTestSuite performTest:] + 239
    16  XCTest                              0x000000010d9c14a7 __27-[XCTestSuite performTest:]_block_invoke + 363
    17  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    18  XCTest                              0x000000010d9c100b -[XCTestSuite performTest:] + 239
    19  XCTest                              0x000000010da1fb9d __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 40
    20  XCTest                              0x000000010d9dc2e2 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 477
    21  XCTest                              0x000000010da1fa3b -[XCTTestRunSession runTestsAndReturnError:] + 281
    22  XCTest                              0x000000010d9b10b5 -[XCTestDriver runTestsAndReturnError:] + 314
    23  XCTest                              0x000000010da10250 _XCTestMain + 833
    24  xctest                              0x000000010d94e3d2 xctest + 9170
    25  libdyld.dylib                       0x00007fff9ab1e235 start + 1
)
2017-09-21 10:53:21.275 xctest[7061:67158] *** Assertion failure in void _XCTFailureHandler(XCTestCase * _Nonnull, BOOL, const char * _Nonnull, NSUInteger, NSString * _Nonnull, NSString * _Nullable, ...)(), /Library/Caches/com.apple.xbs/Sources/XCTest/XCTest-13201/Sources/XCTestFramework/Core/XCTestAssertionsImpl.m:41
<unknown>:0: error: ConfigurationManagerTest : Parameter "test" must not be nil.
(
    0   CoreFoundation                      0x00007fff84fdc57b __exceptionPreprocess + 171
    1   libobjc.A.dylib                     0x00007fff9a23d1da objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff84fe1132 +[NSException raise:format:arguments:] + 98
    3   Foundation                          0x00007fff86ab0da6 -[NSAssertionHandler handleFailureInFunction:file:lineNumber:description:] + 166
    4   XCTest                              0x000000010d9ce687 _XCTFailureHandler + 583
    5   XCTest                              0x000000010d9ce923 _XCTPreformattedFailureHandler + 81
    6   libswiftXCTest.dylib                0x00000001115bf019 _T06XCTest7XCTFailySS_s12StaticStringV4fileSu4linetFTf4gXxn_n + 473
    7   libswiftXCTest.dylib                0x00000001115b9b66 _T06XCTest7XCTFailySS_s12StaticStringV4fileSu4linetF + 22
    8   ConfigurationPackageTests           0x000000010e16ea44 _T018ConfigurationTests0A11ManagerTestC8tearDownyyFZ + 1236
    9   ConfigurationPackageTests           0x000000010e16eaa8 _T018ConfigurationTests0A11ManagerTestC8tearDownyyFZTo + 24
    10  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    11  XCTest                              0x000000010d9c1050 -[XCTestSuite performTest:] + 308
    12  XCTest                              0x000000010d9c14a7 __27-[XCTestSuite performTest:]_block_invoke + 363
    13  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    14  XCTest                              0x000000010d9c100b -[XCTestSuite performTest:] + 239
    15  XCTest                              0x000000010d9c14a7 __27-[XCTestSuite performTest:]_block_invoke + 363
    16  XCTest                              0x000000010d9c0e0e -[XCTestSuite _performProtectedSectionForTest:testSection:] + 26
    17  XCTest                              0x000000010d9c100b -[XCTestSuite performTest:] + 239
    18  XCTest                              0x000000010da1fb9d __44-[XCTTestRunSession runTestsAndReturnError:]_block_invoke + 40
    19  XCTest                              0x000000010d9dc2e2 -[XCTestObservationCenter _observeTestExecutionForBlock:] + 477
    20  XCTest                              0x000000010da1fa3b -[XCTTestRunSession runTestsAndReturnError:] + 281
    21  XCTest                              0x000000010d9b10b5 -[XCTestDriver runTestsAndReturnError:] + 314
    22  XCTest                              0x000000010da10250 _XCTestMain + 833
    23  xctest                              0x000000010d94e3d2 xctest + 9170
    24  libdyld.dylib                       0x00007fff9ab1e235 start + 1
)
Test Suite 'ConfigurationManagerTest' failed at 2017-09-21 10:53:21.276.
     Executed 0 tests, with 2 failures (2 unexpected) in 0.000 (0.038) seconds
Test Suite 'ConfigurationNodeTest' started at 2017-09-21 10:53:21.276
Test Case '-[ConfigurationTests.ConfigurationNodeTest testMergeOverwrite]' started.
Test Case '-[ConfigurationTests.ConfigurationNodeTest testMergeOverwrite]' passed (0.198 seconds).
Test Case '-[ConfigurationTests.ConfigurationNodeTest testRawValue]' started.
Test Case '-[ConfigurationTests.ConfigurationNodeTest testRawValue]' passed (0.001 seconds).
Test Case '-[ConfigurationTests.ConfigurationNodeTest testSplitKeys]' started.
Test Case '-[ConfigurationTests.ConfigurationNodeTest testSplitKeys]' passed (0.000 seconds).
Test Case '-[ConfigurationTests.ConfigurationNodeTest testSubscript]' started.
Test Case '-[ConfigurationTests.ConfigurationNodeTest testSubscript]' passed (0.001 seconds).
Test Suite 'ConfigurationNodeTest' passed at 2017-09-21 10:53:21.476.
     Executed 4 tests, with 0 failures (0 unexpected) in 0.200 (0.200) seconds
Test Suite 'ConfigurationPackageTests.xctest' failed at 2017-09-21 10:53:21.476.
     Executed 4 tests, with 2 failures (2 unexpected) in 0.200 (0.238) seconds
Test Suite 'All tests' failed at 2017-09-21 10:53:21.476.
     Executed 4 tests, with 2 failures (2 unexpected) in 0.200 (0.239) seconds

The tests will complete without this runtime failure if I comment out `setup` and `teardown` functions in `ConfigurationManagerTest.swift`. Those functions deal with creating and deleting symlinks inside `/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/Library/Xcode/Agents/`, so it's entirely possibly that I'm running into a permissions issue. However, that is not at all clear from the assertion failure printed.

[SR-4218] [String] Non-empty strings compare equal to empty strings

Previous ID SR-4218
Radar None
Original Reporter @tbkka
Type Bug
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s Standard Library, XCTest
Labels Bug, Linux
Assignee None
Priority Medium

md5: e2cfdb4ceda2c28163bf20fcffb6441d

Issue Description:

The following test succeeds on macOS, fails on Linux:

   XCTAssertNotEqual("", "\u{01}")

I believe it should succeed on all platforms.

I've seen this with the 3.0.2 release version on Ubuntu 16.04, the 3.1 snapshot from March 8 on Ubuntu 16.10, and the trunk snapshot from March 9 on Ubuntu 16.10.

[SR-1046] XCTest build_script.py should use the same lit executable as Swift utils/build-script

Previous ID SR-1046
Radar None
Original Reporter @modocache
Type Improvement
Status Resolved
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee [email protected] (JIRA)
Priority Medium

md5: ae21cddaa2a59d35e7f4efd58604c6f4

Issue Description:

swift-corelibs-xctest uses the LLVM lit test runner to compile test cases and run them. The Swift build script, utils/build-script, does the same. The Swift build script already contains logic to find a lit executable. XCTest should use the same lit executable.

XCTest's build_script.py has a "test" subcommand that takes a "--lit" path argument (added in #76 A modification should be made to the Swift build script to pass something like the following to the Linux XCTest build: https://github.com/apple/swift/blob/723559526475551de6c8211961c7f355279b85ef/utils/build-script-impl#L1763.

Here's where the Linux tests for XCTest are kicked off: https://github.com/apple/swift/blob/e8eba770d58ce512046951b0647941194987525e/utils/build-script-impl#L2203

[SR-1047] Run xctest_checker.py tests as part of XCTest tests

Previous ID SR-1047
Radar None
Original Reporter @modocache
Type Improvement
Status Closed
Resolution Done
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee t_martinho (JIRA)
Priority Medium

md5: 3e4c771b76e6529a0d1d3fb05cab6efb

Issue Description:

swift-corelibs-xctest uses a Python program called xctest_checker to run its functional test suite. That program itself has unit tests, written in Python.

These unit tests should be run as part of the greater XCTest test suite. This is probably possible by modifying the lit.cfg in the swift-corelibs-xctest project such that it picks up on a file with a RUN: directive that triggers the Python tests. See apple/swift#778 for an example.

See llvm.org/docs/CommandGuide/lit.html for additional details on lit--it's the test runner for Swift, LLVM, and other projects, so it's nice to know about it.

[SR-875] Make swift-corelibs-xctest Functional tests regex matching more like FileCheck

Previous ID SR-875
Radar None
Original Reporter @modocache
Type Improvement
Status Closed
Resolution Won't Do
Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Improvement, StarterBug
Assignee @modocache
Priority Medium

md5: 36d197324a801db0dd6d5b0d341b6a3e

Issue Description:

swift-corelibs-xctest includes a functional test suite that uses the lit test runner and a Python program that matches output line by line. Here's an example.

Notice that the parentheses in the CHECK lines need to be escaped. This is because each line is interpreted as a regular expression. This makes writing CHECK lines very cumbersome and error-prone.

On the other hand, FileCheck, a matching program used by most of the LLVM and Swift test suites, does not interpret CHECK lines as regexes. Instead, parts of the line may be interpreted as a regex using a special pattern, such as {{.*}}.

The Python program used to match output in corelibs-xctest should use a similar mechanism: regexes should delineated, instead of having the entire line be interpreted as a regex. For example, the CHECK lines in the test linked above should become:

// CHECK: Test Case 'SingleFailingTestCase.test_fails' started.
// CHECK: {{.*}}/Tests/Functional/SingleFailingTestCase/main.swift:24: error: SingleFailingTestCase.test_fails : XCTAssertTrue failed - 
// CHECK: Test Case 'SingleFailingTestCase.test_fails' failed ({{\d+}}.{{\d+}} seconds).
// CHECK: Executed 1 test, with 1 failure (0 unexpected) in {{\d+}}.{{\d+}} ({{\d+}}.{{\d+}}) seconds
// CHECK: Total executed 1 test, with 1 failure (0 unexpected) in {{\d+}}.{{\d+}} ({{\d+}}.{{\d+}}) seconds

(Of course, even better would be to have corelibs-xctest use FileCheck directly, but that program is only available as part of a full LLVM build, and we don't want to introduce such a dependency.)

[SR-2284] Cannot invoke 'XCTAssertEqual' with an argument list of type '([Array<Int>], [[Int]])'

Previous ID SR-2284
Radar None
Original Reporter @masters3d
Type Bug
Status Resolved
Resolution Done

Attachment: Download

Environment

swift-DEVELOPMENT-SNAPSHOT-2016-08-04-a-osx

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels Bug
Assignee None
Priority Medium

md5: dffa2b0d994403b9420f7549454b2423

is duplicated by:

  • SR-5494 can't compare multidimensional arrays for equality in Swift 4

relates to:

  • SR-894 Foundation is able to compare multidimensional arrays in OS X but not in Linux
  • SR-1535 No == overload exists for comparing optional arrays.

Issue Description:

Cannot invoke 'XCTAssertEqual' with an argument list of type '([Array<Int>], [[Int]])'

import Cocoa
import XCTest

struct Row{
    static var rows:[[Int]] {
        return [[1]]
    }
}

class RowTests: XCTestCase {
    
    func testOneRow() {
        
        XCTAssertEqual([[1]], Row.rows)
}
}

[SR-1110] Add convenience XCTestExpectation constructors for NSPredicate

Previous ID SR-1110
Radar None
Original Reporter @modocache
Type New Feature
Status Closed
Resolution Done

Attachment: Download

Additional Detail from JIRA
Votes 0
Component/s XCTest
Labels New Feature, StarterBug
Assignee apps.yoon (JIRA)
Priority Medium

md5: 80de972c44d4ad944e39eb5cd68136b2

Issue Description:

Apple XCTest offers several convenient methods for constructing XCTestExpectations:

@interface XCTestCase : XCTest
// ...
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath expectedValue:(nullable id)expectedValue;
- (XCTestExpectation *)keyValueObservingExpectationForObject:(id)objectToObserve keyPath:(NSString *)keyPath handler:(nullable XCKeyValueObservingExpectationHandler)handler;
- (XCTestExpectation *)expectationForNotification:(NSString *)notificationName object:(nullable id)objectToObserve handler:(nullable XCNotificationExpectationHandler)handler;
- (XCTestExpectation *)expectationForPredicate:(NSPredicate *)predicate evaluatedWithObject:(id)object handler:(nullable XCPredicateExpectationHandler)handler;
// ...
@end

Key-value observing doesn't exist in native Swift, but NSNotification and NSPredicate do (thanks to swift-corelibs-foundation). swift-corelibs-xctest already defines a constructor for NSNotification: https://github.com/yoonapps/swift-corelibs-xctest/blob/c2a9258c7c17e32bf9082709ef111e59aba86a95/Sources/XCTest/XCTestCase.swift#L324. This was added in #85

Add the constructor for NSPredicate: -[XCTestCase expectationForPredicate:evaluatedWithObject:handler:].

Also, note that you'll need to add the following closure typealias as well: typedef BOOL (^XCPredicateExpectationHandler)();.

https://bugs.swift.org/browse/SR-907 is a great reference on how to approach this task.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.