An early look at the future of testing with swift-testing

Wednesday, November 8, 2023

A couple of months ago, Stuart Montgomery, a software engineer in the XCTest team at Apple, shared a new macro-based open-source Swift testing library.

The library is called swift-testing and, as its documentation states it is meant to be a proof of concept for a new testing API for Swift that is based on macros and that is integrated into Swift the same way that XCTest is.

For this reason, the library is meant to be short-lived and it is not meant to be used in production or migrated to from XCTest. However, it is a great way to get a glimpse of what the future of testing in Swift might look like, which is exactly what I did this week!

Getting started

The first thing I did was to create a small Swift package that would allow me to write some tests using the new library:

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "SwiftTestingDemo",
    products: [
        .library(
            name: "SwiftTestingDemo",
            targets: ["SwiftTestingDemo"]),
    ],
    dependencies: [
    ],
    targets: [
        .target(
            name: "SwiftTestingDemo",
            swiftSettings: [
                .enableUpcomingFeature("BareSlashRegexLiterals")
            ]
        )
    ]
)

I then added some code that I had from my app QReate which decodes WIFI URL strings into Swift structs, which I thought would be an ideal candidate to try swift-testing on:

import Foundation

struct WifiNetwork {
    let ssid: String
    let password: String
    let security: String?
    let hidden: String?
}

protocol ErrorMonitoring {
    func monitor(_ error: Error)
}

struct WifiParser {
    enum Error: Swift.Error {
        case noMatches
    }

    private let monitoring: ErrorMonitoring

    init(monitoring: ErrorMonitoring) {
        self.monitoring = monitoring
    }

    func parse(wifi: String) throws -> WifiNetwork {
        let regex = /WIFI:S:(?<ssid>[^;]+);(?:T:(?<security>[^;]*);)?P:(?<password>[^;]+);(?:H:(?<hidden>[^;]*);)?;/

        guard let result = try? regex.wholeMatch(in: wifi) else {
            let error = Error.noMatches
            monitoring.monitor(error)
            throw error
        }

        return WifiNetwork(
            ssid: String(result.ssid),
            password: String(result.password),
            security: result.security.map(String.init),
            hidden: result.hidden.map(String.init)
        )
    }
}

Setting up swift-testing

Now that I had code to test, I went ahead and set up the test target to be able to start using swift-testing.

Depending on swift-testing

The first thing I had to do was add swift-testing as a dependency to both the package and the test target:

// swift-tools-version: 5.9
// The swift-tools-version declares the minimum version of Swift required to build this package.

import PackageDescription

let package = Package(
    name: "SwiftTestingDemo",
    platforms: [
        .macOS(.v13), .iOS(.v16), .watchOS(.v9), .tvOS(.v16), .visionOS(.v1)
    ],
    products: [
        // Products define the executables and libraries a package produces, making them visible to other packages.
        .library(
            name: "SwiftTestingDemo",
            targets: ["SwiftTestingDemo"]),
    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-testing.git", branch: "main"),
    ],
    targets: [
        // Targets are the basic building blocks of a package, defining a module or a test suite.
        // Targets can depend on other targets in this package and products from dependencies.
        .target(
            name: "SwiftTestingDemo",
            swiftSettings: [
                .enableUpcomingFeature("BareSlashRegexLiterals")
            ]
        ),
        .testTarget(
            name: "SwiftTestingDemoTests",
            dependencies: ["SwiftTestingDemo", .product(name: "Testing", package: "swift-testing")]),
    ]
)

Running all tests

Due to the way that tests are organised in swift-testing and the fact that it is not integrated into Xcode or SPM yet, you need to create a small XCTestCase subclass that creates a single test method that finds and runs all tests using XCTestScaffold:

import XCTest

final class AllTests: XCTestCase {
    func testAll() async {
        await XCTestScaffold.runAllTests(hostedBy: self)
    }
}

Writing tests

Now that I had set up my test target correctly, I could start writing some tests. Based on my code, I decided to write three separate tests:

1. A test that checks that the parser can parse a valid WIFI string with all fields present.
2. A test that checks that the parser throws an error when the WIFI string is invalid.
3. A test that checks that the parser can parse a valid WIFI string with only the required fields present.

Defining tests

Something that has changed compared to XCTest is how you define unit tests. You no longer need to prefix your functions with test and you can now instead attach the @Test function to any function that you want to be a test:

@Test
func whenParseIsCalledWithAllFields_ThenNetworkIsInitialisedCorrectly() {
    // ...
}

Organising tests

In this new approach to testing, you have a lot more freedom in how you organise your tests. You can either declare them all as global functions or encapsulate them by purpose under either a struct or a final class.

There is also a dedicated Suite macro which you can attach to your structs and allows you to do things such as changing the test suite name and not having to add the @Test macro to methods inside the struct:

@Suite
struct WifiParserTests {
    func whenParseIsCalledWithAllFields_ThenNetworkIsInitialisedCorrectly() {
        // ...
    }
}

From what I can gather by reading the documentation, it seems to be good practice to attach the @Suite macro to structs that group unit tests.

Testing when the WIFI string is not valid

The first test I wanted to add, and that would showcase some of the differences between swift-testing and XCTest, was to verify that the parser throws an error and sends a monitoring event when the WIFI string is not valid:

@Suite
struct WifiParserTests {
    let sut: WifiParser
    let errorMonitoring: SpyErrorMonitoring

    // 1
    init() async throws {
        errorMonitoring = SpyErrorMonitoring()
        sut = WifiParser(monitoring: errorMonitoring)
    }

    func whenParseIsCalledWithWrongString_ThenNoMatchesErrorIsThrownAndMonitoringEventIsSent() {
        // 2
        #expect(throws: WifiParser.Error.noMatches) { try sut.parse(wifi: "") }
        // 3
        #expect(errorMonitoring.capturedErrors.compactMap { $0 as? WifiParser.Error } == [.noMatches])
    }
}

Let's break down the test suite above and understand what is going on:

1. The first thing I did was to set up the system under test (WifiParser) and the test double to spy on the monitoring events (SpyErrorMonitoring). In XCTest, you would usually do this in the setUp method so that the instances are created before each test. However, in swift-testing, since the suite is created before every test is run, it is sufficient to do this in the init method.
2. Use an overload of the #expect macro that allows you to assert that a closure throws a specific error. You can also pass in a type instead of an instance of the error if you don't want to be as specific.
3. Use the #expect macro with a boolean condition to assert that the spy has captured the correct error.

Testing when all test fields are present

The second test I wrote checked that the parser could parse a valid WIFI string with all fields present:

@Suite
struct WifiParserTests {
    func whenParseIsCalledWithAllFields_ThenNetworkIsInitialisedCorrectly() throws {
        let wifi = "WIFI:S:superwificonnection;T:WPA;P:strongpassword;H:YES;;"

        let network = try sut.parse(wifi: wifi)

        #expect(try #require(network.security) == "WPA")
        #expect(try #require(network.hidden) == "YES")
        #expect(network.ssid == "superwificonnection")
        #expect(network.password == "strongpassword")
    }
}

As you can see in the example above and similarly to the way you would write tests in XCTest, you can still declare the methods as throwing (and even async) but, instead of having dedicated XCTAssert methods, you now have a single #expect macro that you can use to assert that a condition is true.

On top of this, and as you can see above where I am unwrapping the optional security and hidden property, instead of using XCTUnwrap, you can now use the throwing #require macro.

Testing when only the required fields are present

The last test I added was to verify that my logic would still work even when the wifi string only had the required fields using only the #expect macro:

@Suite
struct WifiParserTests {
    func whenParseIsCalledWithStringContainingOnlyRequiredFields_ThenCorrectValuesAreReturned() throws {
        let wifi = "WIFI:S:superwificonnection;P:strongpassword;;"

        let network = try sut.parse(wifi: wifi)

        #expect(network.hidden == nil)
        #expect(network.security == nil)
        #expect(network.ssid == "superwificonnection")
        #expect(network.password == "strongpassword")
    }
}

Further reading

The examples above are just a small subset of what you can do with swift-testing. If you're trying the library out and struggling to find out how to do a specific assertion that you would normally do in XCTest, I recommend you check out this migration chart from the documentation that compares both libraries at an assertion level:

There is also a set of examples and more in-depth explanation of the library itself in its documentation, which is hosted on the Swift Package Index site.