jeffh / fox Goto Github PK

View Code? Open in Web Editor NEW

624.0 23.0 70.0 2.18 MB

Property Based Testing Library for Objective-C and Swift. QuickCheck for Apple's Platforms.

Home Page: http://fox-testing.rtfd.org

License: Other

Ruby 0.47% Objective-C 68.84% Objective-C++ 26.54% Swift 2.93% Shell 1.22%

fox's Introduction

Fox

Property Based Testing for Objective-C and Swift. Automatic generation of software tests.

You might have heard of this or similar technologies through the various genres of testing frameworks and/or libraries:

Haskell QuickCheck
Clojure test.check
Erlang QuickCheck (most advanced but commerical)
Property Based Testing
Model Based Testing
Simulation Testing

Fox is a port of test.check for Objective-C. Unlike some ports of QuickCheck, Fox does implement shrinking (test.check does implement that too).

More thorough than Example-Based Tests

Test generation can provide better coverage than example-based tests. Instead of having to manually code test cases, Fox can generate tests for you.

Data Generation

The simpliest of test generation is providing random data. Fox can generate them for use if you can define specifications -- known properties of the subject under test:

FOXAssert(FOXForAll(FOXTuple(@[FOXInteger(), FOXInteger()]), ^BOOL(NSArray *values){
    NSInteger x = [tuple[0] integerValue];
    NSInteger y = [tuple[1] integerValue];
    return x + y > x;
});

Once a failing example is produced, Fox will attempt to find the smallest possible example that also exhibits the same failure:

Property failed with: @[@0, @0].

Stateful Testing

How can you test stateful APIs? Represent the state changes as data! Using a state machine, define a model of how your API is suppose to work. Here's one for a queue:

// define a state machine. Model state is the state of your application and
// can be represented with any object you want -- Fox does not interpret it.
FOXFiniteStateMachine *stateMachine = [[FOXFiniteStateMachine alloc] initWithInitialModelState:@[]];

// Adds a transition to the state machine:
// - The API to test is -[addObject:]
// - The generator for the argument is a random integer in an NSNumber
// - A block indicating how to update the model state. This should not mutate the original model state.
[stateMachine addTransition:[FOXTransition byCallingSelector:@selector(addObject:)
                                               withGenerator:FOXInteger()
                                              nextModelState:^id(NSArray *modelState, id generatedValue) {
    return [modelState arrayByAddingObject:generatedValue];
}]];
// Add a custom transition (see FOXStateTransition protocol)
[stateMachine addTransition:[[QueueRemoveTransition alloc] init]];

Now, you can generate tests that exercise an API:

// Generate a sequence of commands executed on the given subject. Since
// this will generate multiple tests, you also give a block of a subject.
id<FOXGenerator> executedCommands = FOXExecuteCommands(stateMachine, ^id {
    return [FOXQueue new];
});

// Verify if the executed commands validated the API conformed to the state machine.
FOXRunnerResult *result = [FOXSpecHelper resultForAll:executedCommands
                                                 then:^BOOL(NSArray *commands) {
    return FOXExecutedSuccessfully(commands);
}];
// result will shrinking to the small sequence of API calls to trigger the
// failure if there is one

Installation

From the documentation.

Fox can be installed in multiple ways. If you don't have a preference, install via git submodule.

Fox honors semantic versioning as humanly possible. If you're unsure if a given update is backwards incompatible with your usage. Check out the releases.

Manually (Git Submodule)

Add Fox as a submodule to your project:

$ git submodule add https://github.com/jeffh/Fox.git Externals/Fox

If you don't want bleeding edge, check out the particular tag of the version:

$ cd Externals/Fox
$ git checkout v1.0.1

Add Fox.xcodeproj to your Xcode project (not Fox.xcworkspace). Then link Fox-iOS or Fox-OSX to your test target.

And you're all set up! Dive right in by following the tutorial.

CocoaPods

Add to your Podfile for you test target to have the latest stable version of Fox:

pod 'Fox', '~>1.0.0'

And then pod install.

And you're all set up! Dive right in by following the tutorial.

Reference

If you want to see examples of usages, see the full reference.

Data Generators

There are many data generators provided for generating data. Most of these generators shrink to zero:

Numerically zero (or as close as possible)
Empty collection (or at least shrunk items)

Function	Generates	Description
FOXInteger	NSNumber *	Generates random integers
FOXPositiveInteger	NSNumber *	Generates random zero or positive integers
FOXNegativeInteger	NSNumber *	Generates random zero or negative integers
FOXStrictPositiveInteger	NSNumber *	Generates random positive integers (non-zero)
FOXStrictNegativeInteger	NSNumber *	Generates random negative integers (non-zero)
FOXChoose	NSNumber *	Generates random integers between the given range (inclusive)
FOXFloat	NSNumber *	Generates random floats that conform to the IEEE standard.
FOXDouble	NSNumber *	Generates random doubles that conforms to the IEEE standard.
FOXDecimalNumber	NSNumber *	Generates random decimal numbers.
FOXReturn	id	Always returns the given value. Does not shrink
FOXTuple	NSArray *	Generates random fixed-sized arrays of generated values. Values generated are in the same order as the generators provided.
FOXTupleOfGenerators	NSArray *	Generates random fixed-sized arrays of generated values. Values generated are in the same order as the generators provided.
FOXArray	NSArray *	Generates random variable-sized arrays of generated values.
FOXArrayOfSize	NSArray *	Generates random fixed-sized arrays of generated values. Values generated are in the same order as the generators provided.
FOXArrayOfSizeRange	NSArray *	Generates random variable-sized arrays of generated values. Array size is within the given range (inclusive).
FOXDictionary	NSDictionary *	Generates random dictionaries of generated values. Keys are known values ahead of time. Specified in `@{<key>: <generator>}` form.
FOXSet	NSSet *	Generates random sets of a given generated values.
FOXCharacter	NSString *	Generates random 1-length sized character string. May be an unprintable character.
FOXAlphabetCharacter	NSString *	Generates random 1-length sized character string. Only generates alphabetical letters.
FOXNumericCharacter	NSString *	Generates random 1-length sized character string. Only generates digits.
FOXAlphanumericCharacter	NSString *	Generates random 1-length sized character string. Only generates alphanumeric.
FOXAsciiCharacter	NSString *	Generates random 1-length sized character string. Only generates ascii characters.
FOXString	NSString *	Generates random variable length strings. May be an unprintable string.
FOXStringOfLength	NSString *	Generates random fixed length strings. May be an unprintable string.
FOXStringOfLengthRange	NSString *	Generates random length strings within the given range (inclusive). May be an unprintable string.
FOXAsciiString	NSString *	Generates random variable length strings. Only generates ascii characters.
FOXAsciiStringOfLength	NSString *	Generates random fixed length strings. Only generates ascii characters.
FOXAsciiStringOfLengthRange	NSString *	Generates random variable length strings within the given range (inclusive). Only generates ascii characters.
FOXAlphabeticalString	NSString *	Generates random variable length strings. Only generates alphabetical characters.
FOXAlphabeticalStringOfLength	NSString *	Generates random fixed length strings. Only generates alphabetical characters.
FOXAlphabeticalStringOfLengthRange	NSString *	Generates random variable length strings within the given range (inclusive). Only generates alphabetical characters.
FOXAlphanumericalString	NSString *	Generates random variable length strings. Only generates alphabetical characters.
FOXAlphanumericalStringOfLength	NSString *	Generates random fixed length strings. Only generates alphanumeric characters.
FOXAlphanumericalStringOfLengthRange	NSString *	Generates random variable length strings within the given range (inclusive). Only generates alphanumeric characters.
FOXNumericalString	NSString *	Generates random variable length strings. Only generates numeric characters.
FOXNumericalStringOfLength	NSString *	Generates random fixed length strings. Only generates numeric characters.
FOXNumericalStringOfLengthRange	NSString *	Generates random variable length strings within the given range (inclusive). Only generates numeric characters.
FOXSimpleType	id	Generates random simple types. A simple type does not compose with other types. May not be printable.
FOXPrintableSimpleType	id	Generates random simple types. A simple type does not compose with other types. Ensured to be printable.
FOXCompositeType	id	Generates random composite types. A composite type composes with the given generator.
FOXAnyObject	id	Generates random simple or composite types.
FOXAnyPrintableObject	id	Generates random printable simple or composite types.

Computation Generators

Also, you can compose some computation work on top of data generators. The resulting generator adopts the same shrinking properties as the original generator.

Function	Description
FOXMap	Applies a block to each generated value.
FOXBind	Applies a block to the value that the original generator generates.
FOXResize	Overrides the given generator's size parameter with the specified size. Prevents shrinking.
FOXOptional	Creates a new generator that has a 25% chance of returning `nil` instead of the provided generated value.
FOXFrequency	Dispatches to one of many generators by probability. Takes an array of tuples (2-sized array) - `@[@[@probability_uint, generator]]`. Shrinking follows whatever generator is returned.
FOXSized	Encloses the given block to create generator that is dependent on the size hint generators receive when generating values.
FOXSuchThat	Returns each generated value iff it satisfies the given block. If the filter excludes more than 10 values in a row, the resulting generator assumes it has reached maximum shrinking.
FOXSuchThatWithMaxTries	Returns each generated value iff it satisfies the given block. If the filter excludes more than the given max tries in a row, the resulting generator assumes it has reached maximum shrinking.
FOXOneOf	Returns generated values by randomly picking from an array of generators. Shrinking is dependent on the generator chosen.
FOXForAll	Asserts using the block and a generator and produces test assertion results (FOXPropertyResult). Shrinking tests against smaller values of the given generator.
FOXForSome	Like FOXForAll, but allows the assertion block to "skip" potentially invalid test cases.
FOXCommands	Generates arrays of FOXCommands that satisfies a given state machine.
FOXExecuteCommands	Generates arrays of FOXExecutedCommands that satisfies a given state machine and executed against a subject. Can be passed to FOXExecutedSuccessfully to verify if the subject conforms to the state machine.

fox's People

Contributors

Stargazers

Watchers

fox's Issues

Project name?

I'm thinking PBT is going to be a focal point of the talk I'm giving at Functional Swift Conference. Get ready for some README and documentation updates, if nothing else!

But I was wondering about the name: "PBT". Is this a codename, or is it a name you're attached to? If the former, let's brainstorm!

Questions about custom Nimble matcher: hold()

I tried writing a Nimble matcher for PBT. This was before I learned about PBTAssert--it might be better to use that instead. For the time being, can you see any problems with this?

// Bridging header

#import <PBT/PBT.h>

import XCTest
import Nimble
// Unfortunately cannot `import PBT` here; must use bridging header

func hold<T: PBTGenerator>(numberOfTests: UInt = 100,
                                    seed: UInt32 = UInt32(time(nil))) -> MatcherFunc<T> {
  return MatcherFunc { actualExpression, failureMessage in
      let runner = PBTRunner() // Should this use `PBTRunner.sharedInstance()`?
      let result = runner.resultForNumberOfTests(numberOfTests,
          property: actualExpression.evaluate(), seed: seed)

      failureMessage.actualValue = "property"
      failureMessage.postfixMessage = "hold, but failed for \(result.smallestFailingValue)"

      return result.succeeded
  }
}

class HoldTests: XCTestCase {
  func testTwentyIsGreaterThanAllIntegers() {
    // Fails with "expected property to hold, but failed for 20"
    expect(PBTForAll(PBTInteger(), { value in
        return 20 > value as Int
    })).to(hold(numberOfTests: 800, seed: 8675309))
  }
}

Also, one thing that confused me about PBTInteger() was that it seemed to only generate numbers between -50 and 50. Is that intended (sorry if dumb question, still reading through source)? It was very unintuitive--as a property-based tester I think I'd expect far out numbers like -879,213, etc.

I like the idea of a Nimble matcher for PBT, but it should probably be packaged as a separate project. It can be distributed using CocoaPods or Git submodules & drag-and-drop into Xcode, like AshFurrow/Nimble-Snapshots..

Is this usable from outside PBTSpecs?

So, ideally I'd like to build a Nimble matcher based off of this in time for the Functional Swift Conference on Dec. 6th.

I've been building my own version of QuickCheck, but I'd rather not reinvent the wheel, and yours is definitely written better. But I'm having trouble running it from outside of PBTSpecs. I tried making it at least importable (see #1), but PBTDeterministicRandom causes build errors.

All I need is something that:

Works in both Swift and Objective-C
Supports shrinking

...so it doesn't have to handle the state machine stuff (which does sound cool, by the way).

Any chance you can take some time from the state machine stuff to first build a sample project that confirms this stuff works from a separate project in Objective-C and Swift?

Given a FSM, generate parallel tests.

Each API call is assumed atomic.

Bonus:

Good ways of testing the compiler besides End-to-End?
Configuring the plugin to automatically refer to its internal copy of the Foxling compiler.

Finalize:

Audit APIs that are alpha
Audit documentation that marks alpha APIs
Grammer / editing check of documentation.
Create and link issues mentioned in the documentation.

Things to consider in other PR/issues:

Dead-lock detection / timeout runs
More rewriting rules for Foxling
Non-brute-force linearizable verifier.

Carthage support?

Not sure if you care about supporting Carthage, but I thought I'd note some difficulties I had trying to integrate Fox with github/Archimedes (I thought it'd make an awesome example during my talk):

Fox isn't versioned at all yet, so I can't install it unless I use my fork, which has temporary versions like v0.0.1. Obviously Fox'll be versioned at some point, so this isn't really a big deal. Also, I've opened an issue for Carthage to support unversioned projects (see Carthage/Carthage#136).
Carthage seems to build the test targets, which include a "Check Pods Manifest.lock" phase. But since Carthage doesn't run pod install, this step fails.

Bullet two on that list blocked me, so I ended up using Git submodules. You can see the end result at this private repo: https://github.com/modocache/Archimedes-Fox/commit/19ae6a8ea5628b6b9e6eeb47fc91bd7e72bf7bcd

Documentation

Probably use Sphinx. Here's some topics to remind myself. Reference (e.g. - function descriptions) should not be in this:

FOXOptional is returning NSNull singleton instead of nil pointer (as docs indicates)

According to FOX's documentation:

FOXOptional - "Creates a new generator that has a 25% chance of returning nil instead of the provided generated value"

But when I tried to use Optionals in my tests, I get a refence to [NSNull null] singleton. So I alway end up managing this myself in tests.

id<FOXGenerator> adultsAndChildrenDictionaryGenerator = FOXDictionary(@{
                                                                        @"adults": FOXOptional(FOXInteger()),
                                                                        @"children": FOXOptional(FOXArray(FOXInteger()))
                                                                        });

FOXAssert(FOXForAll(adultsAndChildrenDictionaryGenerator, ^BOOL(NSDictionary *adultsAndChildrenDictionary) {

    NSLog(@"%@",adultsAndChildrenDictionary);

    NSNumber *adults = adultsAndChildrenDictionary[@"adults"];

    if ([adults isEqual:[NSNull null]]) {
        adults = nil;
    }
    NSArray *childrenArray = adultsAndChildrenDictionary[@"children"];

    if ([childrenArray isEqual:[NSNull null]]) {
        childrenArray = nil;
    }

    //Test
    NSString *currentDistribution = [self distributionWithAdultsQuantity:adults
                                                             childrenAge:childrenArray];

Can you do anything to directly retrieve a nil pointer?

thanks!

Fox doesn't support Swift 2.1

Swift changes pretty fast, as an user, I really hope Fox could support Swift 2.1.

This is cool.

That is all.

Floats, Doubles, and NSDecimalNumbers should shrink reliably

Currently, they do not shrink in a consistent fashion.

See flakiness on FOXFloatSpecs.

Provide a testing module to verify asynchronous APIs.

A temporal model keeps track of time to allow verification of sequences of multiple async calls.

Research:

Erlang QuickCheck has this model that is used to test ejabberd

FOXBind does not shrink consistently when nested

FOXBind should prefer shrinking first values. But it seems to shrink more arbitrarily.

Needs more investigation.

Drawing from memory.

id<Generator> generator = FOXBind(FOXInteger(), ^id (id num1){
    return FOXBind(FOXInteger(), ^id (id num2) {
        return @[num1, num2];
    });
});

FOXAssert(FOXForAll(generator, ^BOOL (NSArray *values) {
    return [values[0] integerValue] + [values[1] integerValue] < 10;
});

Is it possible to generate functions?

I'm writing a test suite to exercise the Functor, Applicative, and Monadic laws (thoughtbot/Runes#2). Right now, I'm only generating the values, but it would be great to be able to generate functions to replace the append and prepend functions that I currently have. Since these laws should hold up for any value and any function, it would be awesome to be able to generate those as well.

Move TODO.md into GitHub issues

The TODO requires a code change to update; issues are much more flexible and easier to search. You should use issues as the source of truth on upcoming features/bugfixes! ☝️ 💡

Finalize Swift API

The Swift API is alpha because:

Fox's (cedar) test suite currently cannot test Swift.
It does not cover all of the objective-c API.
It is not ideal in its design (it would be nice to re-introduce generics somehow).

It would be great to properly expand the API to swift. Although it might be intrusive to work on #28.

"Famous" value generators

For more comprehensive coverage, it would be useful to provide famous value generators. These generators prioritize "famous" values (minimum/maximum values) at a higher probability than can be achieved through normal generation:

Integer (min, max)
PositiveInteger (min, max)
NegativeInteger (min, max)
StrictPositiveInteger (min, max)
StrictNegativeInteger (min, max)
Float (min, max, inf, -inf, NaN)
Double (min, max, inf, -inf, NaN)
DecimalNumber (min, max, NaN)

This can be accomplished with FOXFrequency with a 1-10% probability of a generating famous values.

Tests:

Backfill StrictPositiveInteger tests
Backfill StrictNegativeInteger tests

Header Docs:

Docs:

Can't use FOXAssert from Swift

Since FOXAssert is a complex macro, it can't be used in tests written in Swift. The only option is to use the underlying _FOXAssert, which ends up being quite messy:

let _ = _FOXAssert(FOXForAll(FOXInteger(), { integer in
  return integer as Int == 1
}), "should be equal to one", __FILE__, __LINE__, FOXOptions(seed: arc4random(), numberOfTests: 100, maximumSize: 1000))

In the meantime, I'm using a custom Nimble matcher (see #4), but it seems like it'd be a good idea to provide an assertion function Swift consumers can use.

When would FOXPropertyStatusSkipped be used?

Searching through the code it seems like this case is never reached. When do you anticipate Fox would use this status?

State Machine Question

Hey guys,
Excited about Fox. I'm working on a project that composes index spaces from NSFetchedResultsController or static arrays into an tree structure that can be indexed by NSIndexPath and back UITableView / UICollectionView's. Whenever any underlying data source changes, change sets are percolated up to update the backing view. Any index space can be transformed via a filter with a few more transformations coming (Like sort and map).

I've written a number of tests to aggrevate a few scenarios, but I'm very interested in writing tests that will aggrevate more scenarios. The majority of test scenarios involve starting a batch update, performing a number of inserts and removals, and then closing the update and confirming the state of the change set.

I wrote a state machine test that had 4 transitions, open, insert, remove, close. I could ensure that the transitions got called in the right order, but it wasn't very ideal. I ran into a few issues, and I was wondering if you had any input:

It took a long time to generate a valid test cases.
It executed a lot of test cases that did not close the batch update. Closing the batch is the test scenario that I'm most interested in testing.
I'm not sure how to control the number of transitions that occur. It appears to be only generating X transitions, 1 for every defined transition. Ideally, the test would open, perform a large number of random insert or remove operations, then close the batch.
I don't know how to define a generator to vary the initial state of the subject factory. Not sure how that would work.

Here's the test for reference:

https://github.com/KingOfBrian/RZAssemblage/blob/testing/fox/RZAssemblageTests/RZAssemblagePropertyTestCase.m#L203

I'm not sure that the state machine approach is the right direction. Maybe I should use a custom generator instead? Anyway, I get this is a pretty open ended question. Any insight would be great!

Brian

Add documentation to getting set up with Carthage

This requires updating the README and the documentation installation.rst.

Problem with cocoa pods and deployment target iOS 7.0

My project has deployment target iOS 7.0 and I am trying to install Fox using cocoa pods (so in my podfile I have platform :ios, "7.0"). I get the following error:

[!] Unable to satisfy the following requirements:

Fox (~> 1.0.0) required by Podfile

If I change my deployment target to 8.0 the error goes away but unfortunately I have to support iOS 7.

Any ideas?

Show failures inline in the editor

I am using Swift -- not sure what happens in Obj-C.

If you call XCTFail instead of throwing an exception, Xcode will show the error inline in the editor at the appropriate line.

I'm willing to make a PR to implement this, but asking for guidance. It feels like I need to pass a closure to use instead of raising, potentially in FOXOptions.

Any thoughts on whether this is a desired PR and how to go about it?

Test script should not clear Derived Data by default

Clearing out Derived Data leads to very long build times on other projects. The test script should be a good citizen and not mess with other projects' build times. So maybe only do it with the flag --clean-derived-data, or something to that effect?

FOXFrequency should shrink by generator order.

It would be great for FOXFrequency to also shrink by the order of the generator:

FOXFrequency(@[@[@1, FOXInteger()],
               @[@1, FOXString()]]);

This generator should shrink towards @0, even if the originally failure was a string (assuming all shrunk values cause failures).

Passing test takes long time to finish (> 30 sec)

I tried to run the following code using XCode 7 both on a physical device and on a simulated ios device.

id<FOXGenerator> generator = FOXArray(FOXAlphabeticalString());

FOXAssert(FOXForAll(generator, ^BOOL(NSArray<NSString *> *array) {
    return array.count >= 0;
}));

It ran in 38 seconds. Is this expected behaviour? Failing tests finish in < 1 sec.

Allow default runner to be configured via command line/env vars

QuickCheck/Fox isn't performant--that's not what it was designed for, after all. So when running Fox locally, I probably only want to test each property a few hundred times, in order to keep my test runs short and my feedback cycle tight. But when running on CI, or for nightly builds, time isn't as much of an issue anymore, and I may want to test each property thousands of times.

FOXRunner should be configurable via the command line, so that my CI servers can run more exhaustive tests, whereas I only run the default amount.

[RFC] FOXRunner should have a runForTimeInterval: option

In addition to being able to configure how many times FOXRunner tests a property, it might be interesting to be able to specify how long it should test a property.

This would be useful in environments in which time is a limiting resource, such as when faced with a ballooning test suite. If I have 100 property-based tests, it would be nice to be able to specify that Fox should spend one second per test, and no more. That way, as I add tests, the time it takes to execute the test suite will increase at a linear, controllable rate.

Swift files are not in the Pod

They aren't listed as source files in the Podspec, so they aren't included when you install via pod install. I realize they are alpha, but if they don't interfere with the Obj-C version, it would be nice to have them

README: More detailed example of stateful testing?

Around the 4'00" mark of this video on QuickCheck, John Hughes demonstrates testing the state of a queue, complete with pre-conditions and post-conditions. How would I model such a thing in PBT?