Perhaps, dotnet-jack could benefit from using SplitMix (or even TFGen¹?), since the current splittable random number generator seems to have known issues, as shown here, here, and here.

¹ In fact, QuickCheck 2 provides a wrapper around either TFGen or StdGen.

Like NonNegative, Positive, etc.

These types do things such as restricting the kind of test data that can be generated. They can be pattern-matched on in properties as a stylistic alternative to using explicit quantification.

(Haskell QuickCheck example shown below.)

prop_TakeDrop (NonNegative n) (xs :: [A]) =
  take n xs ++ drop n xs == xs

As soon as we proceed with hedgehogqa/haskell-hedgehog#124.

open Fare
open Hedgehog

[<EntryPoint>]
let main _ =
    let fromRegex (pattern: string) : Gen<string> =
        Gen.sized (fun size ->
            let xeger = Xeger pattern
            Gen.item [ for _ in 1..size -> xeger.Generate() ]
        )
    
    property {
        let! s = fromRegex @"^\[\<([A-Z][a-zA-Z0-9]*)*\>\]$"
        printfn "%s" s
        return true
    } |> Property.print' 10<tests>
 
    0

[<N28UhdD2m8m589286x>]
[<R>]
[<K3Mol1RRwF8U1Kmh84>]
[<J>]
[<>]
[<>]
[<>]
[<>]
[<>]
[<R5gn44pB2>]

Where 5 of 10 tests generates the same sample.

If (say) an array is declared before the generator binds in a property computation expression, it doesn't get cleared when shrinking after a test failure. If the declaration is moved after the let! then it's ok.

This can produce misleading failure messages where the reported state is a combination of what happened during the shrinkage, not what caused the original failure.

As a (contrived) repro case, this test takes 10 integers and copies the distinct values into an array of int options. It then tests that the Some values are the sorted distinct values (so it fails).

let copyIntoArray (target : array<_>) source =
    source |> Array.iteri (fun i x -> target.[i] <- Some x)

[<Test>]
let ``strange`` () =
    property {
        let buffer = Array.zeroCreate 10
        
        let! input =
            Gen.int (Range.constant 1 100)
            |> Gen.array (Range.constant 10 10)

        // Failure message is sensible if buffer is declared here instead
        // let buffer = Array.zeroCreate 10

        let actual =                 
            input  |> Array.distinct |> copyIntoArray buffer
            buffer |> Array.choose id

        let expected = 
            input  |> Array.distinct |> Array.sort
       
        printfn "testing %A vs %A" actual expected
        
        test <@ actual = expected @>
    } |> Property.check

During shrinkage, the input array is made shorter but the buffer array has items left-over. The output reads:

testing [|38; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|13; 21; 23; 38; 43; 44; 56; 59; 80; 99|]
testing [|1; 21; 80; 43; 23; 59; 13; 56; 44; 99|] vs [|1; 13; 21; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 80; 43; 23; 59; 13; 56; 44; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 80; 99|]
testing [|1; 43; 23; 59; 13; 56; 44; 99; 99; 99|] vs [|1; 13; 23; 43; 44; 56; 59; 99|]
testing [|1; 23; 59; 13; 56; 44; 99; 99; 99; 99|] vs [|1; 13; 23; 44; 56; 59; 99|]
testing [|1; 59; 13; 56; 44; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 59; 99|]
testing [|1; 13; 56; 44; 99; 99; 99; 99; 99; 99|] vs [|1; 13; 44; 56; 99|]
testing [|1; 56; 44; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 56; 99|]
testing [|1; 44; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 44; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1; 99|]
testing [|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] vs [|1|]
Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 10 shrinks):
[|1; 1; 1; 1; 1; 1; 1; 1; 1; 1|]
NUnit.Framework.AssertionException: 

[|1; 99; 99; 99; 99; 99; 99; 99; 99; 99|] = [|1|]
false

But if the buffer declaration is moved, the shrinkage works properly and the message reads:

Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 24 shrinks):
[|1; 1; 1; 1; 1; 1; 3; 2; 1; 1|]
NUnit.Framework.AssertionException: 

[|1; 3; 2|] = [|1; 2; 3|]
false

Even if this is by-design behaviour, it might be worth noting it in the docs

in a file, or a module, using Erased Provided Types.

I haven't thoroughly think about this, but it should act similarly to adding the Template Haskell splice which runs all the properties in Haskell.

into dotnet-jack codebase using Paket's neat support for GitHub dependencies.

This way we can bundle LazyList.fs and LazyList.fsi into dotnet-jack during ./build.sh and so #45 can be a whole lot easier.

Now that fsprojects/FSharpx.Collections#67 is merged, I have already something locally that seems to work, and looks like this:

diff --git a/Jack.Tests/Jack.Tests.fsproj b/Jack.Tests/Jack.Tests.fsproj
index d407817..753a507 100644
--- a/Jack.Tests/Jack.Tests.fsproj
+++ b/Jack.Tests/Jack.Tests.fsproj
@@ -97,17 +97,6 @@
   </Target>
   -->
   <Choose>
-    <When Condition="$(TargetFrameworkIdentifier) == '.NETFramework' And ($(TargetFrameworkVersion) == 'v4.0' Or $(TargetFrameworkVersion) == 'v4.5' Or $(TargetFrameworkVersion) == 'v4.5.1' Or $(TargetFrameworkVersion) == 'v4.5.2' Or $(TargetFrameworkVersion) == 'v4.5.3' Or $(TargetFrameworkVersion) == 'v4.6' Or $(TargetFrameworkVersion) == 'v4.6.1' Or $(TargetFrameworkVersion) == 'v4.6.2' Or $(TargetFrameworkVersion) == 'v4.6.3')">
-      <ItemGroup>
-        <Reference Include="FSharpx.Collections">
-          <HintPath>..\packages\FSharpx.Collections\lib\net40\FSharpx.Collections.dll</HintPath>
-          <Private>True</Private>
-          <Paket>True</Paket>
-        </Reference>
-      </ItemGroup>
-    </When>
-  </Choose>
-  <Choose>
     <When Condition="$(TargetFrameworkIdentifier) == '.NETFramework' And $(TargetFrameworkVersion) == 'v4.0'">
       <ItemGroup>
         <Reference Include="Unquote">

diff --git a/Jack.Tests/paket.references b/Jack.Tests/paket.references
index 09cabf4..46dd121 100644
--- a/Jack.Tests/paket.references
+++ b/Jack.Tests/paket.references
@@ -2,4 +2,3 @@ xunit.runner.console
 xunit.core
 Unquote
 
-FSharpx.Collections

diff --git a/Jack/Jack.fsproj b/Jack/Jack.fsproj
index ac7e5ac..39efba8 100644
--- a/Jack/Jack.fsproj
+++ b/Jack/Jack.fsproj
@@ -45,6 +45,14 @@
     <Reference Include="System.Numerics" />
   </ItemGroup>
   <ItemGroup>
+    <Compile Include="..\paket-files\fsprojects\FSharpx.Collections\src\FSharpx.Collections\LazyList.fsi">
+      <Paket>True</Paket>
+      <Link>paket-files/LazyList.fsi</Link>
+    </Compile>
+    <Compile Include="..\paket-files\fsprojects\FSharpx.Collections\src\FSharpx.Collections\LazyList.fs">
+      <Paket>True</Paket>
+      <Link>paket-files/LazyList.fs</Link>
+    </Compile>
     <Compile Include="Numeric.fs" />
     <Compile Include="Seed.fs" />
     <Compile Include="Tree.fs" />
@@ -58,15 +66,4 @@
     <Compile Include="Script.fsx" />
   </ItemGroup>
   <Import Project="$(FSharpTargetsPath)" />
-  <Choose>
-    <When Condition="$(TargetFrameworkIdentifier) == '.NETFramework' And ($(TargetFrameworkVersion) == 'v4.0' Or $(TargetFrameworkVersion) == 'v4.5' Or $(TargetFrameworkVersion) == 'v4.5.1' Or $(TargetFrameworkVersion) == 'v4.5.2' Or $(TargetFrameworkVersion) == 'v4.5.3' Or $(TargetFrameworkVersion) == 'v4.6' Or $(TargetFrameworkVersion) == 'v4.6.1' Or $(TargetFrameworkVersion) == 'v4.6.2' Or $(TargetFrameworkVersion) == 'v4.6.3')">
-      <ItemGroup>
-        <Reference Include="FSharpx.Collections">
-          <HintPath>..\packages\FSharpx.Collections\lib\net40\FSharpx.Collections.dll</HintPath>
-          <Private>True</Private>
-          <Paket>True</Paket>
-        </Reference>
-      </ItemGroup>
-    </When>
-  </Choose>
-</Project>
+</Project>
\ No newline at end of file

diff --git a/Jack/Shrink.fs b/Jack/Shrink.fs
index fc096d0..24fda13 100644
--- a/Jack/Shrink.fs
+++ b/Jack/Shrink.fs
@@ -20,6 +20,9 @@ module LazyList =
             else
                 LazyList.cons x <| LazyList.cons y ys
 
+module private List =
+    let inline cons hd tl = hd::tl
+
 module Shrink =
     /// Permutes a list by removing 'k' consecutive elements from it:
     ///

diff --git a/Jack/paket.references b/Jack/paket.references
index 89cf281..a24c8a2 100644
--- a/Jack/paket.references
+++ b/Jack/paket.references
@@ -1 +1,2 @@
-FSharpx.Collections
+File: LazyList.fs
+File: LazyList.fsi
\ No newline at end of file

diff --git a/paket.dependencies b/paket.dependencies
index f716e48..559aa22 100644
--- a/paket.dependencies
+++ b/paket.dependencies
@@ -4,7 +4,8 @@ source https://www.nuget.org/api/v2/
 nuget FAKE 4.41.1
 nuget xunit.runner.console 2.1.0
 
-nuget FSharpx.Collections 1.15.2
+github fsprojects/FSharpx.Collections:944b8c8e464cdd10070889f53c78e9fbeb478698 src/FSharpx.Collections/LazyList.fs
+github fsprojects/FSharpx.Collections:944b8c8e464cdd10070889f53c78e9fbeb478698 src/FSharpx.Collections/LazyList.fsi
 
 nuget xunit.core 2.1.0
 nuget Unquote 3.1.2

diff --git a/paket.lock b/paket.lock
index 10eca13..e9e32d7 100644
--- a/paket.lock
+++ b/paket.lock
@@ -2,9 +2,6 @@ REDIRECTS: ON
 NUGET
   remote: https://www.nuget.org/api/v2
     FAKE (4.41.1)
-    FSharp.Core (4.0.0.1)
-    FSharpx.Collections (1.15.2)
-      FSharp.Core
     Microsoft.NETCore.Platforms (1.0.1) - framework: dnxcore50
     Microsoft.NETCore.Targets (1.0.1) - framework: dnxcore50
     System.Collections (4.0.11) - framework: dnxcore50
@@ -133,3 +130,7 @@ NUGET
       xunit.abstractions (>= 2.0) - framework: dnxcore50
       xunit.extensibility.core (2.1) - framework: >= net45, dnx451, dnxcore50, monoandroid, monotouch, xamarinios, winv4.5, wpv8.0, wpav8.1
     xunit.runner.console (2.1)
+GITHUB
+  remote: fsprojects/FSharpx.Collections
+    src/FSharpx.Collections/LazyList.fs (944b8c8e464cdd10070889f53c78e9fbeb478698)
+    src/FSharpx.Collections/LazyList.fsi (944b8c8e464cdd10070889f53c78e9fbeb478698)
\ No newline at end of file

for full control over the scope of generated numbers and collections.

FSCheck can be used in C# with slightly less convenient ergonomics.

Can fsharp-hedgehog be fully exercised from C# code as well?

I've just barely started experimenting with Hedgehog. I've previously used FsCheck, and what I really like about FsCheck is its simplicity. For many tests with simple (mostly primitive) inputs, I can simply define the test as a function of the relevant parameters and off we go.

Hedgehog on the other hand seems to require me to explicitly state in all tests how values are generated, even for simple primitive types like ints. For example, here's a test for a function constrain that constrains an int to a given interval (I'm using Xunit and Unquote):

[<Fact>]
let ``interval order doesn't matter`` () = Property.check <| property {
  let! x1, x2, x = Range.exponentialBounded() |> Gen.int |> Gen.tuple3
  test <@ constrain (x1, x2) x = constrain (x2, x1) x @> }

Here's the same in FsCheck:

[<Property>]
let ``interval order doesn't matter`` (x1:int) x2 x =
  test <@ constrain (x1, x2) x = constrain (x2, x1) x @>

Now, I've read blog posts and FsCheck issue discussions with strong arguments from @ploeh for why it's safer to explicitly define how values are generated, and I certainly get that for more complex or constrained types. Still, having to explicitly create default generators (e.g. specifying ranges for ints, ranges and character generators for strings) for even the most simple types seems a bit long-winded when all I want much of the time is to say "gimme a string" or "gimme an int".

This is of course not helped by there being almost no documentation (I've only found the tutorial, and there seems to be very little doc comments), forcing me to experiment myself to e.g. realize that exponentialBounded seems a good choice for a default int generator (and likely several other generators). And while I can just factor that out to common test generators, that would need to be repeated for all test projects. All things considered, Hedgehog as compared to FsCheck might be a bit of a hard sell to my colleagues (I'm doing an F# course for my company this week and wondering if I should recommend FsCheck or Hedgehog).

What do you think about defining some standard generators for primitive types (ints, strings, etc.) to make some of the Hedgehog usage simpler?

I'm completely open to this being a terrible idea for whatever reason. If that is the case, please let me know why so that I can learn.

When using Range.Linear I got this:

System.NotSupportedException: 'Specified method is not supported.'
   at [email protected](Int32 sz)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed0, Int32 size)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed, Int32 size)
   at [email protected](Seed seed0, Int32 size)
   at Hedgehog.Property.loop@332-21(Int32 n, Random`1 random, Seed seed, Int32 size, Int32 tests, Int32 discards)
   at Hedgehog.Property.Check(Property`1 g)

Range.Constant works fine, so I'm sticking with that at the moment. I can't see any obvious NotSupportedException uses in the code around line 155...

How can I generate discriminated union cases? FsCheck handles this automatically, and I can't see any methods in Hedgehog that could easily allow me to do that. The only thing I can come up with is to create generators for each case of every DU, and then use Gen.item or similar to pick one of the generators. (In the case of a very simple DU with little or no data, only case labels, Gen.item could be used directly.)

This has the drawback that if I expand the DU, I have to manually add the new cases to the generator.

It'll make it a whole lot easier to use dotnet-hedgehog if we put it on NuGet. Right now it's usable mostly via Paket, as shown in this sample.

If we put it on NuGet, I'd propose to also keep http://semver.org/ in mind, and start with version 0.x, so we can be more flexible with any breaking changes we might introduce during the next couple of months.

To publish on NuGet, we need a profile on the NuGet Gallery. May I proceed by creating one for Hedgehog QA? Otherwise we could use @jystic's or the one I use already?

Is there a special reason that the API documentation is written as it is?

In the Visual Studio IDE, the documentation is hard to read:

Perhaps the formatting is written like that because it's readable in some other editor..?

FsCheck uses reflection to derive default implementations of various F# types as per [1]. After quick look hedgehog doesn't seem to have this functionality (if not so, please let me know). Should this functionality added and how high of a priority should it be?

Furthermore, by using reflection, FsCheck can derive default implementations of record types, discriminated unions, tuples, enums and basic classes in terms of any primitive types that are defined (either in FsCheck or by you).

[1] FsCheckDocs

Nevermind me, I misunderstood the interface :)

When I write test assertions in F#, I normally prefer to use Unquote. I have, however, noticed that it doesn't work as well as one could wish with Hedgehog.

Hedgehog with Unquote

Consider this simple repro:

open Xunit
open Hedgehog
open Swensen.Unquote

[<Fact>]
let ``Property with Unquote assertion`` () = Property.check <| property {
    let! (x, y, z) = Range.constant 0 9 |> Gen.int |> Gen.tuple3
    test <@ x + z = y + z @> // Provoke a test failure
    }

In this test, I've written an assertion that should cause the property to fail. I did that on purpose. When the test fails, it produces output like this:

Test 'HedgehogProperties.Property with Unquote assertion' failed: Swensen.Unquote.AssertionFailedException : Test failed:

3 + 3 = 7 + 3
6 = 10
false

[...]

I've omitted the stack trace that's also produced, because it's irrelevant for my point.

What I particularly like about Unquote is that it tells me, step-by-step, how the assertion evaluated to false, through a series of reductions.

That seems to work well, so what's the problem?

Hedgehog without Unquote

The problem is that the above output doesn't show me the generated values.

Consider this alternative way to write the property, without Unquote:

open Xunit
open Hedgehog

[<Fact>]
let ``Property with boolean assertion`` () = Property.check <| property {
    let! (x, y, z) = Range.constant 0 9 |> Gen.int |> Gen.tuple3
    return x + z = y + z // Provoke a test failure
    }

When the property fails, it produces output like this:

Test 'HedgehogProperties.Property with boolean assertion' failed: Hedgehog.FailedException : *** Failed! Falsifiable (after 1 test and 4 shrinks):
(0, 1, 0)

[...]

Notice that in this case, I do get information about the values generated, the number of tests and shrinks.

FsCheck with Unquote

FWIW, the Unquote combination works fine in FsCheck:

open Xunit
open FsCheck
open Swensen.Unquote

[<Fact>]
let ``Property with Unquote assertion`` () =
    let prop = Arb.generate |> Gen.three |> Arb.fromGen |> Prop.forAll <| fun (x, y, z) ->
        test <@ x + z = y + z @> // Provoke a test failure
    Check.QuickThrowOnFailure prop

Running this property produces output like this:

Test 'FsCheckProperties.Property with Unquote assertion' failed: System.Exception : Falsifiable, after 1 test (0 shrinks) (StdGen (1270052570,296354013)):
Original:
(2, 1, -2)
with exception:
Swensen.Unquote.AssertionFailedException: Test failed:

2 + -2 = 1 + -2
0 = -1
false

Notice that FsCheck both reports the values generated, the number of shrinks, and the Unquote assertion message.

Am I doing something wrong?

Is it possible to get the best of both worlds from Hedehog?

When auto-generation (#133) is implemented (e.g. in Hedgehog.Experimental as discussed in #139), it might be nice to offer a simple kind of xUnit integration (similar to FsCheck) that can be used for simple purposes (for example in a new assembly Hedgehog.Xunit that depends on Hedgehog.Experimental). It might turn out to be counterproductive, but I'm opening an issue anyway so it can be discussed. Here's what I'm talking about:

A simple test using the current API, where all values can be auto-generated:

[<Fact>]
let ``Reverse of reverse of a list is the original list`` () =
  Property.check <| property {
    let! (xs : int list) = Gen.auto()
    return xs |> List.rev |> List.rev = xs
  }

The same simple test in a proposed new API (note the new attribute):

[<Property>]
let ``Reverse of reverse of a list is the original list`` (xs : int list) =
    xs |> List.rev |> List.rev = xs

One could customize the run using the attribute, e.g. [<Property(Tests=500)>].

Open questions:

1. Will this push developers away from best practices? I can imagine the simpler syntax being used in cases where the full property { } syntax would be a better choice.

2. Are there actually that many usecases where one can rely exclusively on auto-generated values? I have currently only used Hedgehog in a single (real-life) project, but in that specific example, that's definitely not the case - I require more control.

3. The need for more control over the generation could be alleviated somewhat (not fully) by filtering. But how should this be done? Without a property expression, the where clause isn't available. (FsCheck has its own operator for this, <boolExpression> ==> [lazy] <restOfTest>.)

4. What happens when not all values can be auto-generated? One could force the user to use the normal API, or we could have a slightly simplified combined API like this:

[<Property>]
let ``Some other test`` (xs : int list) = property {
    let! otherValue = Gen.somethingNotAuto |> Gen.combinatorStuff |> Gen.etc
    ...
  }

All of the above points considered, I'm not convinced this is a good idea, but it has its use cases.

The F# version should also support State Machine Testing similar to the Haskell version.

See hedgehogqa/haskell-hedgehog#89, and at least up to hedgehogqa/haskell-hedgehog#96.

To have a solid foundation to build on, we first need to have Tree and Gen data types and combinators in line with the Haskell version, as per #111 and #89.

This is the test in question:

I ran this under the profiler and the problem is entirely due to the sprintf in forAll:

Replacing this with x.ToString() reduces the runtime from to 40 s to 0.55 s (!)

However, ToString isn't as useful as sprintf "%A" for 'native' F# types. A better fix would be to make this function lazily-invoked, so that it's only called if needed.

instead of returning bool and printing to stdout, because otherwise xUnit.net (and I believe also NUnit and MSTest) can't figure out whether a property fails or not:

[<Fact>]
let ``the reverse of the reverse of a list is the list itself`` () =
    Property.check <| property {
        let! xs = Gen.list Gen.int
        return List.rev xs = xs // Fails, on purpose.
    }

Output:

------ Test started: Assembly: Jack.Tests.dll ------

1 passed, 0 failed, 0 skipped, took 0.69 seconds (xUnit.net 2.1.0 build 3179).

Expected output:

------ Test started: Assembly: Jack.Tests.dll ------

Test 'Jack.Tests.ShrinkTests.the reverse of the reverse of a list is the list itself' 
failed: System.Exception : *** Failed! Falsifiable (after 5 tests and 3 shrinks):
[1; 0]

    at Microsoft.FSharp.Core.Operators.FailWith[T](String message)
    ..\Property.fs(215,0): at Jack.PropertyModule.check'(Int32 n, Property`1 p)
    ..\Property.fs(218,0): at Jack.PropertyModule.check(Property`1 p)
    ..\PropertyTests.fs(126,0):
        at Jack.Tests.ShrinkTests.the reverse of the reverse of a list is the list itself()

0 passed, 1 failed, 0 skipped, took 2.86 seconds (xUnit.net 2.1.0 build 3179).

That, or we could add another function in the Property module that throws a System.Exception instead. – Let me know and I'd happily send a pull request 😃

Hedgehog generates strings (haven't tested other types) more than an order of magnitude slower than FsCheck.

I discovered this after converting tests from FsCheck to Hedgehog and noticing a drastic slowdown of the test suite.

Code to test:

module H =

  open Hedgehog

  let s = Gen.string (Range.singleton 10) Gen.alphaNum

  let check () = s |> Gen.sample 0 10000

module F =

  open FsCheck

  let alphaNum = ['0'..'9'] @ ['A'..'Z'] @ ['a'..'z']

  let s = Gen.elements alphaNum
          |> Gen.listOfLength 10
          |> Gen.map (fun cs -> System.String.Concat(Array.ofList(cs)))

  let check () = s |> Gen.sample 0 10000
  

Result of interactive execution using #time:

> F.check();;
> Real: 00:00:00.057, CPU: 00:00:00.062, GC gen0: 11, gen1: 0, gen2: 0
val it : string list =
  ["mj3bVSmKEY"; "smj3bVSmKE"; "Ksmj3bVSmK"; "dKsmj3bVSm"; "OdKsmj3bVS";
   "LOdKsmj3bV"; "VLOdKsmj3b"; "7VLOdKsmj3"; "X7VLOdKsmj"; "xX7VLOdKsm"; ...]

> H.check();;
> Real: 00:00:01.838, CPU: 00:00:01.890, GC gen0: 52, gen1: 14, gen2: 3
val it : List<string> =
  ["ffmhtVdB5q"; "283b0i3P88"; "RZUYtob07L"; "PSEp4C9Q1c"; "PkPv63NQ83";
   "b9Gnw9ZN7N"; "3HFgXJB5tl"; "TcwRo7AjON"; "HfZGkhHb0h"; "Oj6Fg0RyBw"; ...]

Notice the insane difference: 57 ms for FsCheck vs. 1838 ms for Hedgehog.

I setup a Virtual Machine with Windows 10, Visual Studio 2017 and making sure to select F# in the installer.

Even though I can compile the solution in Visual Studio 2017, when I run ./build.sh in Git Bash it fails:

$ ./build.sh
Paket version 5.1.8
Performance:
 - Disk IO: 92 milliseconds
 - Runtime: 1 second
Building project with version: LocalBuild
Shortened DependencyGraph for Target Test:
<== Test
   <== Doctest
      <== Build

The resulting target order is:
 - Build
 - Doctest
 - Test
Starting Target: Build
Building project: C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln
  c:\Windows\Microsoft.NET\Framework\v4.0.30319\MSBuild.exe  C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln /t:Rebuild /m      /p:RestorePackages="False" /p:Configuration="Release" /logger:Fake.MsBuildLogger+ErrorLogger,"C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\packages\FAKE\tools\FakeLib.dll"
Microsoft (R) Build Engine version 4.7.2046.0
[Microsoft .NET Framework, version 4.0.30319.42000]
Copyright (C) Microsoft Corporation. All rights reserved.

Build started 6/23/2017 2:16:16 PM.
     1>Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" on node 1 (Rebuild target(s)).
     1>ValidateSolutionConfiguration:
         Building solution configuration "Release|Any CPU".
     1>Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" (1) is building "C:\Users\Nikos\Documen
       ts\Hedgehog\fsharp-hedgehog\src\Hedgehog\Hedgehog.fsproj" (2) on node 1 (Rebuild target(s)).
     2>C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\src\Hedgehog\Hedgehog.fsproj(42,3): error MSB4019: The importe
       d project "C:\Program Files (x86)\Microsoft SDKs\F#\3.0\Framework\v4.0\Microsoft.FSharp.Targets" was not found.
       Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.
     2>Done Building Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\src\Hedgehog\Hedgehog.fsproj" (Rebuild
       target(s)) -- FAILED.
     1>Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" (1) is building "C:\Users\Nikos\Documen
       ts\Hedgehog\fsharp-hedgehog\tests\Hedgehog.Tests\Hedgehog.Tests.fsproj" (3) on node 2 (Rebuild target(s)).
     3>C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\tests\Hedgehog.Tests\Hedgehog.Tests.fsproj(51,3): error MSB410
       2: The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero len
       gth.
     3>Done Building Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\tests\Hedgehog.Tests\Hedgehog.Tests.fsp
       roj" (Rebuild target(s)) -- FAILED.
     1>Done Building Project "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" (Rebuild target(s)) -- FA
       ILED.

Build FAILED.

       "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" (Rebuild target) (1) ->
       "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\src\Hedgehog\Hedgehog.fsproj" (Rebuild target) (2) ->
         C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\src\Hedgehog\Hedgehog.fsproj(42,3): error MSB4019: The impor
       ted project "C:\Program Files (x86)\Microsoft SDKs\F#\3.0\Framework\v4.0\Microsoft.FSharp.Targets" was not found
       . Confirm that the path in the <Import> declaration is correct, and that the file exists on disk.


       "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln" (Rebuild target) (1) ->
       "C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\tests\Hedgehog.Tests\Hedgehog.Tests.fsproj" (Rebuild target)
       (3) ->
         C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\tests\Hedgehog.Tests\Hedgehog.Tests.fsproj(51,3): error MSB4
       102: The value "" of the "Project" attribute in element <Import> is invalid. Parameter "path" cannot have zero l
       ength.

    0 Warning(s)
    2 Error(s)

Time Elapsed 00:00:00.76
Running build failed.
Error:
Building C:\Users\Nikos\Documents\Hedgehog\fsharp-hedgehog\Hedgehog.sln failed with exitcode 1.

---------------------------------------------------------------------
Build Time Report
---------------------------------------------------------------------
No target was successfully completed
---------------------------------------------------------------------

It seems that the built-in DateTime generator will always generate whole seconds, i.e. that the Milliseconds part is always zero. I suggest that the default implementation be changed so that the generator also generates random sub-second values. This also fits the current generator documentation, which says "a random instant in time".

dotnet-jack doesn't have an explicit licence. Should we continue with the GitHub's (implicit) license, or should we change the licence to something more mainstream, such as MIT License, Apache License 2.0, or GNU GPLv3?

http://choosealicense.com provides more information.

See #110.

See hedgehogqa/haskell-hedgehog#125.

Add sample generators for

• functions (#125)
• discriminated unions and classes (#133, #93)

Is it possible to check if a function throws an exception without having try ... with in the property and explicitly returning true or false?

Do you plan on this kind of feature? Gen.type<'a> from already registered Gen types.

Do we still need validate.sh?

One of Hedgehog's main strengths, IMHO, is its flexibility due to all the Gen combinators. However, practical use entirely depends on the user to define commonly used combinations themselves to avoid too much tedious repetition in the tests, and then either repeat these definitions for each test project, or create a separate solution/assembly with common usecases and add that to all test projects. I think most people will just copy-paste the same common combinations across all of their test projects (copy-paste).

I'd like Hedgehog to have a bit more "batteries included" approach. This would make the framework a much easier sell, so to speak. This could either be done by including common shortcuts in Hedgehog itself, or it could be released as a separate nuget (Hedgehog.Batteries?) maintained by Hedgehog maintainers or someone else. If it's a separate library maintained by someone else, it should preferably be officially endorsed by Hedgehog.

Here are examples of what I'm talking about (I'm not saying everything here must be included):

namespace Hedgehog.Batteries

module Gen =

  open Hedgehog

  /// Generates an int using sensible defaults (Range.exponentialBounded)
  let dInt = Gen.int <| Range.exponentialBounded()

  /// Generates a string using sensible defaults (Range.exponential 0 50, latin1)
  let dString = Gen.string (Range.exponential 0 50) Gen.latin1

  /// Generates a list using sensible defaults (Range.exponential 0 50)
  let dList g =
    g |> Gen.list (Range.exponential 0 50)

  /// Generates a list with length determined using Range.exponential lower upper
  let eList lower upper =
    Gen.list (Range.exponential lower upper)

  /// Generates a list with length determined using Range.linear lower upper
  let lList lower upper =
    Gen.list (Range.linear lower upper)

  /// Generates a list with length determined using Range.constant lower upper
  let cList lower upper =
    Gen.list (Range.constant lower upper)

  /// Generates null part of the time
  let withNull g =
    g |> Gen.option |> Gen.map (fun xOpt ->
      match xOpt with Some x -> x | None -> null)

  /// Generates a value that is not null
  let noNull g =
    g |> Gen.filter (not << isNull)

  /// Generates a value that is not equal to another value
  let notEqual other =
    Gen.filter ((<>) other)

  /// Generates a value that is not equal to another option-wrapped value
  let notEqualOpt other =
    Gen.filter (fun x -> match other with Some o -> x <> o | None -> true)


  /// Generates a 2-tuple with sorted elements.
  let sorted2 g =
    g |> Gen.map (fun (x1, x2) -> 
           let l = [x1; x2] |> List.sort
           (l.Item 0, l.Item 1))

  /// Generates a 3-tuple with sorted elements.
  let sorted3 g =
    g |> Gen.map (fun (x1, x2, x3) -> 
           let l = [x1; x2; x3] |> List.sort
           (l.Item 0, l.Item 1, l.Item 2))

  /// Generates a 4-tuple with sorted elements.
  let sorted4 g =
    g |> Gen.map (fun (x1, x2, x3, x4) -> 
           let l = [x1; x2; x3; x4] |> List.sort
           (l.Item 0, l.Item 1, l.Item 2, l.Item 3))

  /// Generates a 2-tuple with distinct elements.
  let distinct2 g =
    g |> Gen.filter (fun (x1, x2) -> x1 <> x2)

  /// Generates a 3-tuple with distinct elements.
  let distinct3 g =
    g |> Gen.filter (fun (x1, x2, x3) -> 
           [x1; x2; x3] |> List.distinct = [x1; x2; x3])

  /// Generates a 4-tuple with distinct elements.
  let distinct4 g =
    g |> Gen.filter (fun (x1, x2, x3, x4) -> 
           [x1; x2; x3; x4] |> List.distinct = [x1; x2; x3; x4])

  /// Generates a 2-tuple with strictly increasing elements
  let increasing2 g =
    g |> sorted2 |> distinct2

  /// Generates a 3-tuple with strictly increasing elements
  let increasing3 g =
    g |> sorted3 |> distinct3

  /// Generates a 4-tuple with strictly increasing elements
  let increasing4 g =
    g |> sorted4 |> distinct4

  /// Generates a tuple of datetimes where the range determines the number
  /// of days apart (positive numbers -> increasing order and vice versa).
  let dateInterval dayRange =
    gen {
      let tickRange = 
        dayRange 
        |> Range.map (fun days -> int64 days * System.TimeSpan.TicksPerDay)
      let! ticksApart = Gen.integral tickRange
      let! dt1 = Gen.dateTime |> Gen.filter (fun dt -> 
        dt.Ticks + ticksApart > System.DateTime.MinValue.Ticks
        && dt.Ticks + ticksApart < System.DateTime.MaxValue.Ticks)
      let dt2 = dt1.AddTicks ticksApart
      return dt1, dt2
    }


  /// Generates a list using inpGen together with a function that maps each of
  /// the distinct elements in the list to values generated by outGen. Distinct
  /// elements in the input list may possibly map to the same values. The generated
  /// function throws if called with values not in the input list.
  let withMapTo (outGen:Gen<'b>) (inpGen:Gen<'a list>) : Gen<'a list * ('a->'b)> =
    gen {
      let! inputs = inpGen
      let! outputs = outGen |> Gen.list (Range.singleton inputs.Length)
      let inOutMap = List.zip inputs outputs |> Map.ofList
      return inputs, (fun x -> inOutMap.Item x)
    }

  /// Generates a list using inpGen together with a function that maps each of
  /// the distinct elements in the list to values generated by outGen. Distinct
  /// elements in the input list are guaranteed to map to distinct values. Only
  /// use this if the output space is large enough that a list of distinct output
  /// values is likely to be generated. The generated function throws if called
  /// with values not in the input list.
  let withDistinctMapTo (outGen:Gen<'b>) (inpGen:Gen<'a list>) : Gen<'a list * ('a->'b)> =
    gen {
      let! inputs = inpGen
      let! outputs =
        outGen
        |> Gen.list (Range.singleton inputs.Length)
        |> Gen.filter (fun lst -> lst = List.distinct lst)
      let inOutMap = List.zip inputs outputs |> Map.ofList
      return inputs, (fun x -> inOutMap.Item x)
    }

  /// Inserts the given element at a random place in the list
  let addElement (x:'a) (g:Gen<'a list>) =
    gen {
      let! xs = g
      let! i = Gen.integral (Range.constant 0 xs.Length)
      let l1, l2 = xs |> List.splitAt i
      return List.concat [l1; [x]; l2]
    }

Often I have the choice whether to use Gen.filter to only generate valid values in a given test, or use where after generation to skip the test for invalid values.

Is either of these preferred? Is there, for example, a performance difference? Or any other reasons why one should be preferred over the other?

Taken from QuickCheck manual:

It is important to be aware of the distribution of test cases: if test data is not well distributed then conclusions drawn from the test results may be invalid.

Thus, we could consider adding label, classify, and collect, in the Property module.

Perhaps, it will allow broader reach if we make sure that the F# version can be used with both .NET CoreStandard and Mono. ― A quick comparison between Mono and .NET Core can be found here.

It'd be nice to keep Gen (and Range after we're done with #85) in sync with Gen.hs and Range.hs.

Related to #110.

as a better alternative to our custom doctest implementation.

Benefits

1. It removes the noise from the tooltips in the API docs on all major F# IDE's and code editors.
2. It has the ability to execute code examples as part of the build process.
3. It makes it easier to have some sort of HTML documentation.

Right now, with our custom doctest implementation, we're tackling only the 2nd item.

See also #117.

A very nice property of FsCheck is that it can generate functions. I can't find this possibility in Hedgehog. Can Hedgehog generate arbitrary functions?

Or is this perhaps fairly easy to implement my own generator for?

Update: For future readers, see #125 (comment) and #125 (comment)

We currently have it setup to look like this:

.
+-- ...
+-- Hedgehog
+-- Hedgehog.Tests
+-- ...

Although, the most projects I see on GitHub use this structure:

.
+-- ...
+-- src/Hedgehog
+-- tests/Hedgehog.Tests
+-- ...

If you also agree, I will open a pull request for it.

I'm currently working with my convex hull F# example code base, and I was adding an addition operator (+) to add two convex hulls together. Since I wanted to demonstrate that hulls are monoids, I wrote this Hedgehog property:

[<Fact>]
let ``Hull addition is associative`` () = Property.check <| property {
    let! (x, y, z) =
        Range.linearBounded ()
        |> Gen.int
        |> Gen.tuple
        |> Gen.list (Range.linear 0 100)
        |> Gen.tuple3
    return (hull x + hull y) + hull z = hull x + (hull y + hull z) }

Much to my surprise, this property fails:

Test 'Ploeh.Geometry.Tests.Hull addition is associative' failed: Hedgehog.FailedException : *** Failed! Falsifiable (after 52 tests and 81 shrinks):
([(0, 0)], [(2283, 43707)], [(50767, 1); (14548, 1)])

	at Hedgehog.ReportModule.tryRaise(Report report)
	at Hedgehog.PropertyModule.check(Property`1 p)
	C:\Users\mark\Documents\Katas\Hull\Geometry.Tests\GeometryTests.fs(79,0): at Ploeh.Geometry.Tests.Hull addition is associative()

Trying to understand the problem, I sent my convex hull code to F# Interactive, and copied the repro data:

> let (x, y, z) = ([(0, 0)], [(2283, 43707)], [(50767, 1); (14548, 1)]);;

val z : (int * int) list = [(50767, 1); (14548, 1)]
val y : (int * int) list = [(2283, 43707)]
val x : (int * int) list = [(0, 0)]

> (hull x + hull y) + hull z = hull x + (hull y + hull z);;
val it : bool = true

As you can see, I defined x, y, and z as reported in the repro, but when I then evaluate the assertion, it returns true.

What am I doing wrong?

Or is this a false positive?

Consider the following test:

[<Fact>]
let ``constrain returns lower interval value if input is below interval`` () = Property.check <| property {
  let! x1, x2, x = Range.exponentialBounded() |> Gen.int |> Gen.tuple3
  let lower = min x1 x2
  where (x < lower)
  test <@ constrain (x1, x2) x = lower @> }

This fails after 100 discards. When I debug, I see that the only value that is generated, even after many tests, is 0. When I remove the where clause, I see other values from the second test run (which probably fails due to the missing conditional).

Is this a bug, or have I not understood how to use Hedgehog?

Quick question first: would you accept a PR for C# support in the F# repository, or prefer that it's a separate package?

Otherwise, these are the things (that I know of...) that are needed for nice C# integration:

• Potentially: depend on the FSharp.Core nuget package so that Hedgehog works without additional setup when installed into a C# project.
• Use CompiledNameAttribute to make the real names of the functions more usable from C# (e.g. the byte combinator is currently Gen.@byte!) This will be a binary-breaking change.
• Add extension methods for Select{Many,} etc, to allow use of query syntax with Gen.
• Figure out exactly how to expose Property to C# code. Assertions won't work in query syntax, so might need to work on something with a boolean return value.

If we write something like this:

forAll {
  printfn "%d" 123
  let! x = Gen.range 0 10
  return! x <= 11
}

It will discard every expression, as the printfn line will turn in to Zero.
edit: turns out this is not the case at all

It seems like the solution is to turn it in to the "other" type of computation expression which uses for instead of let!. I prefer the let one personally, but I don't think it matters too much. The for style of computation expression is a lot more flexible, allowing for custom operations like where which we can use instead of the Zero hack.

I'm going to start work on this now and see how I go.

The plan (if I can make it work) is to allow something like this:

forAll {
  printfn "%d" 123
  for x in Gen.int do
  for y in Gen.int do
  where x > 0 && y > 0
  yield x * y > 0
  yield someOtherThing // maybe we can even use yield for multiple asserts?
  assert x * y > 0 // or perhaps a custom operation which is yield in disguise
}

I suggest to remove the <tests> unit of measure from Property.check'. I don't see the point - it adds no clarity, and I can't see how the added type safety gives any benefit in this particular case. I think it just adds noise.

Just a suggestion. :)

Perhaps, it makes sense to have something similar with disorder-jack/src/Test/QuickCheck/Jack.hs here, in dotnet-jack, that basically maps Jack Gen combinators to FsCheck ones.

Maybe we should have a more compact README, like the one in TypeScript, or similar? I think, right now it's a bit TL;DR. Any content that we take out could be moved to specific docs/Xyz.md files.

Specially if I'm done with what I'm working on #80, this project will be more like an F# project, than a general .NET one.

After all, computation expressions like gen and property (and what I'll probably come up with #80) aren't consumable outside of F#.

I think that should be a fair rename, like we have haskell-hedgehog (instead of ghc-hedgehog) and scala-hedgehog (instead of jvm-hedgehog).

I'm trying to generate floats, but if I write Gen.float (Range.exponentialBounded()) I just get an error saying No overloads match for method 'ToBigInt'.

I have searched this repo for "float" but no usages seemed to turn up. How can I generate floats?

When writing tests for Hedgehog.Experimental I came across the need for just checking if a property holds at least some of the time. For example, when testing withNull, I create a property I run once that generates a list of, say, 1000 items and checks if it contains null.

When testing if a property holds some of the time, one might need to generate many items to avoid false negatives. But most of the time you might get a positive fairly quickly, say after 10 items and the rest of the 990 generated items are wasted. I thought about using Gen.seq instead of Gen.list, but it just generates a list and passes it to Seq.ofList and thus provides no performance benefits.

Is it possible to have Gen.seq actually generate a lazily evaluated sequence?

The reasons are explained in #109, and particularly in this comment, but the gist is basically:

• Add FSharp.Core as a dependency, and you'll make development difficult for F# developers.
• Omit FSharp.Core as a dependency, and you'll make development difficult for C# (and, I suppose, VB) developers.

The Hedgehog.CSharp NuGet Package should be identical to Hedgehog with the only exception that it also includes a NuGet package dependency on FSharp.Core as @ploeh very well suggested here.

Theoretically, this should be done using only the tooling available via Paket and FAKE, and perhaps also by adding additional step(s) to the build script.