TableDrivenPropertyCheck reports only the first failure about scalatest HOT 6 OPEN

What might make sense is to add forEvery methods that handle tables. In inspectors, forAll short circuit's at the first sign of trouble, so it is consistent that this behavior is the same in table-driven property checks. In inspectors, forEvery is the one that keeps going and reports all. I'll think a bit more about it, but it seems reasonable to add forEvery to table-driven.

from scalatest.

Thank you!
currently I have little time. I will check the detail afterward.

from scalatest.

I'd like to cast another vote in support of a non-short-circuiting alternative to forAll. I've worked on a couple of projects where I've written patches that break a table-driven property check, but the correct diagnosis of the problem would have been easier if I had a complete list of the failing rows rather than only the first one.

from scalatest.

Thanks for posting your comment. I think adding forEvery makes sense. We will plan to add that in a future release.

from scalatest.

This enhancement has just landed in master. Can you take a look at the error message and let us know if you think it could be improved. We did what we do in Inspector's forEvery error message, except in the table-driven case each error message is more verbose. Scroll down and you can see an example error message:

console
[info] Starting scala interpreter...
[info]
import org.scalatest._
import org.scalactic._
import Matchers._
Welcome to Scala version 2.11.4 (Java HotSpot(TM) 64-Bit Server VM, Java 1.6.0_65).
Type in expressions to have them evaluated.
Type :help for more information.

scala> import prop.TableDrivenPropertyChecks._
import prop.TableDrivenPropertyChecks._

scala> val examples =
| Table(
| ("n", "d", "q"),
| (1.0, 2.0, 0.5),
| (1.0, 3.0, 0.333),
| (1.0, 4.0, 0.25),
| (1.0, 5.0, 0.2),
| (1.0, 6.0, 0.1667),
| (1.0, 7.0, 0.1429)
| )
examples: org.scalatest.prop.TableFor3[Double,Double,Double] = TableFor3((n,d,q), (1.0,2.0,0.5), (1.0,3.0,0.333), (1.0,4.0,0.25), (1.0,5.0,0.2), (1.0,6.0,0.1667), (1.0,7.0,0.1429))

scala> forAll (examples) { (n, d, q) => n / d shouldEqual q }
org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (:24)
Message: 0.3333333333333333 did not equal 0.333
Location: (:24)
Occurred at table row 1 (zero based, not counting headings), which had values (
n = 1.0,
d = 3.0,
q = 0.333
)
at org.scalatest.prop.TableFor3$$anonfun$apply$12.apply(TableFor1.scala:602)
at org.scalatest.prop.TableFor3$$anonfun$apply$12.apply(TableFor1.scala:593)
at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:778)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:777)
at org.scalatest.prop.TableFor3.apply(TableFor1.scala:593)
at org.scalatest.prop.TableDrivenPropertyChecks$class.forAll(TableDrivenPropertyChecks.scala:420)
at org.scalatest.prop.TableDrivenPropertyChecks$.forAll(TableDrivenPropertyChecks.scala:1110)
... 43 elided
Caused by: org.scalatest.exceptions.TestFailedException: 0.3333333333333333 did not equal 0.333
at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160)
at org.scalatest.Matchers$AnyShouldWrapper.shouldEqual(Matchers.scala:6963)
at $anonfun$1.apply(:24)
at $anonfun$1.apply(:24)
at org.scalatest.prop.TableFor3$$anonfun$apply$12.apply(TableFor1.scala:595)
... 51 more

scala> forEvery (examples) { (n, d, q) => n / d shouldEqual q }
org.scalatest.exceptions.TestFailedException: forEvery failed, because:
org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (:6)
Message: 0.3333333333333333 did not equal 0.333
Location: (:24)
Occurred at table row 1 (zero based, not counting headings), which had values (
n = 1.0
d = 3.0
q = 0.333 ),
org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (:6)
Message: 0.16666666666666666 did not equal 0.1667
Location: (:24)
Occurred at table row 4 (zero based, not counting headings), which had values (
n = 1.0
d = 6.0
q = 0.1667 ),
org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (:6)
Message: 0.14285714285714285 did not equal 0.1429
Location: (:24)
Occurred at table row 5 (zero based, not counting headings), which had values (
n = 1.0
d = 7.0
q = 0.1429 )
at org.scalatest.prop.TableDrivenPropertyChecks$class.doForEvery(TableDrivenPropertyChecks.scala:681)
at org.scalatest.prop.TableDrivenPropertyChecks$.doForEvery(TableDrivenPropertyChecks.scala:1110)
at org.scalatest.prop.TableDrivenPropertyChecks$class.forEvery(TableDrivenPropertyChecks.scala:738)
at org.scalatest.prop.TableDrivenPropertyChecks$.forEvery(TableDrivenPropertyChecks.scala:1110)
... 43 elided
Caused by: org.scalatest.exceptions.TableDrivenPropertyCheckFailedException: TestFailedException was thrown during property evaluation. (:6)
Message: 0.3333333333333333 did not equal 0.333
Location: (:24)
Occurred at table row 1 (zero based, not counting headings), which had values (
n = 1.0
d = 3.0
q = 0.333 )
at org.scalatest.prop.TableDrivenPropertyChecks$class.runAndCollectErrorMessage$1(TableDrivenPropertyChecks.scala:648)
at org.scalatest.prop.TableDrivenPropertyChecks$class.doForEvery(TableDrivenPropertyChecks.scala:679)
... 46 more
Caused by: org.scalatest.exceptions.TestFailedException: 0.3333333333333333 did not equal 0.333
at org.scalatest.MatchersHelper$.newTestFailedException(MatchersHelper.scala:160)
at org.scalatest.Matchers$AnyShouldWrapper.shouldEqual(Matchers.scala:6963)
at $anonfun$1.apply(:24)
at $anonfun$1.apply(:24)
at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:35)
at scala.Function3$$anonfun$tupled$1.apply(Function3.scala:34)
at org.scalatest.prop.TableDrivenPropertyChecks$class.runAndCollectErrorMessage$1(TableDrivenPropertyChecks.scala:642)
... 47 more

from scalatest.

FYI, we are going to cherry pick this back to 2.3.x and release it as part of 2.3.

from scalatest.