I am writing a unit test which checks whether some specific exception is thrown.

it should "raise a BadArgumentException if invalid encoding flags are given" in {
  a [BadArgumentException] shouldBe thrownBy {
    // do stuff which raises an ArithmeticException
  }
}

Due to a bug another exception is raised which is then also caught by scala tests. For finding the problem it would be grate to have the stacktrace of this unexpected exception.

I already looked at the code, and it seems simply rethrowing the unwanted exception will conflict the scalatest assertion handling. So it seem to be make more sense to either print the stacktrace when the exception is caught or to integrate it into the assertion failure message.

If this change is desired and well specified I would also be able to implement it myself.

Using scalatest_2.10-2.1.0

import org.scalatest._

class ScalaTestBugTestCase extends FunSpec {
  describe("A bug") {
    it("scalatest bug test case") {
      assert(<A>foo</A> == <A></A>) // This should cause an assertion failure, but it gets a stack overflow.
    }
  }
}

if I run the NetLogo test suite over and over again without restarting sbt, hundreds of threads are created and never cleaned up. here's a sample thread dump: https://gist.github.com/SethTisue/c015b30703de1597fe8e

I'm using Java 7 Update 25, sbt 0.12.4, ScalaTest 2.0.M6-SNAP35.

On another branch of my project, running under ScalaTest 1.8 (and Java 6), I don't get the same behavior.

(Let me know if I should try to isolate this...)

On page http://www.scalatest.org/user_guide/using_the_scalatest_maven_plugin there is pom fragment:

/.../
  <groupId>org.scalatest</groupId>
  <artifactId>scalatest-maven-plugin</artifactId>
  <version>1.0.RC2</version>
/.../

Version should changed to 1.0-RC2 (dash instead of a dot)

Simply creating a thread which causes thousands of dismissals, and then awaiting that (large) number of dismissals in the main thread, usually causes the await() call to timeout - even though all the dismissals are made well within the timeout interval.

I think I have a handle on the root cause. I posted a failing test and some analysis here: https://gist.github.com/ndwade/6621018

The Scaladoc for org.scalatest.Suite states:

def run(testName: Option[String], args: Args): Status

Runs this suite of tests.

If testName is None, this trait's implementation of this method calls these two methods on this object in this order:

runNestedSuites
runTests

However, in reality, the nested suite is run after the suite in which it is nested in!

Following up from issue #337, adding

parallelExecution in Test := false

to the build file appears to help org.scalatest.tools.HtmlReporter process the suite listed in #337:

import org.scalatest._

class ListSpec extends Spec with Matchers with ParallelTestExecution {

  object `mkString for list` {
    def `with 1, 2, 3` {
      List(1, 2, 3).mkString shouldBe "123"
    }
    def `with 4, 5, 6` {
      List(4, 5, 6).mkString(",") shouldBe "4,5,6"
    }
  }
}

However, the generated HTML report lists as many DistributedTestRunnerSuite instances as there are tests, with each instance having an empty report and all statistics set to zero.

Likewise for the above suite, though the only thing rendered is the initial test scope,
that is, mkString for list

import org.testng.annotations.Test
import org.scalatest.matchers.MustMatchers._

@Test(groups = Array("unit"))
def testScalaTest() {
  val actualValue: String = null
  actualValue must be("expected value")
}

this code produces the following fail message (see bellow).

org.scalatest.exceptions.TestFailedException: "expected value" was not null
    at org.scalatest.matchers.MustMatchers$class.newTestFailedException(MustMatchers.scala:955)
    at org.scalatest.matchers.MustMatchers$.newTestFailedException(MustMatchers.scala:10710)
    at org.scalatest.matchers.MustMatchers$MustMethodHelper$.mustMatcher(MustMatchers.scala:8982)
    at org.scalatest.matchers.MustMatchers$StringMustWrapper.must(MustMatchers.scala:9124)

This message is very confusing. It's hart to see what is expected value and what is actual value in the message. Compare it to how testng reports failures

import org.testng.annotations.Test
import org.testng.Assert._

  @Test(groups = Array("unit"))
  def testTestng() {
    val actualValue: String = null
    assertEquals(actualValue, "expected value")
  }

output from scala -classpath testng-6.2.jar

java.lang.AssertionError: expected:<expected value> but was:<null>
    at org.testng.Assert.fail(Assert.java:89)
    at org.testng.Assert.failNotEquals(Assert.java:489)
    at org.testng.Assert.assertEquals(Assert.java:118)
    at org.testng.Assert.assertEquals(Assert.java:171)
    at org.testng.Assert.assertEquals(Assert.java:181)

IDEA 12.1 can parse that exception and transforms it into even more readable form

java.lang.AssertionError: 
Expected :expected value
Actual   :null
  <Click to see difference>


    at org.testng.Assert.fail(Assert.java:94)
    at org.testng.Assert.failNotEquals(Assert.java:494)
    at org.testng.Assert.assertEquals(Assert.java:123)
    at org.testng.Assert.assertEquals(Assert.java:176)
    at org.testng.Assert.assertEquals(Assert.java:186)

Could you change failure messages in scalatest to be more human readable please?

according to sbt/sbt#228, sbt supports this, though it's not clear if it's received much use/testing

this is the number one reason I end up having to restart sbt (killing my nice warm JITted compiler)

If I have code like this:

import org.scalatest._
import org.scalatest.Matchers._
import org.scalautils._
import TripleEquals._
import Tolerance._

val r = 4
val rr = (r === 2 +- 1)

I get the error:
[error] /Users/greg.zoller/git/feedValidator/src/test/scala/br.feed/ServiceTests.scala:90: value +- is not a member of Int
[error] val rr = (r === 2 +- 1)
[error] ^

If I comment out the Matchers._ line I don't get a compile error. I'm guessing some kind of conflict between the two libraries. I'm running Scala 2.11.1, ScalaUtils 2.1.7, ScalaTest 2.2.0.

When using the selenium features in WebBrowser, tests become bogged down with excessive casting.

Consider:

class BlogSpec extends HtmlUnit {

  class HomePage extends Page {
    val url = "localhost:9000/index.html"
  }

  val homePage = new HomePage
  go to homePage
}

This works, but only because HomePage is an inner class. Now consider:

class BlogSpec extends HtmlUnit {

  val homePage = new HomePage
  go to homePage
}

class HomePage extends Page {
  val url = "localhost:9000/index.html"
}

This will not compile. Error message is: Type mismatch; expected: BlogSpec.this.type#Page, actual: HomePage.

To fix the warning, add asInstance[Page] to the homePage val:

class BlogSpec extends HtmlUnit {

  val homePage = new HomePage
  go to homePage.asInstance[Page]
}

class HomePage extends Page {
  val url = "localhost:9000/index.html"
}

This issue (and workaround) exists with many functions and behaviors, including go to and click on. It prevents reusability of PageObjects and prevents web elements to be placed in their own classes with their own properties and such.

System Information:

• Scala version: 2.10.3
• Java version: 1.7.0_25
• ScalaTest: 2.10-2.0.RC2

The following input types are new for HTML5:

color
date
datetime
datetime-local
email
month
number
range
search
tel
time
url
week

Could we please support these natively in WebBrowser please? If necessary I can do a PR.

Since scalatest 2.0, MustMatchers has been deprecated.

By deprecating MustMatchers and making people switch over from "must" to "should", the language in specifications becomes softer. For specifications, soft language is bad. Specification statements no longer read like assertions, but rather like something that the test framework might optionally attempt to assert.

I've tried to find information on the reason for the deprecation, beyond the rather fuzzy "we want to make the verb available to be used for a different purpose" in scalatest's documentation. Searching the mailing list and issue database left me none the wiser... but
I did find a twitter thread where Bill Venners had responded, saying that he planned to replace "must" with something that returned a result rather than performed an assertion. In that thread, "will" was suggested as a possible (unused) word for this new result-returning verb: https://twitter.com/viktorklang/status/412992663236124672

That left me somewhat hopeful... but I now see that MustMatchers remain deprecated in the upcoming 2.1.0 release, so I'm filing this issue.

(Luckily, I've discovered that org.scalatest's package object has a type alias for MustMatchers, and that has not (yet) been deprecated, so I can still have my must-using code compile without warnings.)

It appears that the zip method of Or does not exist in the source, although it appears in the documentation of how to do accumulation?

I had to add these two deps to use the selenium htmlunit driver as described here:
http://www.scalatest.org/user_guide/using_selenium
"org.seleniumhq.selenium" % "selenium-htmlunit-driver" % "2.37.1" and "org.seleniumhq.selenium" % "selenium-support" % "2.37.1"
Not sure if this was already supposed to have been known; I've never used Selenium before - but the compiler error w/out those on the classpath wasn't very friendly.

Given the following suite definition:

@org.scalatest.WeakAsAKitten
class BaseKittenSuite extends FunSuite {}

class TagAnnotationInSuperClassSuite extends BaseKittenSuite {
test('assert good") { assert(1 === 1) }
}

I would expect that running the following in sbt:

testOnly *.TagAnnotationInSuperClassSuite -- -n org.scalatest.WeakAsAKitten

Should result in one test failing. However, it runs 0 tests because it skips the tagged test suite due to not searching super classes for the indicated annotation.

When I run a failing test in my project which uses nested inside I get an error something like this:

"[ ]Comment" was not equal to "[]Comment", inside List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,None,Some( Comment)), SingleTransaction(Bank,None,None)), inside Transaction(Date(2013,6,1),List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,None,Some( Comment)), SingleTransaction(Bank,None,None)),None,None,List()), inside List(Transaction(Date(2013,3,1),List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,Some(NumericLiteralExpr(-100)),None), SingleTransaction(Bank,None,None)),None,None,List()), Transaction(Date(2013,6,1),List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,None,Some( Comment)), SingleTransaction(Bank,None,None)),None,None,List())), inside [11.5] parsed: List(Transaction(Date(2013,3,1),List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,Some(NumericLiteralExpr(-100)),None), SingleTransaction(Bank,None,None)),None,None,List()), Transaction(Date(2013,6,1),List(SingleTransaction(Expense,Some(NumericLiteralExpr(200)),None), SingleTransaction(Cash,None,Some( Comment)), SingleTransaction(Bank,None,None)),None,None,List())) (ParserTest.scala:124)

The objects of the nested insides are printed on a single line which is very difficult to parse for humans. It is probably possible to show each nesting level of the inside on a different line and indentation level, which would significantly improve the usability.

For reference my test looks like this:

inside(parseResult) {
      case AbandonParser.Success(result, _) =>
        inside(result) {
          case List(txnGroup1, txnGroup2) =>
            inside(txnGroup1) {
              case Transaction(date, txns, None, None, Nil) =>
                date should be(Date(2013, 3, 1))
                inside(txns) {
                  case List(SingleTransaction(acc1, expr1, _), SingleTransaction(acc2, expr2, _), SingleTransaction(acc3, expr3, _)) =>
                    acc1 should be (AccountName(Seq("Expense")))
                    acc2 should be (AccountName(Seq("Cash")))
                    acc3 should be (AccountName(Seq("Bank")))
                    expr1 should be (Some(NumericLiteralExpr(200)))
                    expr2 should be (Some(NumericLiteralExpr(-100)))
                    expr3 should be (None)
                }
            }
            inside(txnGroup2) {
              case Transaction(date, txns, None, None, Nil) =>
                date should be(Date(2013, 6, 1))
                inside(txns) {
                  case List(SingleTransaction(acc1, expr1, None), SingleTransaction(acc2, expr2, Some(comment)), SingleTransaction(acc3, expr3, None)) =>
                    acc1 should be (AccountName(Seq("Expense")))
                    acc2 should be (AccountName(Seq("Cash")))
                    acc3 should be (AccountName(Seq("Bank")))
                    expr1 should be (Some(NumericLiteralExpr(200)))
                    expr2 should be (None)
                    expr3 should be (None)
                    comment should be ("Comment")
                }
            }
        }
    }

http://www.scalatest.org/getting_started_with_flat_spec returns a page that says

Action not found
For request 'GET /getting_started_with_flat_spec'

What ever I do like:
etl/test:testOnly *EventEtlJob* -- -oDF or etl/test:testOnly *EventEtlJob* -- -oW ... option are not take into account.

The weird thing is that if I have a o defined in testOptions I get:
java.lang.IllegalArgumentException: Only one -o allowed

The test case below will result in a RuntimeException: Unable to load a Suite class that was discovered in the runpath: scalatest_issue.TargetRefTest2

Output:

WARNING: -p has been deprecated and will be reused for a different (but still very cool) purpose in ScalaTest 2.0. Please change all uses of -p to -R.
Discovery starting.
*** RUN ABORTED ***
  java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: scalatest_issue.TargetRefTest2
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:84)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:38)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:37)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
  at scala.collection.Iterator$class.foreach(Iterator.scala:743)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1174)
  at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
  at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
  ...

Scala 2.11.1
ScalaTest 2.1.7

package scalatest_issue

// The following test will not run with Scala 2.11.1 and ScalaTest 2.1.7

import org.scalatest.FreeSpec

object TargetRefs2 {
  def apply(targetRefs: String*): TargetRefs2 = new TargetRefs2(Seq(targetRefs))
}

class TargetRefs2 private (val targetRefGroups: Seq[Seq[String]]) {

  private[this] def closedGroups: Seq[Seq[String]] = targetRefGroups.size match {
    case 1 => Seq()
    case n => targetRefGroups.take(n - 1)
  }

  private[this] def openGroup: Seq[String] = targetRefGroups.lastOption match {
    case Some(last) => last
    case None => Seq()
  }

  def ~(targetRefs: TargetRefs2): TargetRefs2 =
    new TargetRefs2((
      closedGroups ++
      Seq((openGroup ++ targetRefs.targetRefGroups.head).distinct) ++
      targetRefs.targetRefGroups.tail
    ).filter(!_.isEmpty))
  def ~(string: String): TargetRefs2 = this.~(TargetRefs2(string))

  def ~~(targetRefs: TargetRefs2): TargetRefs2 =
    new TargetRefs2((
      targetRefGroups ++
      Seq(targetRefs.targetRefGroups.head) ++
      targetRefs.targetRefGroups.tail
    ).filter(!_.isEmpty))
  def ~~(string: String): TargetRefs2 = ~~(TargetRefs2(string))

  override def toString: String = targetRefGroups.map { _.mkString(" ~ ") }.mkString(" ~~ ")

}

class TargetRefTest2 extends FreeSpec {

  "TargetRefs2 merge" - {

    def mergeTest(refs: TargetRefs2, expected: Seq[Seq[String]]): Unit = {
      s"merge ${refs} to ${expected}" in {
        assert(refs.targetRefGroups === expected)
      }
    }
    mergeTest(TargetRefs2("a"), Seq(Seq("a")))
    mergeTest(TargetRefs2("a") ~ "b", Seq(Seq("a", "b")))
    mergeTest(TargetRefs2("a") ~ "a", Seq(Seq("a"))) 
    mergeTest(TargetRefs2("a") ~~ "a", Seq(Seq("a"), Seq("a")))
  }

}

When I apply one of the minimal changes below, the test will run!

• remove the .distinct in line 26, unfortunately now test case in line 54 fails
• remove line 39, unfortunately, now I have no toString method anymore
• remove line 54, unfortunately, now I don't test the expected behaviour

See also this gist

org.scalatest.tools.HtmlReporter completes abruptly when used with suites mixing in the ParallelTestExecution trait. In a project containing only the following test:

import org.scalatest._

class ListSpec extends Spec with Matchers with ParallelTestExecution {

  object `mkString for list` {
    def `with 1, 2, 3` {
      List(1, 2, 3).mkString shouldBe "123"
    }
    def `with 4, 5, 6` {
      List(4, 5, 6).mkString(",") shouldBe "4,5,6"
    }
  }
}

running sbt test with testOptions in Test += Tests.Argument("-h", "target/html-reports") produces this exception:

Reporter completed abruptly with an exception after receiving event: SuiteCompleted(Ordinal(0, 8, 3),DistributedTestRunnerSuite,org.scalatest.tools.DistributedTestRunnerSuite,Some(org.scalatest.tools.DistributedTestRunnerSuite),Some(27),Some(MotionToSuppress),Some(TopOfClass(org.scalatest.tools.DistributedTestRunnerSuite)),None,None,pool-4-thread-1,1401819440674).
java.lang.IllegalStateException: Expected SuiteStarting for completion event: SuiteCompleted(Ordinal(0, 8, 3),DistributedTestRunnerSuite,org.scalatest.tools.DistributedTestRunnerSuite,Some(org.scalatest.tools.DistributedTestRunnerSuite),Some(27),Some(MotionToSuppress),Some(TopOfClass(org.scalatest.tools.DistributedTestRunnerSuite)),None,None,pool-4-thread-1,1401819440674) in the head of suite events, but we got no suite event at all
    at org.scalatest.tools.HtmlReporter.apply(HtmlReporter.scala:948)
    at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
    at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
    at scala.collection.immutable.List.foreach(List.scala:383)
    at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)
    at java.lang.Thread.run(Thread.java:695)

Are there other test options to make the above example work?

I've written a trait that extends WebBrowser to support Appium. However, I need to inject two dependencies: the web driver server and the application path, which are required to instantiate the web driver. The best solution seems to be defining the vals in a class extending the trait. However, I can't get at those during instantiation. What do you think would be the best way to inject these dependencies into a subtrait of WebBrowser?

I'm comparing two traversables s and l where s is a set an l is a list and both contain an element e . I want to write a test that all elemets of s are contained in l. Something like "s should be (l)". But this test fails with a message org.scalatest.exceptions.TestFailedException: Set(e) was not equal to List(e)

Details and a solution defining a helper method sameAs can be found at stackoverflow http://stackoverflow.com/questions/7434762/comparing-collection-contents-with-scalatest

But I think it would be better to have a matcher for this.

It seems for me that Selenide is getting very similar to ScalaTest philosophy.

My assumption that WebBrowser might wrap some syntax of selenide to have it scalaway, the rest of "should" Selenide can worry about.

@asolntsev, @bvenners, what do you think?

Inspired by: Selenium Camp 2014

I am experiencing very strange behaviour of the list generator. The following test case does not always produce arrays of size 10000. In case it starts by producing arrays of the wanted size but the size drops down in every following step.

  val SkewnessAccuracy = 1E-7

      def mom( seq: Seq[Double], p: Int, around: Double) = seq.map( x => math.pow( x - around, p) ).sum / seq.length

      def skewness( seq: Seq[Double]): Double = {
        val avg = mom(seq, 1, 0)
        val num = mom(seq, 3, avg)
        val denom = math.pow(mom(seq, 2, avg), 1.5)
        num / denom
      }

      forAll((
        Gen.containerOfN[Array, Double]( 10000, Gen.choose(ElemLowerLimit, ElemUpperLimit)), "sequence")) {
        seq: Array[Double] => {
            println( seq.length )
            val skew = seq.foldLeft( OnlineMoments() ) { (acc,x) => acc :+ x }.skewness
            seq match {
              case xs if xs.length < 3 => skew should be(None)
              case xs => skewness(xs) should be(skew.get +- SkewnessAccuracy)
          }
        }
      }

Any explanations/ fixes ?

The documentation (ScalaDoc) for WebBrowser trait claims that execute script returns:
One of Boolean, Long, String, List or WebElement. Or null

In my test, using HtmlUnit and executing code that returns an object, executeScript returned a HashMap.

Here's my code if you want to try it.
https://github.com/jessitron/js-with-scalacheck/blob/master/src/test/scala/summarizeObject.scala
(line 65)

The question is, does it behave the same way with other WebDrivers? How wrong is the doc?

When I write test like follows, scalatest reports only the first failure among table rows.

it("description") {
  forAll(table) {
    ...
  }
}

In contrast, groovy's spock reports all failure in parameterized test (http://code.google.com/p/spock/wiki/Parameterizations).
I prefer this style because it can provide more information per test run.
I do not care about the case of exceptions (not test assertion failure).

How can I achieve such behavior without modification?
If impossible, can I send a patch?

We've got a test to check the presence- and loading of native libraries. When the library can not be loaded, it throws an UnsatisfiedLinkError.

Scalatest runs by the Maven Scalatest plugin, on our Jenkins environment. The bottom line is that the build succeeds because no result xml files were written. Presumably because the error stopped Scalatest in its tracks. Preventing it from writing the result files.

Throwing a regular RuntimeException does not prevent xml result files to be written.

According to http://www.scala-sbt.org/release/docs/Detailed-Topics/Testing, filtering is supposed to be applied to classes:

If you want to only run test classes whose name ends with "Test", use Tests.Filter:

E.g. junit-interface works this way. In ScalaTest, it is instead applied to test names.

This is also inconsistent with testOnly behaviour, which looks at the class name.

Using:

scalaVersion := "2.10.4"

        "org.scalacheck"          %% "scalacheck"         % "1.11.3"                % "test"
        "org.scalatest"           %% "scalatest"          % "2.1.3"                 % "test"

This is the actual stacktrace :

java.io.NotSerializableException: org.scalacheck.util.FreqMap$$anon$1
...
at org.scalatest.tools.SocketReporter.apply(SocketReporter.scala:31)
at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:240)
at org.scalatest.DispatchReporter$Propagator$$anonfun$run$1.apply(DispatchReporter.scala:239)
at scala.collection.immutable.List.foreach(List.scala:318)
at org.scalatest.DispatchReporter$Propagator.run(DispatchReporter.scala:239)

Furthermore, this breaks the reporter, any further test gives:

java.net.SocketException: Software caused connection abort: socket write error
...
at org.scalatest.tools.SocketReporter.apply(SocketReporter.scala:31)

My understanding of the issue is that an Event (which should be Serializable) holds a reference to a payload of type Option[Any] which may not be Serializable.

Possible fixes include :

• in SocketReporter.apply, check if the event payload is serializable.
• in Event, change the payload type to Option[Serializable]

Since the initialization of a test suite can start heavy things like thread pools (e.g. for Akka) it is crucial that afterAll runs as well, independent of what happened between these two events. The Akka test suite currently has the problem that it leaks actor systems due to this (and thereby class loaders => runs out of PermGen space). We can work around it by adding a dummy test case for now.

In plain old scalacheck, I can nest forAll calls with impunity. I get the desired distribution of values and don't end up with exponentially more tests running. Scalatest wraps forAll and loses this property. For example, the following code will run 10000 tests in scalatest and 100 tests in scalacheck.

forAll[Int] { i =>
  forAll(Gen.choose(i, i +10)) { j =>
    // something with i and j
  }
}

Of course, it's always possible to rework any nested forAll calls into flat ones by bundling generators into tuples (essentially uncurrying) but that leads to much more verbosity in properties.

It seems hard to restore the property given the exception-based Unit-returning properties that scalatest promotes, but it also seems desirable.

when I try to use ScalaTest 2.0 and ScalaCheck 1.11.0 together, I get:

[error] bad symbolic reference. A signature in Configuration.class refers to type Params
[error] in object org.scalacheck.Test which is not available.
[error] It may be completely missing from the current classpath, or the version on
[error] the classpath might be incompatible with the version used when compiling Configuration.class.
[error] one error found
[error] (test:compile) Compilation failed

if I should isolate/minimize this, let me know, I'd be happy to do so.

SBT's results differ from ScalaTests's results when using nested suites.
I'm making a big assumption here that ScalaTest is providing incorrect results but I don't know that that's fact.

My setup:

class VanillaSuite extends Suites(new A, new B, new C)

class ParalellisableDbTests extends Suites(new X, new Y, new Z)
class DbSuite extends Stepwise(
  new ParalellisableDbTests,
  new F, new G, new H
)

In SBT I have: testOptions in Test := Seq(Tests.Filter(_.matches(".*Suite$")))

In my console, when I run tests, ScalaTest says:

[info] ScalaTest
[info] Total number of tests run: 416
[info] Suites: completed 38, aborted 0
[info] Tests: succeeded 416, failed 0, canceled 0, ignored 0, pending 0

SBT says:

[info] Passed: Total 7, Failed 0, Errors 0, Passed 7

I expect SBT to say:

[info] Passed: Total 416, Failed 0, Errors 0, Passed 416

Environment:

• scalatest 2.2.0
• Scala 2.11.1

Bug

0l should be >= (0)

Does not compile:

overloaded method value should with alternatives:
[error]   (endWithWord: org.scalatest.words.EndWithWord)(implicit ev: <:<[Long,String])HyperLogLogCommandsSpec.this.ResultOfEndWithWordForString <and>
[error]   (startWithWord: org.scalatest.words.StartWithWord)(implicit ev: <:<[Long,String])HyperLogLogCommandsSpec.this.ResultOfStartWithWordForString <and>
[error]   (includeWord: org.scalatest.words.IncludeWord)(implicit ev: <:<[Long,String])HyperLogLogCommandsSpec.this.ResultOfIncludeWordForString <and>
[error]   (notExist: org.scalatest.words.ResultOfNotExist)(implicit existence: org.scalatest.enablers.Existence[Long])Unit <and>
[error]   (existWord: org.scalatest.words.ExistWord)(implicit existence: org.scalatest.enablers.Existence[Long])Unit <and>
[error]   (containWord: org.scalatest.words.ContainWord)org.scalatest.words.ResultOfContainWord[Long] <and>
[error]   (haveWord: org.scalatest.words.HaveWord)HyperLogLogCommandsSpec.this.ResultOfHaveWordForExtent[Long] <and>
[error]   (beWord: org.scalatest.words.BeWord)HyperLogLogCommandsSpec.this.ResultOfBeWordForAny[Long] <and>
[error]   (inv: org.scalactic.TripleEqualsSupport.TripleEqualsInvocationOnSpread[Long])(implicit ev: Numeric[Long])Unit <and>
[error]   [U](inv: org.scalactic.TripleEqualsSupport.TripleEqualsInvocation[U])(implicit constraint: org.scalactic.Constraint[Long,U])Unit <and>
[error]   (notWord: org.scalatest.words.NotWord)org.scalatest.words.ResultOfNotWordForAny[Long] <and>
[error]   [TYPECLASS1[_], TYPECLASS2[_]](rightMatcherFactory2: org.scalatest.matchers.MatcherFactory2[Long,TYPECLASS1,TYPECLASS2])(implicit typeClass1: TYPECLASS1[Long], implicit typeClass2: TYPECLASS2[Long])Unit <and>
[error]   [TYPECLASS1[_]](rightMatcherFactory1: org.scalatest.matchers.MatcherFactory1[Long,TYPECLASS1])(implicit typeClass1: TYPECLASS1[Long])Unit <and>
[error]   (rightMatcherX1: org.scalatest.matchers.Matcher[Long])Unit
[error]  cannot be applied to (org.scalatest.matchers.Matcher[Int])
[error]         0l should be >= (0)

This works fine:

0l.toInt should be >= (0)

I use Or at our system edges quite a bit. This means that it is a common need to transform from Or[G, B] to Or[G, H]. transform is capable, but means supplying a transform for good, which is un-needed. It would be very useful to have something built in to do this. For reference, currently I use this very simple implementation:

  implicit class OrPimps[G, B](wrapped: G Or B) {
    def bmap[H](f: B => H): G Or H = wrapped.swap.map(f).swap
  }

E.g.: Here http://www.scalatest.org/user_guide/tests_as_specifications with:

  when("when pop is invoked on the stack")

Becomes:

  + When when pop is invoked on the stack 

org.scalatest.tools.HtmlReporter uses default encoding to write html reports.
It should use UTF-8.

Change makeSuiteFile method.

From:

val pw = new PrintWriter(new BufferedOutputStream(new FileOutputStream(new File(targetDir, name + ".html")), BufferSize))

To:

val pw = new PrintWriter(new OutputStreamWriter(new BufferedOutputStream(new FileOutputStream(new File(targetDir, name + ".html")), BufferSize), "UTF-8"))

And change makeIndexFile method.

From:

val pw = new PrintWriter(new BufferedOutputStream(new FileOutputStream(new File(targetDir, "index.html")), BufferSize))

To:

val pw = new PrintWriter(new OutputStreamWriter(new BufferedOutputStream(new FileOutputStream(new File(targetDir, "index.html")), BufferSize), "UTF-8"))

private def neverRuns(f: => Unit): Boolean = true

@Test
def sadTimes() = {
  assert(neverRuns(sys.error("Sad times")))
}

With regular Predef.assert, this passes, of course. With the Assertions.assert macro, it throws the RuntimeException.

I think that JunitXMLReporter is generating an XML that is not being recognized by Jenkins.
It looks like the issue is that there may be space (or newline) between <property> and </property>.

One solution is to use <property name="path.separator" value=";" />, ending with />

While trying to figure out what changed between 1.9.1 and 1.9.2-SNAP1, I discovered there are no tags in this repo. Could you please tag releases?

(Note that github has a nice "releases" tab now, that allows you add release notes and downloads to each tag.)

Thanks!

When I supply a have a generator for a custom type, and I write a Pretty for it and supply that to the property, it doesn't get used in the exception printouts from Checkers.

Following code: https://gist.github.com/arturaz/8180793#file-droidtest-scala-L170

Generates this error: https://gist.github.com/arturaz/8180793/raw/c021b705a88207b69373b52225b0ddc02b18f8d9/gistfile1.txt

Scalatest version: https://gist.github.com/arturaz/8180793#file-serverbuild-scala-L36

import org.scalatest.concurrent.{AsyncAssertions, ScalaFutures}
import org.scalatest.{Matchers, FunSpec}
import scala.concurrent.Future
import scala.concurrent.ExecutionContext.Implicits._

class WaiterDismissalTestSuite extends FunSpec with ScalaFutures with Matchers with AsyncAssertions {
  describe("Waiter") {
    it("should wait") {
      val w = new Waiter
      Future {
        w.dismiss()
        w.dismiss()
      }
      w.await(dismissals(1))
    }
  }
}

Produces the following stacktrace:

assertion failed: unreachable condition - maybe time went backwards?!
java.lang.AssertionError: assertion failed: unreachable condition - maybe time went backwards?!
    at scala.Predef$.assert(Predef.scala:179)
    at org.scalatest.concurrent.AsyncAssertions$Waiter.awaitImpl(AsyncAssertions.scala:400)
    at org.scalatest.concurrent.AsyncAssertions$Waiter.await(AsyncAssertions.scala:498)

I think we are falling through the condition on this line:

Are there any reasons we chose to do dismissedCount == dismissals instead of dismissedCount >= dismissals?

Here's a tiny self-contained example that fails on Scalatest 2.0 with scalac 2.10.2:

class Bug extends FunSuite {
  def woof(f: => Unit) = "woof"
  def meow(x: Int = 0, y: Int = 3) = "meow"

  test("bug") {
    assert(woof { meow(y = 5) } == "ugh")
  }
}

The error is a massive beast showing internal AST representations. Ultimately, it seems that the assert macro doesn't like me not specifying that x optional parameter inside a closure in an equality assertion. I haven't dug deeper into the macro code, but I'd imagine it was just an oversight in the AST processing code?

On page http://scalatest.org/user_guide/using_the_scalatest_maven_plugin
plugin version should be 1.0-RC2
but not 1.0.RC2

I've been looking into an issue where Jenkins reports a build as successful when a test suite fails with "RUN ABORTED" exception.

For example add the following test file to a Maven project:

import org.scalatest.{MustMatchers, WordSpecLike}

class AbortedTest extends WordSpecLike with MustMatchers
{
  throw new Error("Exception during construction to demonstrate issue...")

  "A scalatest suite" must
    {
      "fail when run from Jenkins" in
        {
          true must equal (false)
        }
    }
}

Then kickoff the build from Jenkins:

Apr 18, 2014 11:04:06 AM org.apache.maven.cli.event.ExecutionEventLogger mojoStarted
INFO: --- scalatest-maven-plugin:1.0-RC2:test (scala-integration-test) @ cse-util-common ---
Discovery starting.
*** RUN ABORTED ***
  java.lang.Error: Exception during construction to demonstrate issue...
  at AbortedTest.(AbortedTest.scala:10)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
  at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
  at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
  at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
  at java.lang.Class.newInstance(Class.java:374)
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:69)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:38)
  at org.scalatest.tools.DiscoverySuite$$anonfun$1.apply(DiscoverySuite.scala:37)
  at scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
  ...
[JENKINS] Recording test results
  ...
[INFO] Deployment done in 0.47 sec
Finished: SUCCESS

However, if I run the build manually on the command line (using the same build parameters passed by Jenkins) it fails as expected:

*** RUN ABORTED ***
  java.lang.Error: Exception during construction to demonstrate issue...
  at AbortedTest.(AbortedTest.scala:10)
 ...
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 39.297s
[INFO] Finished at: Fri Apr 18 11:32:31 MDT 2014
[INFO] Final Memory: 22M/256M
[INFO] ------------------------------------------------------------------------

So it seems this issue only arises when building a Maven project from Jenkins which uses the scalatest maven plugin. It's hard to tell if this is an issue with Jenkins or the ScalatTest Maven plugin, but I'm guessing the problem could be at least worked around with a small change to the scalatest-maven-plugin. In particular, it seems related to issue #273: "The bottom line is that the build succeeds because no result xml files were written."

I plan to also open an issue for Jenkins and cross-reference this ticket in hopes that we can see a fix or workaround in the near future.

I'm trying to use path.FreeSpec with Akka. Because Akka's TestKit is a class, I have to use the trait version of path.FreeSpec: path.FreeSpecLike. However, at runtime I get an exception when the test class is cast to path.FreeSpec:

... cannot be cast to org.scalatest.path.FreeSpec

I was able to fix the problem locally by changing the cast to use the trait instead of the class at line 51 of FreeSpecLike.java:

override def newInstance: FreeSpecLike = this.getClass.newInstance.asInstanceOf[FreeSpecLike]

This problem was also in FunSpecLike and I suspect it's in the other traits.

I'm using ScalaTest 2.1.5, Scala 2.11.0, Akka 2.3.2.

scalatest version 2.0

class ASpec extends WordSpecLike with Matchers with BeforeAndAfterAll {

  override def afterAll {
    println("afterAll")
  }

  "test" must {
    "call afterAll" ignore {}
  }
}