Unit test for Concurrent Code about learning-examples HOT 13 CLOSED

Absolutely. It's one of the things that I didn't manage to talk bout in book in the end, but concurrent code should be unit tested just like any other code. ScalaCheck is especially useful in detecting concurrency bugs, since it can easily produce a lot of different test inputs, that test different concurrent execution interleavings.

from learning-examples.

And I'm glad you liked the book!

from learning-examples.

I particular like that the book isn't too theoretical. It has a bunch of code snippets with all necessary theory, which makes understanding of the material pretty good.

from learning-examples.

Yes, that was one of the goals - to introduce various concurrency topics through hands-on examples. Very happy to hear it worked for you! :)

from learning-examples.

@axel22 do you have any good resource or tutorial in mind regarding this topic?

I'm looking on your test to Ctrie but cannot understand why it is valid to update a value then assert it given that the code is multithreaded?

from learning-examples.

The code is not multithreaded if you call update from thread T, and then call assert after that from that same thread T (see Chapters 2 and 3: there is a happens-before relationship between updating a value in the Ctrie, and reading it - if you write to a Ctrie, you will immediately see the update if you read on the same thread).

The advantage of a concurrent data structure, such as a Ctrie, is that if the code is multithreaded, then the changes made by one thread are guaranteed to be visible to another thread.

The ScalaCheck book should be a good resource (otherwise, check out http://www.scalacheck.org/):

http://www.artima.com/shop/scalacheck

For Ctries, we have checks in the standard library already:

https://github.com/scala/scala/blob/2.11.x/test/files/scalacheck/Ctrie.scala

from learning-examples.

Thanks for @axel22 reply. I wrote a simple test to your code in chapter 3. Atomic Stack.

Here's a test

"atomic stack in for each " in {
      val ct = AtomicStack

      class Updater extends Thread {
        override def run() {
          for (i <- 0 until 1000) {
            ct.push(i)
            assert(ct.pop() == Some(i))
          }
        }
      }

      val threads = for (i <- 0 until 10) yield new Updater()
      threads.foreach(_.start())
      threads.foreach(_.join())
    }

It fails with the following message. The interesting thing is that test are non-deterministic. It fails in 1 out of 3 times.

Given that AtomicStack is thread safe implemented and I'm asserting value on the same thread, then what is going wrong?

com.intellij.rt.execution.application.AppMain org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestRunner -s RandomTest -showProgressMessages true -C org.jetbrains.plugins.scala.testingSupport.scalaTest.ScalaTestReporter
Testing started at 7:35 PM ...
Exception in thread "Thread-8" org.scalatest.exceptions.TestFailedException: Some(834) did not equal Some(12)
    at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:468)
    at org.scalatest.WordSpec.newAssertionFailedException(WordSpec.scala:1882)
    at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:383)
    at RandomTest$$anonfun$1$$anonfun$apply$mcV$sp$2$Updater$1$$anonfun$run$1.apply$mcVI$sp(RandomTest.scala:58)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at RandomTest$$anonfun$1$$anonfun$apply$mcV$sp$2$Updater$1.run(RandomTest.scala:56)
Exception in thread "Thread-4" org.scalatest.exceptions.TestFailedException: Some(12) did not equal Some(834)
    at org.scalatest.Assertions$class.newAssertionFailedException(Assertions.scala:468)
    at org.scalatest.WordSpec.newAssertionFailedException(WordSpec.scala:1882)
    at org.scalatest.Assertions$AssertionsHelper.macroAssert(Assertions.scala:383)
    at RandomTest$$anonfun$1$$anonfun$apply$mcV$sp$2$Updater$1$$anonfun$run$1.apply$mcVI$sp(RandomTest.scala:58)
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
    at RandomTest$$anonfun$1$$anonfun$apply$mcV$sp$2$Updater$1.run(RandomTest.scala:56)

Process finished with exit code 0

from learning-examples.

My comment about Ctries was only about the case in which there is a single thread that updates the Ctrie. In your atomic stack example, there are multiple threads pushing and popping to the stack concurrently. This means that between the call to push by the Updater thread T1, and the subsequent call to pop by the same thread T1, another thread T2 may call any number of push and pop operations. This is why your test fails.

Maybe I misunderstood your earlier question about testing Ctries. If you would like to test Ctrie operations (or atomic stack operations) when multiple threads execute an operation, then you should only call an assert statement after you are sure that all the threads have finished executing - in other words, you may only call assert after the calls to join in the above example.

from learning-examples.

@axel22 thank you! Greatly appreciate your explanation.

Almost forgot it's 31 of Dec, Happy New Year! =)

from learning-examples.

Thanks, Happy New Year! :)

from learning-examples.

One more question regarding testing for concurrent code: is there any method to determine if the concurrent realization is correctly implemented?

In the book was mentioned that it's possible to determine a deadlock with visual vm. Is there any tool or way to think about race conditions?

In terms of Atomic stack example, how can we reason about its methods are properly synchronized?

from learning-examples.

That section of the book describes how to debug concurrent code. The trouble with data races, unlike deadlocks, is that you usually see some after-effect of invalid behavior, instead of immediate failure. Backwards reasoning and log traces are often the only way to track data races.

A way to avoid (or reduce) the troubles associated with debugging is to test the code extensively. In practice, exhaustive unit tests that simulate different execution scenarios are the way to detect and localize errors early. This is why I recommended using ScalaCheck - it can conveniently create a large number of inputs for the test.

Reasoning about the correctness of the program is the third way of ensuring correctness, and it is the least developed one. Automatic verification frameworks for concurrent code are still an active research topic, but there is nothing very solid and general available. At the moment, you should reason about the correctness yourself, as is often the case in programming.

The best general advice I can give is:

• keep your concurrency utilities modular, short and self-contained, to reason about them more easily
• do heavy unit testing on self-contained concurrency modules

In the AtomicStack example, you should ensure that the items XS the thread T has pushed onto the stack are popped in the opposite order. Items YS that were pushed by another thread S should be ignored, as long as they do not change the order in which items XS are popped.

A ScalaCheck test would be something like the following:

...scalacheck for-all statement (see scalacheck docs)... {
  val stack = newStack
  val t = thread {
    for (i <- 0 until 100) stack.push(i)
    val seen = new mutable.ArrayBuffer[Int]
    while (stack.nonEmpty) seen += stack.pop()
    assert(seen.filter(x => x >= 0 && x < 100).reverse == (0 until 100))
  }
  val s = thread {
    for (i <- 1000 until 1100) stack.push(i)
  }
  t.join()
  s.join()
}

from learning-examples.

Wow, @axel22 thanks for the thoughtful answer with a decent example! I think, I finally got the point with multithreading testing.

from learning-examples.