org.scalatest.FunSuite.scala Maven / Gradle / Ivy
Show all versions of scalatest_2.8.0 Show documentation
/*
* Copyright 2001-2008 Artima, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.scalatest
import scala.collection.immutable.ListSet
import java.util.ConcurrentModificationException
import java.util.concurrent.atomic.AtomicReference
import org.scalatest.StackDepthExceptionHelper.getStackDepth
import org.scalatest.events._
import Suite.anErrorThatShouldCauseAnAbort
/**
* A suite of tests in which each test is represented as a function value. The “Fun” in FunSuite stands
* for “function.” Here's an example FunSuite:
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* test("addition") {
* val sum = 1 + 1
* assert(sum === 2)
* assert(sum + 2 === 4)
* }
*
* test("subtraction") {
* val diff = 4 - 1
* assert(diff === 3)
* assert(diff - 2 === 1)
* }
* }
*
*
*
* “test” is a method, defined in FunSuite, which will be invoked
* by the primary constructor of MySuite. You specify the name of the test as
* a string between the parentheses, and the test code itself between curly braces.
* The test code is a function passed as a by-name parameter to test, which registers
* it for later execution. One benefit of FunSuite compared to Suite is you need not name all your
* tests starting with “test.” In addition, you can more easily give long names to
* your tests, because you need not encode them in camel case, as most people would tend to do
* for test method names.
*
*
*
* A FunSuite's lifecycle has two phases: the registration phase and the
* ready phase. It starts in registration phase and enters ready phase the first time
* run is called on it. It then remains in ready phase for the remainder of its lifetime.
*
*
*
* Tests can only be registered with the test method while the FunSuite is
* in its registration phase. Any attempt to register a test after the FunSuite has
* entered its ready phase, i.e., after run has been invoked on the FunSuite,
* will be met with a thrown TestRegistrationClosedException. The recommended style
* of using FunSuite is to register tests during object construction as is done in all
* the examples shown here. If you keep to the recommended style, you should never see a
* TestRegistrationClosedException.
*
*
* Shared fixtures
*
*
* A test fixture is objects or other artifacts (such as files, sockets, database
* connections, etc.) used by tests to do their work. You can use fixtures in
* FunSuites with the same approaches suggested for Suite in
* its documentation. The same text that appears in the test fixture
* section of Suite's documentation is repeated here, with examples changed from
* Suite to FunSuite.
*
*
*
* If a fixture is used by only one test, then the definitions of the fixture objects can
* be local to the test function, such as the objects assigned to sum and diff in the
* previous MySuite examples. If multiple tests need to share a fixture, the best approach
* is to assign them to instance variables. Here's a (very contrived) example, in which the object assigned
* to shared is used by multiple test functions:
*
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* // Sharing immutable fixture objects via instance variables
* val shared = 5
*
* test("addition") {
* val sum = 2 + 3
* assert(sum === shared)
* }
*
* test("subtraction") {
* val diff = 7 - 2
* assert(diff === shared)
* }
* }
*
*
*
* In some cases, however, shared mutable fixture objects may be changed by tests such that
* they need to be recreated or reinitialized before each test. Shared resources such
* as files or database connections may also need to be created and initialized before,
* and cleaned up after, each test. JUnit offers methods setUp and
* tearDown for this purpose. In ScalaTest, you can use the BeforeAndAfterEach trait,
* which will be described later, to implement an approach similar to JUnit's setUp
* and tearDown, however, this approach often involves reassigning vars
* between tests. Before going that route, you should consider some approaches that
* avoid vars. One approach is to write one or more create-fixture methods
* that return a new instance of a needed object (or a tuple or case class holding new instances of
* multiple objects) each time it is called. You can then call a create-fixture method at the beginning of each
* test that needs the fixture, storing the fixture object or objects in local variables. Here's an example:
*
*
*
* import org.scalatest.FunSuite
* import scala.collection.mutable.ListBuffer
*
* class MySuite extends FunSuite {
*
* // create objects needed by tests and return as a tuple
* def createFixture = (
* new StringBuilder("ScalaTest is "),
* new ListBuffer[String]
* )
*
* test("easy") {
* val (builder, lbuf) = createFixture
* builder.append("easy!")
* assert(builder.toString === "ScalaTest is easy!")
* assert(lbuf.isEmpty)
* lbuf += "sweet"
* }
*
* test("fun") {
* val (builder, lbuf) = createFixture
* builder.append("fun!")
* assert(builder.toString === "ScalaTest is fun!")
* assert(lbuf.isEmpty)
* }
* }
*
*
*
* If different tests in the same FunSuite require different fixtures, you can create multiple create-fixture methods and
* call the method (or methods) needed by each test at the begining of the test. If every test requires the same set of
* mutable fixture objects, one other approach you can take is make them simply vals and mix in trait
* OneInstancePerTest. If you mix in OneInstancePerTest, each test
* will be run in its own instance of the FunSuite, similar to the way JUnit tests are executed.
*
*
*
* Although the create-fixture and OneInstancePerTest approaches take care of setting up a fixture before each
* test, they don't address the problem of cleaning up a fixture after the test completes. In this situation,
* one option is to mix in the BeforeAndAfterEach trait.
* BeforeAndAfterEach's beforeEach method will be run before, and its afterEach
* method after, each test (like JUnit's setUp and tearDown
* methods, respectively).
* For example, you could create a temporary file before each test, and delete it afterwords, like this:
*
*
*
* import org.scalatest.FunSuite
* import org.scalatest.BeforeAndAfterEach
* import java.io.FileReader
* import java.io.FileWriter
* import java.io.File
*
* class MySuite extends FunSuite with BeforeAndAfterEach {
*
* private val FileName = "TempFile.txt"
* private var reader: FileReader = _
*
* // Set up the temp file needed by the test
* override def beforeEach() {
* val writer = new FileWriter(FileName)
* try {
* writer.write("Hello, test!")
* }
* finally {
* writer.close()
* }
*
* // Create the reader needed by the test
* reader = new FileReader(FileName)
* }
*
* // Close and delete the temp file
* override def afterEach() {
* reader.close()
* val file = new File(FileName)
* file.delete()
* }
*
* test("reading from the temp file") {
* var builder = new StringBuilder
* var c = reader.read()
* while (c != -1) {
* builder.append(c.toChar)
* c = reader.read()
* }
* assert(builder.toString === "Hello, test!")
* }
*
* test("first char of the temp file") {
* assert(reader.read() === 'H')
* }
*
* test("without a fixture") {
* assert(1 + 1 === 2)
* }
* }
*
*
*
* In this example, the instance variable reader is a var, so
* it can be reinitialized between tests by the beforeEach method.
*
*
*
* Although the BeforeAndAfterEach approach should be familiar to the users of most
* test other frameworks, ScalaTest provides another alternative that also allows you to perform cleanup
* after each test: overriding withFixture(NoArgTest).
* To execute each test, Suite's implementation of the runTest method wraps an invocation
* of the appropriate test method in a no-arg function. runTest passes that test function to the withFixture(NoArgTest)
* method, which is responsible for actually running the test by invoking the function. Suite's
* implementation of withFixture(NoArgTest) simply invokes the function, like this:
*
*
*
* // Default implementation
* protected def withFixture(test: NoArgTest) {
* test()
* }
*
*
*
* The withFixture(NoArgTest) method exists so that you can override it and set a fixture up before, and clean it up after, each test.
* Thus, the previous temp file example could also be implemented without mixing in BeforeAndAfterEach, like this:
*
*
*
* import org.scalatest.FunSuite
* import org.scalatest.BeforeAndAfterEach
* import java.io.FileReader
* import java.io.FileWriter
* import java.io.File
*
* class MySuite extends FunSuite {
*
* private var reader: FileReader = _
*
* override def withFixture(test: NoArgTest) {
*
* val FileName = "TempFile.txt"
*
* // Set up the temp file needed by the test
* val writer = new FileWriter(FileName)
* try {
* writer.write("Hello, test!")
* }
* finally {
* writer.close()
* }
*
* // Create the reader needed by the test
* reader = new FileReader(FileName)
*
* try {
* test() // Invoke the test function
* }
* finally {
* // Close and delete the temp file
* reader.close()
* val file = new File(FileName)
* file.delete()
* }
* }
*
* test("reading from the temp file") {
* var builder = new StringBuilder
* var c = reader.read()
* while (c != -1) {
* builder.append(c.toChar)
* c = reader.read()
* }
* assert(builder.toString === "Hello, test!")
* }
*
* test("first char of the temp file") {
* assert(reader.read() === 'H')
* }
*
* test("without a fixture") {
* assert(1 + 1 === 2)
* }
* }
*
*
*
* If you prefer to keep your test classes immutable, one final variation is to use the
* FixtureFunSuite trait from the
* org.scalatest.fixture package. Tests in an org.scalatest.fixture.FixtureFunSuite can have a fixture
* object passed in as a parameter. You must indicate the type of the fixture object
* by defining the Fixture type member and define a withFixture method that takes a one-arg test function.
* (A FixtureFunSuite has two overloaded withFixture methods, therefore, one that takes a OneArgTest
* and the other, inherited from Suite, that takes a NoArgTest.)
* Inside the withFixture(OneArgTest) method, you create the fixture, pass it into the test function, then perform any
* necessary cleanup after the test function returns. Instead of invoking each test directly, a FixtureFunSuite will
* pass a function that invokes the code of a test to withFixture(OneArgTest). Your withFixture(OneArgTest) method, therefore,
* is responsible for actually running the code of the test by invoking the test function.
* For example, you could pass the temp file reader fixture to each test that needs it
* by overriding the withFixture(OneArgTest) method of a FixtureFunSuite, like this:
*
*
*
* import org.scalatest.fixture.FixtureFunSuite
* import java.io.FileReader
* import java.io.FileWriter
* import java.io.File
*
* class MySuite extends FixtureFunSuite {
*
* type FixtureParam = FileReader
*
* def withFixture(test: OneArgTest) {
*
* val FileName = "TempFile.txt"
*
* // Set up the temp file needed by the test
* val writer = new FileWriter(FileName)
* try {
* writer.write("Hello, test!")
* }
* finally {
* writer.close()
* }
*
* // Create the reader needed by the test
* val reader = new FileReader(FileName)
*
* try {
* // Run the test using the temp file
* test(reader)
* }
* finally {
* // Close and delete the temp file
* reader.close()
* val file = new File(FileName)
* file.delete()
* }
* }
*
* test("reading from the temp file") { reader =>
* var builder = new StringBuilder
* var c = reader.read()
* while (c != -1) {
* builder.append(c.toChar)
* c = reader.read()
* }
* assert(builder.toString === "Hello, test!")
* }
*
* test("first char of the temp file") { reader =>
* assert(reader.read() === 'H')
* }
*
* test("without a fixture") { () =>
* assert(1 + 1 === 2)
* }
* }
*
*
*
* It is worth noting that the only difference in the test code between the mutable
* BeforeAndAfterEach approach shown here and the immutable FixtureFunSuite
* approach shown previously is that two of the FixtureFunSuite's test functions take a FileReader as
* a parameter via the "reader =>" at the beginning of the function. Otherwise the test code is identical.
* One benefit of the explicit parameter is that, as demonstrated
* by the "without a fixture" test, a FixtureFunSuite
* test need not take the fixture. So you can have some tests that take a fixture, and others that don't.
* In this case, the FixtureFunSuite provides documentation indicating which
* tests use the fixture and which don't, whereas the BeforeAndAfterEach approach does not.
* (If you have want to combine tests that take different fixture types in the same FunSuite, you can
* use MultipleFixtureFunSuite.)
*
*
*
* If you want to execute code before and after all tests (and nested suites) in a suite, such
* want to execute code before and after all tests (and nested suites) in a suite, such
* as you could do with @BeforeClass and @AfterClass
* annotations in JUnit 4, you can use the beforeAll and afterAll
* methods of BeforeAndAfterAll. See the documentation for BeforeAndAfterAll for
* an example.
*
*
* Shared tests
*
*
* Sometimes you may want to run the same test code on different fixture objects. In other words, you may want to write tests that are "shared"
* by different fixture objects.
* To accomplish this in a FunSuite, you first place shared tests in
* behavior functions. These behavior functions will be
* invoked during the construction phase of any FunSuite that uses them, so that the tests they contain will
* be registered as tests in that FunSuite.
* For example, given this stack class:
*
*
*
* import scala.collection.mutable.ListBuffer
*
* class Stack[T] {
*
* val MAX = 10
* private var buf = new ListBuffer[T]
*
* def push(o: T) {
* if (!full)
* o +: buf
* else
* throw new IllegalStateException("can't push onto a full stack")
* }
*
* def pop(): T = {
* if (!empty)
* buf.remove(0)
* else
* throw new IllegalStateException("can't pop an empty stack")
* }
*
* def peek: T = {
* if (!empty)
* buf(0)
* else
* throw new IllegalStateException("can't pop an empty stack")
* }
*
* def full: Boolean = buf.size == MAX
* def empty: Boolean = buf.size == 0
* def size = buf.size
*
* override def toString = buf.mkString("Stack(", ", ", ")")
* }
*
*
*
* You may want to test the Stack class in different states: empty, full, with one item, with one item less than capacity,
* etc. You may find you have several tests that make sense any time the stack is non-empty. Thus you'd ideally want to run
* those same tests for three stack fixture objects: a full stack, a stack with a one item, and a stack with one item less than
* capacity. With shared tests, you can factor these tests out into a behavior function, into which you pass the
* stack fixture to use when running the tests. So in your FunSuite for stack, you'd invoke the
* behavior function three times, passing in each of the three stack fixtures so that the shared tests are run for all three fixtures.
*
*
*
* You can define a behavior function that encapsulates these shared tests inside the FunSuite that uses them. If they are shared
* between different FunSuites, however, you could also define them in a separate trait that is mixed into
* each FunSuite that uses them.
* For example, here the nonEmptyStack behavior function (in this case, a
* behavior method) is defined in a trait along with another
* method containing shared tests for non-full stacks:
*
*
*
* import org.scalatest.FunSuite
*
* trait FunSuiteStackBehaviors { this: FunSuite =>
*
* def nonEmptyStack(createNonEmptyStack: => Stack[Int], lastItemAdded: Int) {
*
* test("empty is invoked on this non-empty stack: " + createNonEmptyStack.toString) {
* val stack = createNonEmptyStack
* assert(!stack.empty)
* }
*
* test("peek is invoked on this non-empty stack: " + createNonEmptyStack.toString) {
* val stack = createNonEmptyStack
* val size = stack.size
* assert(stack.peek === lastItemAdded)
* assert(stack.size === size)
* }
*
* test("pop is invoked on this non-empty stack: " + createNonEmptyStack.toString) {
* val stack = createNonEmptyStack
* val size = stack.size
* assert(stack.pop === lastItemAdded)
* assert(stack.size === size - 1)
* }
* }
*
* def nonFullStack(createNonFullStack: => Stack[Int]) {
*
* test("full is invoked on this non-full stack: " + createNonFullStack.toString) {
* val stack = createNonFullStack
* assert(!stack.full)
* }
*
* test("push is invoked on this non-full stack: " + createNonFullStack.toString) {
* val stack = createNonFullStack
* val size = stack.size
* stack.push(7)
* assert(stack.size === size + 1)
* assert(stack.peek === 7)
* }
* }
* }
*
*
*
* Given these behavior functions, you could invoke them directly, but FunSuite offers a DSL for the purpose,
* which looks like this:
*
*
*
* testsFor(nonEmptyStack(stackWithOneItem, lastValuePushed))
* testsFor(nonFullStack(stackWithOneItem))
*
*
*
* If you prefer to use an imperative style to change fixtures, for example by mixing in BeforeAndAfterEach and
* reassigning a stack var in beforeEach, you could write your behavior functions
* in the context of that var, which means you wouldn't need to pass in the stack fixture because it would be
* in scope already inside the behavior function. In that case, your code would look like this:
*
*
*
* testsFor(nonEmptyStack) // assuming lastValuePushed is also in scope inside nonEmptyStack
* testsFor(nonFullStack)
*
*
*
* The recommended style, however, is the functional, pass-all-the-needed-values-in style. Here's an example:
*
*
*
* import org.scalatest.FunSuite
*
* class StackFunSuite extends FunSuite with FunSuiteStackBehaviors {
*
* // Stack fixture creation methods
* def emptyStack = new Stack[Int]
*
* def fullStack = {
* val stack = new Stack[Int]
* for (i <- 0 until stack.MAX)
* stack.push(i)
* stack
* }
*
* def stackWithOneItem = {
* val stack = new Stack[Int]
* stack.push(9)
* stack
* }
*
* def stackWithOneItemLessThanCapacity = {
* val stack = new Stack[Int]
* for (i <- 1 to 9)
* stack.push(i)
* stack
* }
*
* val lastValuePushed = 9
*
* test("empty is invoked on an empty stack") {
* val stack = emptyStack
* assert(stack.empty)
* }
*
* test("peek is invoked on an empty stack") {
* val stack = emptyStack
* intercept[IllegalStateException] {
* stack.peek
* }
* }
*
* test("pop is invoked on an empty stack") {
* val stack = emptyStack
* intercept[IllegalStateException] {
* emptyStack.pop
* }
* }
*
* testsFor(nonEmptyStack(stackWithOneItem, lastValuePushed))
* testsFor(nonFullStack(stackWithOneItem))
*
* testsFor(nonEmptyStack(stackWithOneItemLessThanCapacity, lastValuePushed))
* testsFor(nonFullStack(stackWithOneItemLessThanCapacity))
*
* test("full is invoked on a full stack") {
* val stack = fullStack
* assert(stack.full)
* }
*
* testsFor(nonEmptyStack(fullStack, lastValuePushed))
*
* test("push is invoked on a full stack") {
* val stack = fullStack
* intercept[IllegalStateException] {
* stack.push(10)
* }
* }
* }
*
*
*
* If you load these classes into the Scala interpreter (with scalatest's JAR file on the class path), and execute it,
* you'll see:
*
*
*
* scala> (new StackFunSuite).execute()
* Test Starting - StackFunSuite: empty is invoked on an empty stack
* Test Succeeded - StackFunSuite: empty is invoked on an empty stack
* Test Starting - StackFunSuite: peek is invoked on an empty stack
* Test Succeeded - StackFunSuite: peek is invoked on an empty stack
* Test Starting - StackFunSuite: pop is invoked on an empty stack
* Test Succeeded - StackFunSuite: pop is invoked on an empty stack
* Test Starting - StackFunSuite: empty is invoked on this non-empty stack: Stack(9)
* Test Succeeded - StackFunSuite: empty is invoked on this non-empty stack: Stack(9)
* Test Starting - StackFunSuite: peek is invoked on this non-empty stack: Stack(9)
* Test Succeeded - StackFunSuite: peek is invoked on this non-empty stack: Stack(9)
* Test Starting - StackFunSuite: pop is invoked on this non-empty stack: Stack(9)
* Test Succeeded - StackFunSuite: pop is invoked on this non-empty stack: Stack(9)
* Test Starting - StackFunSuite: full is invoked on this non-full stack: Stack(9)
* Test Succeeded - StackFunSuite: full is invoked on this non-full stack: Stack(9)
* Test Starting - StackFunSuite: push is invoked on this non-full stack: Stack(9)
* Test Succeeded - StackFunSuite: push is invoked on this non-full stack: Stack(9)
* Test Starting - StackFunSuite: empty is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Succeeded - StackFunSuite: empty is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Starting - StackFunSuite: peek is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Succeeded - StackFunSuite: peek is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Starting - StackFunSuite: pop is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Succeeded - StackFunSuite: pop is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Starting - StackFunSuite: full is invoked on this non-full stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Succeeded - StackFunSuite: full is invoked on this non-full stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Starting - StackFunSuite: push is invoked on this non-full stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Succeeded - StackFunSuite: push is invoked on this non-full stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
* Test Starting - StackFunSuite: full is invoked on a full stack
* Test Succeeded - StackFunSuite: full is invoked on a full stack
* Test Starting - StackFunSuite: empty is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Succeeded - StackFunSuite: empty is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Starting - StackFunSuite: peek is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Succeeded - StackFunSuite: peek is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Starting - StackFunSuite: pop is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Succeeded - StackFunSuite: pop is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1, 0)
* Test Starting - StackFunSuite: push is invoked on a full stack
* Test Succeeded - StackFunSuite: push is invoked on a full stack
*
*
*
* One thing to keep in mind when using shared tests is that in ScalaTest, each test in a suite must have a unique name.
* If you register the same tests repeatedly in the same suite, one problem you may encounter is an exception at runtime
* complaining that multiple tests are being registered with the same test name.
* In a FunSuite there is no nesting construct analogous to Spec's describe clause.
* Therefore, you need to do a bit of
* extra work to ensure that the test names are unique. If a duplicate test name problem shows up in a
* FunSuite, you'll need to pass in a prefix or suffix string to add to each test name. You can pass this string
* the same way you pass any other data needed by the shared tests, or just call toString on the shared fixture object.
* This is the approach taken by the previous FunSuiteStackBehaviors example.
*
*
*
* Given this FunSuiteStackBehaviors trait, calling it with the stackWithOneItem fixture, like this:
*
*
*
* testsFor(nonEmptyStack(stackWithOneItem, lastValuePushed))
*
*
*
* yields test names:
*
*
*
* empty is invoked on this non-empty stack: Stack(9)peek is invoked on this non-empty stack: Stack(9)pop is invoked on this non-empty stack: Stack(9)
*
*
* Whereas calling it with the stackWithOneItemLessThanCapacity fixture, like this:
*
*
*
* testsFor(nonEmptyStack(stackWithOneItemLessThanCapacity, lastValuePushed))
*
*
*
* yields different test names:
*
*
*
* empty is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)peek is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)pop is invoked on this non-empty stack: Stack(9, 8, 7, 6, 5, 4, 3, 2, 1)
*
* Tagging tests
*
*
* A FunSuite's tests may be classified into groups by tagging them with string names.
* As with any suite, when executing a FunSuite, groups of tests can
* optionally be included and/or excluded. To tag a FunSuite's tests,
* you pass objects that extend abstract class org.scalatest.Tag to methods
* that register tests, test and ignore. Class Tag takes one parameter, a string name. If you have
* created Java annotation interfaces for use as group names in direct subclasses of org.scalatest.Suite,
* then you will probably want to use group names on your FunSuites that match. To do so, simply
* pass the fully qualified names of the Java interfaces to the Tag constructor. For example, if you've
* defined Java annotation interfaces with fully qualified names, com.mycompany.groups.SlowTest and
* com.mycompany.groups.DbTest, then you could
* create matching groups for FunSuites like this:
*
*
*
* import org.scalatest.Tag
*
* object SlowTest extends Tag("com.mycompany.groups.SlowTest")
* object DbTest extends Tag("com.mycompany.groups.DbTest")
*
*
*
* Given these definitions, you could place FunSuite tests into groups like this:
*
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* test("addition", SlowTest) {
* val sum = 1 + 1
* assert(sum === 2)
* assert(sum + 2 === 4)
* }
*
* test("subtraction", SlowTest, DbTest) {
* val diff = 4 - 1
* assert(diff === 3)
* assert(diff - 2 === 1)
* }
* }
*
*
*
* This code marks both tests, "addition" and "subtraction," with the com.mycompany.groups.SlowTest tag,
* and test "subtraction" with the com.mycompany.groups.DbTest tag.
*
*
*
* The primary run method takes a Filter, whose constructor takes an optional
* Set[String]s called tagsToInclude and a Set[String] called
* tagsToExclude. If tagsToInclude is None, all tests will be run
* except those those belonging to tags listed in the
* tagsToExclude Set. If tagsToInclude is defined, only tests
* belonging to tags mentioned in the tagsToInclude set, and not mentioned in tagsToExclude,
* will be run.
*
*
* Ignored tests
*
*
* To support the common use case of “temporarily” disabling a test, with the
* good intention of resurrecting the test at a later time, FunSuite provides registration
* methods that start with ignore instead of test. For example, to temporarily
* disable the test named addition, just change “test” into “ignore,” like this:
*
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* ignore("addition") {
* val sum = 1 + 1
* assert(sum === 2)
* assert(sum + 2 === 4)
* }
*
* test("subtraction") {
* val diff = 4 - 1
* assert(diff === 3)
* assert(diff - 2 === 1)
* }
* }
*
*
*
* If you run this version of MySuite with:
*
*
*
* scala> (new MySuite).execute()
*
*
*
* It will run only subtraction and report that addition was ignored:
*
*
*
* Test Ignored - MySuite: addition
* Test Starting - MySuite: subtraction
* Test Succeeded - MySuite: subtraction
*
*
* Pending tests
*
*
* A pending test is one that has been given a name but is not yet implemented. The purpose of
* pending tests is to facilitate a style of testing in which documentation of behavior is sketched
* out before tests are written to verify that behavior (and often, the before the behavior of
* the system being tested is itself implemented). Such sketches form a kind of specification of
* what tests and functionality to implement later.
*
*
*
* To support this style of testing, a test can be given a name that specifies one
* bit of behavior required by the system being tested. The test can also include some code that
* sends more information about the behavior to the reporter when the tests run. At the end of the test,
* it can call method pending, which will cause it to complete abruptly with TestPendingException.
* Because tests in ScalaTest can be designated as pending with TestPendingException, both the test name and any information
* sent to the reporter when running the test can appear in the report of a test run. (In other words,
* the code of a pending test is executed just like any other test.) However, because the test completes abruptly
* with TestPendingException, the test will be reported as pending, to indicate
* the actual test, and possibly the functionality, has not yet been implemented.
*
*
*
* Although pending tests may be used more often in specification-style suites, such as
* org.scalatest.Spec, you can also use it in FunSuite, like this:
*
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* test("addition") {
* val sum = 1 + 1
* assert(sum === 2)
* assert(sum + 2 === 4)
* }
*
* test("subtraction") (pending)
* }
*
*
*
* (Note: "(pending)" is the body of the test. Thus the test contains just one statement, an invocation
* of the pending method, which throws TestPendingException.)
* If you run this version of MySuite with:
*
*
*
* scala> (new MySuite).execute()
*
*
*
* It will run both tests, but report that subtraction is pending. You'll see:
*
*
*
* Test Starting - MySuite: addition
* Test Succeeded - MySuite: addition
* Test Starting - MySuite: subtraction
* Test Pending - MySuite: subtraction
*
*
* Informers
*
*
* One of the parameters to the primary run method is a Reporter, which
* will collect and report information about the running suite of tests.
* Information about suites and tests that were run, whether tests succeeded or failed,
* and tests that were ignored will be passed to the Reporter as the suite runs.
* Most often the reporting done by default by FunSuite's methods will be sufficient, but
* occasionally you may wish to provide custom information to the Reporter from a test.
* For this purpose, an Informer that will forward information to the current Reporter
* is provided via the info parameterless method.
* You can pass the extra information to the Informer via one of its apply methods.
* The Informer will then pass the information to the Reporter via an InfoProvided event.
* Here's an example:
*
*
*
* import org.scalatest.FunSuite
*
* class MySuite extends FunSuite {
*
* test("addition") {
* val sum = 1 + 1
* assert(sum === 2)
* assert(sum + 2 === 4)
* info("Addition seems to work")
* }
* }
*
*
* If you run this Suite from the interpreter, you will see the following message
* included in the printed report:
*
*
* Test Starting - MySuite: addition
* Info Provided - MySuite.addition: Addition seems to work
* Test Succeeded - MySuite: addition
*
*
* @author Bill Venners
*/
trait FunSuite extends Suite { thisSuite =>
private val IgnoreTagName = "org.scalatest.Ignore"
private abstract class FunNode
private case class TestNode(testName: String, fun: () => Unit) extends FunNode
private case class InfoNode(message: String) extends FunNode
// Access to the testNamesList, testsMap, and tagsMap must be synchronized, because the test methods are invoked by
// the primary constructor, but testNames, tags, and runTest get invoked directly or indirectly
// by run. When running tests concurrently with ScalaTest Runner, different threads can
// instantiate and run the suite. Instead of synchronizing, I put them in an immutable Bundle object (and
// all three collections--testNamesList, testsMap, and tagsMap--are immuable collections), then I put the Bundle
// in an AtomicReference. Since the expected use case is the test method will be called
// from the primary constructor, which will be all done by one thread, I just in effect use optimistic locking on the Bundle.
// If two threads ever called test at the same time, they could get a ConcurrentModificationException.
// Test names are in reverse order of test registration method invocations
private class Bundle private(
val testNamesList: List[String],
val doList: List[FunNode],
val testsMap: Map[String, TestNode],
val tagsMap: Map[String, Set[String]],
val registrationClosed: Boolean
) {
def unpack = (testNamesList, doList, testsMap, tagsMap, registrationClosed)
}
private object Bundle {
def apply(
testNamesList: List[String],
doList: List[FunNode],
testsMap: Map[String, TestNode],
tagsMap: Map[String, Set[String]],
registrationClosed: Boolean
): Bundle =
new Bundle(testNamesList, doList,testsMap, tagsMap, registrationClosed)
}
private val atomic = new AtomicReference[Bundle](Bundle(List(), List(), Map(), Map(), false))
private def updateAtomic(oldBundle: Bundle, newBundle: Bundle) {
val shouldBeOldBundle = atomic.getAndSet(newBundle)
if (!(shouldBeOldBundle eq oldBundle))
throw new ConcurrentModificationException(Resources("concurrentFunSuiteBundleMod"))
}
private class RegistrationInformer extends Informer {
def apply(message: String) {
if (message == null)
throw new NullPointerException
val oldBundle = atomic.get
var (testNamesList, doList, testsMap, tagsMap, registrationClosed) = oldBundle.unpack
doList ::= InfoNode(message)
updateAtomic(oldBundle, Bundle(testNamesList, doList, testsMap, tagsMap, registrationClosed))
}
}
// The informer will be a registration informer until run is called for the first time. (This
// is the registration phase of a FunSuite's lifecycle.)
private final val atomicInformer = new AtomicReference[Informer](new RegistrationInformer)
/**
* Returns an Informer that during test execution will forward strings (and other objects) passed to its
* apply method to the current reporter. If invoked in a constructor, it
* will register the passed string for forwarding later during test execution. If invoked while this
* FunSuite is being executed, such as from inside a test function, it will forward the information to
* the current reporter immediately. If invoked at any other time, it will
* throw an exception. This method can be called safely by any thread.
*/
implicit protected def info: Informer = atomicInformer.get
private val zombieInformer =
new Informer {
private val complaint = Resources("cantCallInfoNow", "FunSuite")
def apply(message: String) {
if (message == null)
throw new NullPointerException
throw new IllegalStateException(complaint)
}
}
/**
* Register a test with the specified name, optional tags, and function value that takes no arguments.
* This method will register the test for later execution via an invocation of one of the run
* methods. The passed test name must not have been registered previously on
* this FunSuite instance.
*
* @param testName the name of the test
* @param testTags the optional list of tags for this test
* @param testFun the test function
* @throws TestRegistrationClosedException if invoked after run has been invoked on this suite
* @throws DuplicateTestNameException if a test with the same name has been registered previously
* @throws NotAllowedException if testName had been registered previously
* @throws NullPointerException if testName or any passed test tag is null
*/
protected def test(testName: String, testTags: Tag*)(f: => Unit) {
if (testName == null)
throw new NullPointerException("testName was null")
if (testTags.exists(_ == null))
throw new NullPointerException("a test tag was null")
if (atomic.get.registrationClosed)
throw new TestRegistrationClosedException(Resources("testCannotAppearInsideAnotherTest"), getStackDepth("FunSuite.scala", "test"))
if (atomic.get.testsMap.keySet.contains(testName))
throw new DuplicateTestNameException(Resources("duplicateTestName", testName), getStackDepth("FunSuite.scala", "test"))
val oldBundle = atomic.get
var (testNamesList, doList, testsMap, tagsMap, registrationClosed) = oldBundle.unpack
val testNode = TestNode(testName, f _)
testsMap += (testName -> testNode)
testNamesList ::= testName
doList ::= testNode
val tagNames = Set[String]() ++ testTags.map(_.name)
if (!tagNames.isEmpty)
tagsMap += (testName -> tagNames)
updateAtomic(oldBundle, Bundle(testNamesList, doList, testsMap, tagsMap, registrationClosed))
}
/**
* Register a test to ignore, which has the specified name, optional tags, and function value that takes no arguments.
* This method will register the test for later ignoring via an invocation of one of the run
* methods. This method exists to make it easy to ignore an existing test by changing the call to test
* to ignore without deleting or commenting out the actual test code. The test will not be run, but a
* report will be sent that indicates the test was ignored. The passed test name must not have been registered previously on
* this FunSuite instance.
*
* @param testName the name of the test
* @param testTags the optional list of tags for this test
* @param testFun the test function
* @throws TestRegistrationClosedException if invoked after run has been invoked on this suite
* @throws DuplicateTestNameException if a test with the same name has been registered previously
* @throws NotAllowedException if testName had been registered previously
*/
protected def ignore(testName: String, testTags: Tag*)(f: => Unit) {
if (testName == null)
throw new NullPointerException("testName was null")
if (testTags.exists(_ == null))
throw new NullPointerException("a test tag was null")
if (atomic.get.registrationClosed)
throw new TestRegistrationClosedException(Resources("ignoreCannotAppearInsideATest"), getStackDepth("FunSuite.scala", "ignore"))
test(testName)(f) // Call test without passing the tags
val oldBundle = atomic.get
var (testNamesList, doList, testsMap, tagsMap, registrationClosed) = oldBundle.unpack
val tagNames = Set[String]() ++ testTags.map(_.name)
tagsMap += (testName -> (tagNames + IgnoreTagName))
updateAtomic(oldBundle, Bundle(testNamesList, doList, testsMap, tagsMap, registrationClosed))
}
/**
* An immutable Set of test names. If this FunSuite contains no tests, this method returns an empty Set.
*
*
* This trait's implementation of this method will return a set that contains the names of all registered tests. The set's iterator will
* return those names in the order in which the tests were registered.
*
*/
override def testNames: Set[String] = {
// I'm returning a ListSet here so that they tests will be run in registration order
ListSet(atomic.get.testNamesList.toArray: _*)
}
// runTest should throw IAE if a test name is passed that doesn't exist. Looks like right now it just reports a test failure.
/**
* Run a test. This trait's implementation runs the test registered with the name specified by testName.
*
* @param testName the name of one test to run.
* @param reporter the Reporter to which results will be reported
* @param stopper the Stopper that will be consulted to determine whether to stop execution early.
* @param configMap a Map of properties that can be used by the executing Suite of tests.
* @throws NullPointerException if any of testName, reporter, stopper, or configMap
* is null.
*/
protected override def runTest(testName: String, reporter: Reporter, stopper: Stopper, configMap: Map[String, Any], tracker: Tracker) {
if (testName == null || reporter == null || stopper == null || configMap == null)
throw new NullPointerException
val stopRequested = stopper
val report = wrapReporterIfNecessary(reporter)
// Create a Rerunner if the FunSuite has a no-arg constructor
val hasPublicNoArgConstructor = Suite.checkForPublicNoArgConstructor(getClass)
val rerunnable =
if (hasPublicNoArgConstructor)
Some(new TestRerunner(getClass.getName, testName))
else
None
val testStartTime = System.currentTimeMillis
report(TestStarting(tracker.nextOrdinal(), thisSuite.suiteName, Some(thisSuite.getClass.getName), testName, None, rerunnable))
try {
val theTest = atomic.get.testsMap(testName)
val informerForThisTest =
new ConcurrentInformer(NameInfo(thisSuite.suiteName, Some(thisSuite.getClass.getName), Some(testName))) {
def apply(message: String) {
if (message == null)
throw new NullPointerException
report(InfoProvided(tracker.nextOrdinal(), message, nameInfoForCurrentThread))
}
}
val oldInformer = atomicInformer.getAndSet(informerForThisTest)
var swapAndCompareSucceeded = false
try {
val theConfigMap = configMap
withFixture(
new NoArgTest {
def name = testName
def apply() { theTest.fun() }
def configMap = theConfigMap
}
)
}
finally {
val shouldBeInformerForThisTest = atomicInformer.getAndSet(oldInformer)
swapAndCompareSucceeded = shouldBeInformerForThisTest eq informerForThisTest
}
if (!swapAndCompareSucceeded) // Do outside finally to workaround Scala compiler bug
throw new ConcurrentModificationException(Resources("concurrentInformerMod", thisSuite.getClass.getName))
val duration = System.currentTimeMillis - testStartTime
report(TestSucceeded(tracker.nextOrdinal(), thisSuite.suiteName, Some(thisSuite.getClass.getName), testName, Some(duration), None, rerunnable))
}
catch {
case _: TestPendingException =>
report(TestPending(tracker.nextOrdinal(), thisSuite.suiteName, Some(thisSuite.getClass.getName), testName))
case e if !anErrorThatShouldCauseAnAbort(e) =>
val duration = System.currentTimeMillis - testStartTime
handleFailedTest(e, false, testName, rerunnable, report, tracker, duration)
case e => throw e
}
}
private def handleFailedTest(throwable: Throwable, hasPublicNoArgConstructor: Boolean, testName: String,
rerunnable: Option[Rerunner], reporter: Reporter, tracker: Tracker, duration: Long) {
val message =
if (throwable.getMessage != null) // [bv: this could be factored out into a helper method]
throwable.getMessage
else
throwable.toString
reporter(TestFailed(tracker.nextOrdinal(), message, thisSuite.suiteName, Some(thisSuite.getClass.getName), testName, Some(throwable), Some(duration), None, rerunnable))
}
/**
* A Map whose keys are String tag names to which tests in this FunSuite belong, and values
* the Set of test names that belong to each tag. If this FunSuite contains no tags, this method returns an empty Map.
*
*
* This trait's implementation returns tags that were passed as strings contained in Tag objects passed to
* methods test and ignore.
*
*/
override def tags: Map[String, Set[String]] = atomic.get.tagsMap
/**
* Run zero to many of this Spec's tests.
*
* @param testName an optional name of one test to run. If None, all relevant tests should be run.
* I.e., None acts like a wildcard that means run all relevant tests in this Suite.
* @param reporter the Reporter to which results will be reported
* @param stopper the Stopper that will be consulted to determine whether to stop execution early.
* @param filter a Filter with which to filter tests based on their tags
* @param configMap a Map of key-value pairs that can be used by the executing Suite of tests.
* @param distributor an optional Distributor, into which to put nested Suites to be run
* by another entity, such as concurrently by a pool of threads. If None, nested Suites will be run sequentially.
* @param tracker a Tracker tracking Ordinals being fired by the current thread.
* @throws NullPointerException if any of the passed parameters is null.
* @throws IllegalArgumentException if testName is defined, but no test with the specified test name
* exists in this Suite
*/
protected override def runTests(testName: Option[String], reporter: Reporter, stopper: Stopper, filter: Filter,
configMap: Map[String, Any], distributor: Option[Distributor], tracker: Tracker) {
if (testName == null)
throw new NullPointerException("testName was null")
if (reporter == null)
throw new NullPointerException("reporter was null")
if (stopper == null)
throw new NullPointerException("stopper was null")
if (filter == null)
throw new NullPointerException("filter was null")
if (configMap == null)
throw new NullPointerException("configMap was null")
if (distributor == null)
throw new NullPointerException("distributor was null")
if (tracker == null)
throw new NullPointerException("tracker was null")
val stopRequested = stopper
// Wrap any non-DispatchReporter, non-CatchReporter in a CatchReporter,
// so that exceptions are caught and transformed
// into error messages on the standard error stream.
val report = wrapReporterIfNecessary(reporter)
// If a testName is passed to run, just run that, else run the tests returned
// by testNames.
testName match {
case Some(tn) => runTest(tn, report, stopRequested, configMap, tracker)
case None =>
val doList = atomic.get.doList.reverse
for (node <- doList) {
node match {
case InfoNode(message) => info(message)
case TestNode(tn, _) =>
val (filterTest, ignoreTest) = filter(tn, tags)
if (!filterTest)
if (ignoreTest)
report(TestIgnored(tracker.nextOrdinal(), thisSuite.suiteName, Some(thisSuite.getClass.getName), tn))
else
runTest(tn, report, stopRequested, configMap, tracker)
}
}
}
}
@volatile private var wasRunBefore = false
override def run(testName: Option[String], reporter: Reporter, stopper: Stopper, filter: Filter,
configMap: Map[String, Any], distributor: Option[Distributor], tracker: Tracker) {
wasRunBefore = true
val stopRequested = stopper
// Set the flag that indicates registration is closed (because run has now been invoked),
// which will disallow any further invocations of "test" or "ignore" with
// an RegistrationClosedException.
val oldBundle = atomic.get
val (testNamesList, doList, testsMap, tagsMap, registrationClosed) = oldBundle.unpack
if (!registrationClosed)
updateAtomic(oldBundle, Bundle(testNamesList, doList, testsMap, tagsMap, true))
val report = wrapReporterIfNecessary(reporter)
val informerForThisSuite =
new ConcurrentInformer(NameInfo(thisSuite.suiteName, Some(thisSuite.getClass.getName), None)) {
def apply(message: String) {
if (message == null)
throw new NullPointerException
report(InfoProvided(tracker.nextOrdinal(), message, nameInfoForCurrentThread))
}
}
atomicInformer.set(informerForThisSuite)
var swapAndCompareSucceeded = false
try {
super.run(testName, report, stopRequested, filter, configMap, distributor, tracker)
}
finally {
val shouldBeInformerForThisSuite = atomicInformer.getAndSet(zombieInformer)
swapAndCompareSucceeded = shouldBeInformerForThisSuite eq informerForThisSuite
}
if (!swapAndCompareSucceeded) // Do outside finally to workaround Scala compiler bug
throw new ConcurrentModificationException(Resources("concurrentInformerMod", thisSuite.getClass.getName))
}
/**
* Registers shared tests.
*
*
* This method enables the following syntax for shared tests in a FunSuite:
*
*
*
* testsFor(nonEmptyStack(lastValuePushed))
*
*
*
* This method just provides syntax sugar intended to make the intent of the code clearer.
* Because the parameter passed to it is
* type Unit, the expression will be evaluated before being passed, which
* is sufficient to register the shared tests. For examples of shared tests, see the
* Shared tests section in the main documentation for this trait.
*
*/
protected def testsFor(unit: Unit) {}
}