org.scalatest.PropSpec.scala Maven / Gradle / Ivy
Show all versions of scalatest_2.11.0-RC2 Show documentation
/*
* Copyright 2001-2013 Artima, Inc.
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
package org.scalatest
import scala.collection.immutable.ListSet
import Suite.autoTagClassAnnotations
/**
* A suite of property-based tests.
*
*
* Recommended Usage:
* Class PropSpec is a good fit for teams that want to write tests exclusively in terms of property checks, and is also a good choice
* for writing the occasional test matrix when a different style trait is chosen as the main unit testing style.
*
*
* Here's an example PropSpec:
*
*
* package org.scalatest.examples.propspec
*
* import org.scalatest._
* import prop._
* import scala.collection.immutable._
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with ShouldMatchers {
*
* val examples =
* Table(
* "set",
* BitSet.empty,
* HashSet.empty[Int],
* TreeSet.empty[Int]
* )
*
* property("an empty Set should have size 0") {
* forAll(examples) { set =>
* set.size should be (0)
* }
* }
*
* property("invoking head on an empty set should produce NoSuchElementException") {
* forAll(examples) { set =>
* evaluating { set.head } should produce [NoSuchElementException]
* }
* }
* }
*
*
*
* You can run a PropSpec by invoking execute on it.
* This method, which prints test results to the standard output, is intended to serve as a
* convenient way to run tests from within the Scala interpreter. For example,
* to run SetSpec from within the Scala interpreter, you could write:
*
*
*
* scala> new SetSpec execute
*
*
*
* And you would see:
*
*
*
* SetSpec:
* - an empty Set should have size 0
* - invoking head on an empty Set should produce NoSuchElementException
*
*
*
* Or, to run just the “an empty Set should have size 0” method, you could pass that test's name, or any unique substring of the
* name, such as "size 0" or even just "0". Here's an example:
*
*
*
* scala> new SetSpec execute "size 0"
* SetSpec:
* - an empty Set should have size 0
*
*
*
* You can also pass to execute a config map of key-value
* pairs, which will be passed down into suites and tests, as well as other parameters that configure the run itself.
* For more information on running in the Scala interpreter, see the documentation for execute (below) and the
* ScalaTest shell.
*
*
*
* The execute method invokes a run method that takes two
* parameters. This run method, which actually executes the suite, will usually be invoked by a test runner, such
* as run, tools.Runner, a build tool, or an IDE.
*
*
*
* “property” is a method, defined in PropSpec, which will be invoked
* by the primary constructor of SetSpec. You specify the name of the test as
* a string between the parentheses, and the test code itself between curly braces.
* The test code is a function passed as a by-name parameter to property, which registers
* it for later execution.
*
*
*
* A PropSpec's lifecycle has two phases: the registration phase and the
* ready phase. It starts in registration phase and enters ready phase the first time
* run is called on it. It then remains in ready phase for the remainder of its lifetime.
*
*
*
* Tests can only be registered with the property method while the PropSpec is
* in its registration phase. Any attempt to register a test after the PropSpec has
* entered its ready phase, i.e., after run has been invoked on the PropSpec,
* will be met with a thrown TestRegistrationClosedException. The recommended style
* of using PropSpec is to register tests during object construction as is done in all
* the examples shown here. If you keep to the recommended style, you should never see a
* TestRegistrationClosedException.
*
*
* Ignored tests
*
*
* To support the common use case of temporarily disabling a test, with the
* good intention of resurrecting the test at a later time, PropSpec provides registration
* methods that start with ignore instead of property. Here's an example:
*
*
*
* package org.scalatest.examples.suite.ignore
*
* import org.scalatest._
* import prop._
* import scala.collection.immutable._
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with ShouldMatchers {
*
* val examples =
* Table(
* "set",
* BitSet.empty,
* HashSet.empty[Int],
* TreeSet.empty[Int]
* )
*
* ignore("an empty Set should have size 0") {
* forAll(examples) { set =>
* set.size should be (0)
* }
* }
*
* property("invoking head on an empty set should produce NoSuchElementException") {
* forAll(examples) { set =>
* evaluating { set.head } should produce [NoSuchElementException]
* }
* }
* }
*
*
*
* If you run this version of SetSuite with:
*
*
*
* scala> new SetSpec execute
*
*
*
* It will run only the second test and report that the first test was ignored:
*
*
*
* SetSuite:
* - an empty Set should have size 0 !!! IGNORED !!!
* - invoking head on an empty Set should produce NoSuchElementException
*
*
* Informers
*
*
* One of the parameters to PropSpec's run method is a Reporter, which
* will collect and report information about the running suite of tests.
* Information about suites and tests that were run, whether tests succeeded or failed,
* and tests that were ignored will be passed to the Reporter as the suite runs.
* Most often the reporting done by default by PropSpec's methods will be sufficient, but
* occasionally you may wish to provide custom information to the Reporter from a test.
* For this purpose, an Informer that will forward information
* to the current Reporter is provided via the info parameterless method.
* You can pass the extra information to the Informer via its apply method.
* The Informer will then pass the information to the Reporter via an InfoProvided event.
* Here's an example that shows both a direct use as well as an indirect use through the methods
* of GivenWhenThen:
*
*
*
* package org.scalatest.examples.propspec.info
*
* import org.scalatest._
* import prop._
* import collection.mutable
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with GivenWhenThen {
*
* val examples =
* Table(
* "set",
* mutable.BitSet.empty,
* mutable.HashSet.empty[Int],
* mutable.LinkedHashSet.empty[Int]
* )
*
* property("an element can be added to an empty mutable Set") {
*
* forAll(examples) { set =>
*
* info("----------------")
*
* Given("an empty mutable " + set.getClass.getSimpleName)
* assert(set.isEmpty)
*
* When("an element is added")
* set += 99
*
* Then("the Set should have size 1")
* assert(set.size === 1)
*
* And("the Set should contain the added element")
* assert(set.contains(99))
* }
* }
* }
*
*
*
* If you run this PropSpec from the interpreter, you will see the following output:
*
*
* scala> new SetSpec execute
* SetSpec:
* - an element can be added to an empty mutable Set
* + ----------------
* + Given an empty mutable BitSet
* + When an element is added
* + Then the Set should have size 1
* + And the Set should contain the added element
* + ----------------
* + Given an empty mutable HashSet
* + When an element is added
* + Then the Set should have size 1
* + And the Set should contain the added element
* + ----------------
* + Given an empty mutable LinkedHashSet
* + When an element is added
* + Then the Set should have size 1
* + And the Set should contain the added element
*
*
* Documenters
*
*
* PropSpec also provides a markup method that returns a Documenter, which allows you to send
* to the Reporter text formatted in Markdown syntax.
* You can pass the extra information to the Documenter via its apply method.
* The Documenter will then pass the information to the Reporter via an MarkupProvided event.
*
*
*
* Here's an example PropSpec that uses markup:
*
*
*
* package org.scalatest.examples.propspec.markup
*
* import org.scalatest._
* import prop._
* import collection.mutable
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with GivenWhenThen {
*
* markup { """
*
* Mutable Set
* -----------
*
* A set is a collection that contains no duplicate elements.
*
* To implement a concrete mutable set, you need to provide implementations
* of the following methods:
*
* def contains(elem: A): Boolean
* def iterator: Iterator[A]
* def += (elem: A): this.type
* def -= (elem: A): this.type
*
* If you wish that methods like `take`,
* `drop`, `filter` return the same kind of set,
* you should also override:
*
* def empty: This
*
* It is also good idea to override methods `foreach` and
* `size` for efficiency.
*
* """ }
*
* val examples =
* Table(
* "set",
* mutable.BitSet.empty,
* mutable.HashSet.empty[Int],
* mutable.LinkedHashSet.empty[Int]
* )
*
* property("an element can be added to an empty mutable Set") {
*
* forAll(examples) { set =>
*
* info("----------------")
*
* Given("an empty mutable " + set.getClass.getSimpleName)
* assert(set.isEmpty)
*
* When("an element is added")
* set += 99
*
* Then("the Set should have size 1")
* assert(set.size === 1)
*
* And("the Set should contain the added element")
* assert(set.contains(99))
* }
*
* markup("This test finished with a **bold** statement!")
* }
* }
*
*
*
* Although all of ScalaTest's built-in reporters will display the markup text in some form,
* the HTML reporter will format the markup information into HTML. Thus, the main purpose of markup is to
* add nicely formatted text to HTML reports. Here's what the above SetSpec would look like in the HTML reporter:
*
*
* 
*
* Notifiers and alerters
*
*
* ScalaTest records text passed to info and markup during tests, and sends the recorded text in the recordedEvents field of
* test completion events like TestSucceeded and TestFailed. This allows string reporters (like the standard out reporter) to show
* info and markup text after the test name in a color determined by the outcome of the test. For example, if the test fails, string
* reporters will show the info and markup text in red. If a test succeeds, string reporters will show the info
* and markup text in green. While this approach helps the readability of reports, it means that you can't use info to get status
* updates from long running tests.
*
*
*
* To get immediate (i.e., non-recorded) notifications from tests, you can use note (a Notifier) and alert
* (an Alerter). Here's an example showing the differences:
*
*
*
* package org.scalatest.examples.propspec.note
*
* import org.scalatest._
* import prop._
* import collection.mutable
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks {
*
* val examples =
* Table(
* "set",
* mutable.BitSet.empty,
* mutable.HashSet.empty[Int],
* mutable.LinkedHashSet.empty[Int]
* )
*
* property("an element can be added to an empty mutable Set") {
*
* info("info is recorded")
* markup("markup is *also* recorded")
* note("notes are sent immediately")
* alert("alerts are also sent immediately")
*
* forAll(examples) { set =>
*
* assert(set.isEmpty)
* set += 99
* assert(set.size === 1)
* assert(set.contains(99))
* }
* }
* }
*
*
*
* Because note and alert information is sent immediately, it will appear before the test name in string reporters, and its color will
* be unrelated to the ultimate outcome of the test: note text will always appear in green, alert text will always appear in yellow.
* Here's an example:
*
*
*
* scala> new SetSpec execute
* SetSpec:
* + notes are sent immediately
* + alerts are also sent immediately
* - an element can be added to an empty mutable Set
* + info is recorded
* + markup is *also* recorded
*
*
*
* In summary, use info and markup for text that should form part of the specification output. Use
* note and alert to send status notifications. (Because the HTML reporter is intended to produce a
* readable, printable specification, info and markup text will appear in the HTML report, but
* note and alert text will not.)
*
*
* Pending tests
*
*
* A pending test is one that has been given a name but is not yet implemented. The purpose of
* pending tests is to facilitate a style of testing in which documentation of behavior is sketched
* out before tests are written to verify that behavior (and often, before the behavior of
* the system being tested is itself implemented). Such sketches form a kind of specification of
* what tests and functionality to implement later.
*
*
*
* To support this style of testing, a test can be given a name that specifies one
* bit of behavior required by the system being tested. The test can also include some code that
* sends more information about the behavior to the reporter when the tests run. At the end of the test,
* it can call method pending, which will cause it to complete abruptly with TestPendingException.
*
*
*
* Because tests in ScalaTest can be designated as pending with TestPendingException, both the test name and any information
* sent to the reporter when running the test can appear in the report of a test run.
* (The code of a pending test is executed just like any other test.) However, because the test completes abruptly
* with TestPendingException, the test will be reported as pending, to indicate
* the actual test, and possibly the functionality, has not yet been implemented.
*
*
*
* You can mark tests pending in PropSpec like this:
*
*
*
* import org.scalatest._
* import prop._
* import scala.collection.immutable._
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with ShouldMatchers {
*
* val examples =
* Table(
* "set",
* BitSet.empty,
* HashSet.empty[Int],
* TreeSet.empty[Int]
* )
*
* property("an empty Set should have size 0") (pending)
*
* property("invoking head on an empty set should produce NoSuchElementException") {
* forAll(examples) { set =>
* evaluating { set.head } should produce [NoSuchElementException]
* }
* }
* }
*
*
*
* (Note: "(pending)" is the body of the test. Thus the test contains just one statement, an invocation
* of the pending method, which throws TestPendingException.)
* If you run this version of SetSuite with:
*
*
*
* scala> new SetSuite execute
*
*
*
* It will run both tests, but report that first test is pending. You'll see:
*
*
*
* SetSuite:
* - An empty Set should have size 0 (pending)
* - Invoking head on an empty Set should produce NoSuchElementException
*
*
*
* One difference between an ignored test and a pending one is that an ignored test is intended to be used during a
* significant refactorings of the code under test, when tests break and you don't want to spend the time to fix
* all of them immediately. You can mark some of those broken tests as ignored temporarily, so that you can focus the red
* bar on just failing tests you actually want to fix immediately. Later you can go back and fix the ignored tests.
* In other words, by ignoring some failing tests temporarily, you can more easily notice failed tests that you actually
* want to fix. By contrast, a pending test is intended to be used before a test and/or the code under test is written.
* Pending indicates you've decided to write a test for a bit of behavior, but either you haven't written the test yet, or
* have only written part of it, or perhaps you've written the test but don't want to implement the behavior it tests
* until after you've implemented a different bit of behavior you realized you need first. Thus ignored tests are designed
* to facilitate refactoring of existing code whereas pending tests are designed to facilitate the creation of new code.
*
*
*
* One other difference between ignored and pending tests is that ignored tests are implemented as a test tag that is
* excluded by default. Thus an ignored test is never executed. By contrast, a pending test is implemented as a
* test that throws TestPendingException (which is what calling the pending method does). Thus
* the body of pending tests are executed up until they throw TestPendingException. The reason for this difference
* is that it enables your unfinished test to send InfoProvided messages to the reporter before it completes
* abruptly with TestPendingException, as shown in the previous example on Informers
* that used the GivenWhenThen trait.
*
*
* Tagging tests
*
*
* A PropSpec's tests may be classified into groups by tagging them with string names.
* As with any suite, when executing a PropSpec, groups of tests can
* optionally be included and/or excluded. To tag a PropSpec's tests,
* you pass objects that extend class org.scalatest.Tag to methods
* that register tests. Class Tag takes one parameter, a string name. If you have
* created tag annotation interfaces as described in the Tag documentation, then you
* will probably want to use tag names on your test functions that match. To do so, simply
* pass the fully qualified names of the tag interfaces to the Tag constructor. For example, if you've
* defined tag annotation interfaces with fully qualified names, com.mycompany.tags.SlowTest and
* com.mycompany.tags.DbTest, then you could
* create matching tags for PropSpecs like this:
*
*
*
* package org.scalatest.examples.propspec.tagging
*
* import org.scalatest.Tag
*
* object SlowTest extends Tag("com.mycompany.tags.SlowTest")
* object DbTest extends Tag("com.mycompany.tags.DbTest")
*
*
*
* Given these definitions, you could place PropSpec tests into groups like this:
*
*
*
* import org.scalatest._
* import prop._
* import scala.collection.immutable._
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with ShouldMatchers {
*
* val examples =
* Table(
* "set",
* BitSet.empty,
* HashSet.empty[Int],
* TreeSet.empty[Int]
* )
*
* property("an empty Set should have size 0", SlowTest) {
* forAll(examples) { set =>
* set.size should be (0)
* }
* }
*
* property("invoking head on an empty set should produce NoSuchElementException",
* SlowTest, DbTest) {
*
* forAll(examples) { set =>
* evaluating { set.head } should produce [NoSuchElementException]
* }
* }
* }
*
*
*
* This code marks both tests with the com.mycompany.tags.SlowTest tag,
* and the second test with the com.mycompany.tags.DbTest tag.
*
*
*
* The run method takes a Filter, whose constructor takes an optional
* Set[String] called tagsToInclude and a Set[String] called
* tagsToExclude. If tagsToInclude is None, all tests will be run
* except those those belonging to tags listed in the
* tagsToExclude Set. If tagsToInclude is defined, only tests
* belonging to tags mentioned in the tagsToInclude set, and not mentioned in tagsToExclude,
* will be run.
*
*
*
* Shared fixtures
*
*
* A test fixture is composed of the objects and other artifacts (files, sockets, database
* connections, etc.) tests use to do their work.
* When multiple tests need to work with the same fixtures, it is important to try and avoid
* duplicating the fixture code across those tests. The more code duplication you have in your
* tests, the greater drag the tests will have on refactoring the actual production code.
*
*
*
* ScalaTest recommends three techniques to eliminate such code duplication:
*
*
*
* - Refactor using Scala
* - Override
withFixture
* - Mix in a before-and-after trait
*
*
* Each technique is geared towards helping you reduce code duplication without introducing
* instance vars, shared mutable objects, or other dependencies between tests. Eliminating shared
* mutable state across tests will make your test code easier to reason about and more amenable for parallel
* test execution.
*
*
* The techniques in PropSpec are identical to those in FunSuite, but with “test”
* replaced by “property”. The following table summarizes the options with a link to the relevant
* documentation for trait FunSuite:
*
*
*
*
*
*
* Refactor using Scala when different tests need different fixtures.
*
*
*
*
*
* get-fixture methods
*
*
* The extract method refactor helps you create a fresh instances of mutable fixture objects in each test
* that needs them, but doesn't help you clean them up when you're done.
*
*
*
*
*
* fixture-context objects
*
*
* By placing fixture methods and fields into traits, you can easily give each test just the newly created
* fixtures it needs by mixing together traits. Use this technique when you need different combinations
* of mutable fixture objects in different tests, and don't need to clean up after.
*
*
*
*
*
* loan-fixture methods
*
*
* Factor out dupicate code with the loan pattern when different tests need different fixtures that must be cleaned up afterwards.
*
*
*
*
*
* Override withFixture when most or all tests need the same fixture.
*
*
*
*
*
*
* withFixture(NoArgTest)
*
*
*
* The recommended default approach when most or all tests need the same fixture treatment. This general technique
* allows you, for example, to perform side effects at the beginning and end of all or most tests,
* transform the outcome of tests, retry tests, make decisions based on test names, tags, or other test data.
* Use this technique unless:
*
*
* - Different tests need different fixtures (refactor using Scala instead)
* - An exception in fixture code should abort the suite, not fail the test (use a before-and-after trait instead)
* - You have objects to pass into tests (override
withFixture(OneArgTest) instead)
*
*
*
*
*
*
*
* withFixture(OneArgTest)
*
*
*
* Use when you want to pass the same fixture object or objects as a parameter into all or most tests.
*
*
*
*
*
* Mix in a before-and-after trait when you want an aborted suite, not a failed test, if the fixture code fails.
*
*
*
*
*
* BeforeAndAfter
*
*
* Use this boilerplate-buster when you need to perform the same side-effects before and/or after tests, rather than at the beginning or end of tests.
*
*
*
*
*
* BeforeAndAfterEach
*
*
* Use when you want to stack traits that perform the same side-effects before and/or after tests, rather than at the beginning or end of tests.
*
*
*
*
*
*
* Using PropSpec to implement a test matrix
*
*
* Using fixture-context objects in a PropSpec is a good way to implement a test matrix.
* What is the matrix? A test matrix is a series of tests that you need to run on a series of subjects. For example, The Scala API contains
* many implementations of trait Set. Every implementation must obey the contract of Set.
* One property of any Set is that an empty Set should have size 0, another is that
* invoking head on an empty Set should give you a NoSuchElementException, and so on. Already you have a matrix,
* where rows are the properties and the columns are the set implementations:
*
*
*
* BitSetHashSetTreeSetAn empty Set should have size 0 pass pass pass Invoking head on an empty set should produce NoSuchElementException pass pass pass
*
*
* One way to implement this test matrix is to define a trait to represent the columns (in this case, BitSet, HashSet,
* and TreeSet) as elements in a single-dimensional Table. Each element in the Table represents
* one Set implementation. Because different properties may require different fixture instances for those implementations, you
* can define a trait to hold the examples, like this:
*
*
* trait SetExamples extends Tables {
*
* def examples = Table("set", bitSet, hashSet, treeSet)
*
* def bitSet: BitSet
* def hashSet: HashSet[Int]
* def treeSet: TreeSet[Int]
* }
*
*
*
* Given this trait, you could provide empty sets in one implementation of SetExamples, and non-empty sets in another.
* Here's how you might provide empty set examples:
*
*
*
* class EmptySetExamples extends SetExamples {
* def bitSet = BitSet.empty
* def hashSet = HashSet.empty[Int]
* def treeSet = TreeSet.empty[Int]
* }
*
*
*
* And here's how you might provide set examples with one item each:
*
*
*
* class SetWithOneItemExamples extends SetExamples {
* def bitSet = BitSet(1)
* def hashSet = HashSet(1)
* def treeSet = TreeSet(1)
* }
*
*
*
* Armed with these example classes, you can define checks of properties that require
* empty or non-empty set fixtures by using instances of these classes as fixture-context
* objects. In other words, the columns of the test matrix are implemented as elements of
* a one-dimensional table of fixtures, the rows are implemented as property
* clauses of a PropSpec.
*
*
*
* Here's a complete example that checks the two properties mentioned previously:
*
*
*
* package org.scalatest.examples.propspec.matrix
*
* import org.scalatest._
* import org.scalatest.prop._
* import scala.collection.immutable._
*
* trait SetExamples extends Tables {
*
* def examples = Table("set", bitSet, hashSet, treeSet)
*
* def bitSet: BitSet
* def hashSet: HashSet[Int]
* def treeSet: TreeSet[Int]
* }
*
* class EmptySetExamples extends SetExamples {
* def bitSet = BitSet.empty
* def hashSet = HashSet.empty[Int]
* def treeSet = TreeSet.empty[Int]
* }
*
* class SetSpec extends PropSpec with TableDrivenPropertyChecks with ShouldMatchers {
*
* property("an empty Set should have size 0") {
* new EmptySetExamples {
* forAll(examples) { set =>
* set.size should be (0)
* }
* }
* }
*
* property("invoking head on an empty set should produce NoSuchElementException") {
* new EmptySetExamples {
* forAll(examples) { set =>
* evaluating { set.head } should produce [NoSuchElementException]
* }
* }
* }
* }
*
*
*
* One benefit of this approach is that the compiler will help you when you need to add either a new row
* or column to the matrix. In either case, you'll need to ensure all cells are checked to get your code to compile.
*
*
* Shared tests
*
*
* Sometimes you may want to run the same test code on different fixture objects. That is to say, you may want to write tests that are "shared"
* by different fixture objects.
* You accomplish this in a PropSpec in the same way you would do it in a FunSuite, except instead of test
* you say property, and instead of testsFor you say propertiesFor.
* For more information, see the Shared tests section of FunSuite's
* documentation.
*
*
* @author Bill Venners
*/
@Finders(Array("org.scalatest.finders.PropSpecFinder"))
class PropSpec extends PropSpecLike {
/**
* Returns a user friendly string for this suite, composed of the
* simple name of the class (possibly simplified further by removing dollar signs if added by the Scala interpeter) and, if this suite
* contains nested suites, the result of invoking toString on each
* of the nested suites, separated by commas and surrounded by parentheses.
*
* @return a user-friendly string for this suite
*/
override def toString: String = Suite.suiteToString(None, this)
}