org.scalatest.tools.Runner.scala Maven / Gradle / Ivy

Go to download
Show more of this group Show more artifacts with this name
Show all versions of scalatest_2.9.0 Show documentation
ScalaTest is a free, open-source testing toolkit for Scala and Java programmers.
There is a newer version: 2.0.M6-SNAP4
/*
 * Copyright 2001-2008 Artima, Inc.
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 *     http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
// TODO: Scaladoc for -F. Forgot this in 1.8. If have a 1.8.1, or a 1.9, then put it in there too.
package org.scalatest.tools

import org.scalatest._
import scala.collection.mutable.ListBuffer
import java.lang.reflect.Constructor
import java.lang.reflect.Modifier
import java.net.URL
import java.net.MalformedURLException
import java.net.URLClassLoader
import java.io.File
import java.io.IOException
import javax.swing.SwingUtilities
import java.util.concurrent.ArrayBlockingQueue
import java.util.regex.Pattern
import org.scalatest.testng.TestNGWrapperSuite
import java.util.concurrent.Semaphore
import org.scalatest.events._
import org.scalatest.junit.JUnitWrapperSuite
import java.util.concurrent.Executors
import java.util.concurrent.ExecutorService
import scala.collection.mutable.ArrayBuffer
import SuiteDiscoveryHelper._
import org.scalatest.time.Span
import org.scalatest.time.Seconds
import org.scalatest.time.Millis

private[tools] case class SuiteParam(className: String, testNames: Array[String], wildcardTestNames: Array[String], nestedSuites: Array[NestedSuiteParam])
private[tools] case class NestedSuiteParam(suiteId: String, testNames: Array[String], wildcardTestNames: Array[String])
private[scalatest] case class ConcurrentConfig(numThreads: Int, enableSuiteSortingReporter: Boolean)

/**
 * Application that runs a suite of tests.
 *
 * 
 * Note: this application offers the full range of ScalaTest features via command line arguments described below. If you just want
 * to run a suite of tests from the command line and see results on the standard output, you may prefer to use ScalaTest's simple runner.
 * 
 *
 * 
 * The basic form of a Runner invocation is:
 * 
 *
 *  * scala [-cp scalatest-<version>.jar:...] org.scalatest.tools.Runner [arguments]
 * 
 *
 * 
 * The arguments Runner accepts are described in the following table:
 * 
 *
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * 
 * argument description example
-Dkey=value defines a key/value pair for the config map -DmaxConnections=100
-R <runpath elements> the specifies the runpath from which tests classes will be
discovered and loaded (Note: only one -R allowed) Unix: -R target/classes:target/generated/classes
Windows: -R target\classes;target\generated\classes
-n <tag name> specifies a tag to include (Note: only one tag name allowed per -n) -n UnitTests -n FastTests
-l <tag name> specifies a tag to exclude (Note: only one tag name allowed per -l) -l SlowTests -l PerfTests
-P[S][integer thread count] specifies a parallel run, with optional suite sorting and thread count
(Note: only one -P allowed) -P, -PS, -PS 8, or -P8
-s <suite class name> specifies a suite class to run -s com.company.project.StackSpec
-m <members-only package> requests that suites that are direct members of the specified package
 be discovered and run -m com.company.project
-w <wildcard package> requests that suites that are members of the specified package or its subpackages
be discovered and run -w com.company.project
-q <suffixes> specify suffixes to discover -q Spec -q Suite
-Q discover only classes whose names end with Spec or Suite
(or other suffixes specified by -q) -Q
-j <JUnit class name> instantiate and run a JUnit test class -j StackTestClass
-b <TestNG XML file> run TestNG tests using the specified TestNG XML file -b testng.xml
-F <span scale factor> a factor by which to scale time spans
(Note: only one -F is allowed) -F 10 or -F 2.5
-T <sorting timeout> specifies a integer timeout (in seconds) for sorting the events of
parallel runs back into sequential order -T 5
-y <chosen styles> specifies chosen styles for your project -y org.scalatest.FlatSpec
-i <suite ID> specifies a suite to run by ID (Note: must follow -s, 
and is intended to be used primarily by tools such as IDEs.) -i com.company.project.FileSpec-file1.txt
-t <test name> select the test with the specified name
(Note: must follow -s or -i) -t "An empty Stack should complain when popped"
-z <test name substring> select tests whose names include the specified substring
(Note: must follow -s or -i) -z "popped"
-g[NCXEHLOPQMD] select the graphical reporter -g
-f[NCXEHLOPQMDWSFU] <filename> select the file reporter -f output.txt
-u <directory name> select the JUnit XML reporter -u target/junitxmldir
-h <directory name> [-Y <css file name>] select the HTML reporter, optionally including the specified CSS file -h target/htmldir -Y src/main/html/customStyles.css
-v print the ScalaTest version -v or, also -version
-o[NCXEHLOPQMDWSFU] select the standard output reporter -o
-e[NCXEHLOPQMDWSFU] select the standard error reporter -e
-C[NCXEHLOPQMD] <reporter class> select a custom reporter -C com.company.project.BarReporter
 *
 * 
 * The simplest way to start Runner is to specify the directory containing your compiled tests as the sole element of the runpath, for example:
 * 
 *
 * scala -classpath scalatest-<version>.jar org.scalatest.tools.Runner -R compiled_tests
 *
 * 
 * Given the previous command, Runner will discover and execute all Suites in the compiled_tests directory and its subdirectories,
 * and show results in graphical user interface (GUI).
 * 
 *
 * 
 * Executing suites
 *
 * 
 * Each -s argument must be followed by one and only one fully qualified class name. The class must either extend Suite and
 * have a public, no-arg constructor, or be annotated by a valid WrapWith annotation.
 * 
 *
 * 
 * Specifying the config map
 *
 * 
 * A config map contains pairs consisting of a string key and a value that may be of any type. (Keys that start with
 * "org.scalatest." are reserved for ScalaTest. Configuration values that are themselves strings may be specified on the
 * Runner command line.
 * Each configuration pair is denoted with a "-D", followed immediately by the key string, an "=", and the value string.
 * For example:
 * 
 *
 * -Ddbname=testdb -Dserver=192.168.1.188
 *
 * 
 * Specifying a runpath
 *
 * 
 * A runpath is the list of filenames, directory paths, and/or URLs that Runner
 * uses to load classes for the running test. If runpath is specified, Runner creates
 * a custom class loader to load classes available on the runpath.
 * The graphical user interface reloads the test classes anew for each run
 * by creating and using a new instance of the custom class loader for each run.
 * The classes that comprise the test may also be made available on
 * the classpath, in which case no runpath need be specified.
 * 
 *
 * 
 * The runpath is specified with the -R option. The -R must be followed by a space,
 * a double quote ("), a white-space-separated list of
 * paths and URLs, and a double quote. If specifying only one element in the runpath, you can leave off
 * the double quotes, which only serve to combine a white-space separated list of strings into one
 * command line argument. If you have path elements that themselves have a space in them, you must
 * place a backslash (\) in front of the space. Here's an example:
 * 
 *
 * -R "serviceuitest-1.1beta4.jar myjini http://myhost:9998/myfile.jar target/class\ files"
 *
 * 
 * Specifying reporters
 *
 * 
 * Reporters can be specified  on the command line in any of the following
 * ways:
 * 
 *
 * 
 *  -g[configs...] - causes display of a graphical user interface that allows
 *    tests to be run and results to be investigated
 *  -f[configs...] <filename> - causes test results to be written to
 *     the named file
 *  -u <directory> - causes test results to be written to
 *      junit-style xml files in the named directory
 *  -h <directory> [-Y <CSS file>] - causes test results to be written to
 *      HTML files in the named directory, optionally included the specified CSS file
 *  -a <number of files to archive> - causes specified number of old
 *      summary and durations files to be archived (in summaries/ and durations/ subdirectories)
 *      for dashboard reporter (default is two)
 *  -o[configs...] - causes test results to be written to
 *     the standard output
 *  -e[configs...] - causes test results to be written to
 *     the standard error
 *  -C[configs...] <reporterclass> - causes test results to be reported to
 *     an instance of the specified fully qualified Reporter class name
 * 
 *
 * 
 * The [configs...] parameter, which is used to configure reporters, is described in the next section.
 * 
 *
 * 
 * The -C option causes the reporter specified in
 * <reporterclass> to be
 * instantiated.
 * Each reporter class specified with a -C option must be public, implement
 * org.scalatest.Reporter, and have a public no-arg constructor.
 * Reporter classes must be specified with fully qualified names. 
 * The specified reporter classes may be
 * deployed on the classpath. If a runpath is specified with the
 * -R option, the specified reporter classes may also be loaded from the runpath.
 * All specified reporter classes will be loaded and instantiated via their no-arg constructor.
 * 
 *
 * 
 * For example, to run a suite named MySuite from the mydir directory
 * using two reporters, the graphical reporter and a file reporter
 * writing to a file named "test.out", you would type:
 * 
 *
 * java -jar scalatest.jar -R mydir -g -f test.out -s MySuite
 *
 * 
 * The -g, -o, or -e options can
 * appear at most once each in any single command line.
 * Multiple appearances of -f and -C result in multiple reporters
 * unless the specified <filename> or <reporterclass> is
 * repeated. If any of -g, -o, -e,
 * <filename> or <reporterclass> are repeated on
 * the command line, the Runner will print an error message and not run the tests.
 * 
 *
 * 
 * Runner adds the reporters specified on the command line to a dispatch reporter,
 * which will dispatch each method invocation to each contained reporter. Runner will pass
 * the dispatch reporter to executed suites. As a result, every
 * specified reporter will receive every report generated by the running suite of tests.
 * If no reporters are specified, a graphical
 * runner will be displayed that provides a graphical report of
 * executed suites.
 * 
 *
 * 
 * Configuring Reporters
 * 
 *
 * 
 * Each reporter option on the command line can include configuration characters. Configuration characters
 * are specified immediately following the -g, -o,
 * -e, -f, or -C. The following configuration
 * characters, which cause reports to be dropped, are valid for any reporter:
 * 
 *
 * 
 *  N - drop TestStarting events
 *  C - drop TestSucceeded events
 *  X - drop TestIgnored events
 *  E - drop TestPending events
 *  H - drop SuiteStarting events
 *  L - drop SuiteCompleted events
 *  O - drop InfoProvided events
 *  P - drop ScopeOpened events
 *  Q - drop ScopeClosed events
 *  M - drop MarkupProvided events
 * 
 *
 * 
 * A dropped event will not be delivered to the reporter at all. So the reporter will not know about it and therefore not
 * present information about the event in its report. For example, if you specify -oN, the standard output reporter
 * will never receive any TestStarting events and will therefore never report them. The purpose of these
 * configuration parameters is to allow users to selectively remove events they find add clutter to the report without
 * providing essential information.
 * 
 *
 * 
 * The following three reporter configuration parameters may additionally be used on standard output (-o), standard error (-e),
 * and file (-f) reporters: 
 * 
 *
 * 
 *  W - without color
 *  D - show all durations
 *  S - show short stack traces
 *  F - show full stack traces
 *  U - unformatted mode
 * 
 *
 * 
 * If you specify a W, D, S, F, or U for any reporter other than standard output, standard error, or file reporters, Runner
 * will complain with an error message and not perform the run.
 * 
 *
 * 
 * Configuring a standard output, error, or file reporter with D will cause that reporter to
 * print a duration for each test and suite.  When running in the default mode, a duration will only be printed for
 * the entire run.
 * 
 *
 * 
 * Configuring a standard output, error, or file reporter with F will cause that reporter to print full stack traces for all exceptions,
 * including TestFailedExceptions. Every TestFailedException contains a stack depth of the
 * line of test code that failed so that users won't need to search through a stack trace to find it. When running in the default,
 * mode, these reporters will only show full stack traces when other exceptions are thrown, such as an exception thrown
 * by production code. When a TestFailedException is thrown in default mode, only the source filename and
 * line number of the line of test code that caused the test to fail are printed along with the error message, not the full stack
 * trace. 
 * 
 *
 * 
 * The 'U' unformatted configuration removes some formatting from the output and adds verbosity.
 * The purpose of unformatted (or, "ugly") mode is to facilitate debugging of parallel runs. If you have tests that fail or hang during parallel runs, but
 * succeed when run sequentially, unformatted mode can help. In unformatted mode, you can see exactly what is happening when it is happening. Rather than attempting to make the output
 * look as pretty and human-readable as possible, unformatted mode will just print out verbose information about each event as it arrives, helping you track down the problem
 * you are trying to debug.
 * 
 *
 * 
 * By default, a standard output, error, or file reporter inserts ansi escape codes into the output printed to change and later reset
 * terminal colors. Information printed as a result of run starting, completed, and stopped events
 * is printed in cyan. Information printed as a result of ignored or pending test events is shown in yellow. Information printed
 * as a result of test failed, suite aborted, or run aborted events is printed in red. All other information is printed in green.
 * The purpose of these colors is to facilitate speedy reading of the output, especially the finding of failed tests, which can
 * get lost in a sea of passing tests. Configuring a standard output, error, or file reporter into without-color mode ('W') will
 * turn off this behavior. No ansi codes will be inserted.
 * 
 *
 * 
 * For example, to run a suite using two reporters, the graphical reporter configured to present every reported event
 * and a standard error reporter configured to present everything but test starting, test succeeded, test ignored, test
 * pending, suite starting, suite completed, and info provided events, you would type:
 * 
 *
 * 
 * scala -classpath scalatest-<version>.jar -R mydir -g -eNDXEHLO -s MySuite
 * 
 *
 * 
 * Note that no white space is allowed between the reporter option and the initial configuration
 * parameters. So "-e NDXEHLO" will not work,
 * "-eNDXEHLO" will work.
 * 
 *
 * 
 * Specifying tags to include and exclude
 * 
 *
 * 
 * You can specify tag names of tests to include or exclude from a run. To specify tags to include,
 * use -n followed by a white-space-separated list of tag names to include, surrounded by
 * double quotes. (The double quotes are not needed if specifying just one tag.)  Similarly, to specify tags
 * to exclude, use -l followed by a white-space-separated
 * list of tag names to exclude, surrounded by double quotes. (As before, the double quotes are not needed
 * if specifying just one tag.) If tags to include is not specified, then all tests
 * except those mentioned in the tags to exclude (and in the org.scalatest.Ignore tag), will be executed.
 * (In other words, the absence of a -n option is like a wildcard, indicating all tests be included.)
 * If tags to include is specified, then only those tests whose tags are mentioned in the argument following -n
 * and not mentioned in the tags to exclude, will be executed. For more information on test tags, see
 * the documentation for Suite. Here are some examples:
 * 
 *
 * 
 * 

 * -n CheckinTests
 * -n FunctionalTests -l SlowTests
 * -n "CheckinTests FunctionalTests" -l "SlowTests NetworkTests"
 * 
 * 
 *
 * 
 * Specifying suffixes to discover
 * 
 *
 * 
 * You can specify suffixes of Suite names to discover. To specify suffixes to discover,
 * use -q followed by a vertical-bar-separated list of suffixes to discover, surrounded by
 * double quotes. (The double quotes are not needed if specifying just one suffix.)  Or you can specify
 * them individually using multiple -q's.
 * If suffixes to discover is not specified, then all suffixes are considered.
 * If suffixes is specified, then only those Suites whose class names end in one of the specified suffixes
 * will be considered during discovery. Here are some examples:
 * 
 *
 * 
 * 

 * -q Spec
 * -q "Spec|Suite"
 * -q Spec -q Suite
 * 
 * 
 *
 * 
 * Option -Q can be used to specify a default set of suffixes "Spec|Suite". If you specify both -Q and -q, you'll get Spec
 * and Suite in addition to the other suffix or suffixes you specify with -q.
 * 
 *
 * 
 * Specifying suffixes can speed up the discovery process because class files with names not ending the specified suffixes
 * can be immediately disqualified, without needing to load and inspect them to see if they either extend Suite
 * and declare a public, no-arg constructor, or are annotated with WrapWith. 
 * 
 *
 * 
 * Executing Suites in parallel
 * 
 *
 * 
 * With the proliferation of multi-core architectures, and the often parallelizable nature of tests, it is useful to be able to run
 * tests in parallel. If you include -P on the command line, Runner will pass a Distributor to
 * the Suites you specify with -s. Runner will set up a thread pool to execute any Suites
 * passed to the Distributor's put method in parallel. Trait Suite's implementation of
 * runNestedSuites will place any nested Suites into this Distributor. Thus, if you have a Suite
 * of tests that must be executed sequentially, you should override runNestedSuites as described in the documentation for Distributor.
 * 
 *
 * 
 * The -P option may optionally be appended with a number (e.g.
 * "-P10" -- no intervening space) to specify the number of
 * threads to be created in the thread pool.  If no number (or 0) is
 * specified, the number of threads will be decided based on the number of
 * processors available.
 * 
 *
 * 
 * Specifying Suites
 * 
 *
 * 
 * Suites are specified on the command line with a -s followed by the fully qualified
 * name of a Suite subclass, as in:
 * 
 *
 * -s com.artima.serviceuitest.ServiceUITestkit
 *
 * 
 * Each specified suite class must be public, a subclass of
 * org.scalatest.Suite, and contain a public no-arg constructor.
 * Suite classes must be specified with fully qualified names. 
 * The specified Suite classes may be
 * loaded from the classpath. If a runpath is specified with the
 * -R option, specified Suite classes may also be loaded from the runpath.
 * All specified Suite classes will be loaded and instantiated via their no-arg constructor.
 * 
 *
 * 
 * The runner will invoke execute on each instantiated org.scalatest.Suite,
 * passing in the dispatch reporter to each execute method.
 * 
 *
 * 
 * Runner is intended to be used from the command line. It is included in org.scalatest
 * package as a convenience for the user. If this package is incorporated into tools, such as IDEs, which take
 * over the role of runner, object org.scalatest.tools.Runner may be excluded from that implementation of the package.
 * All other public types declared in package org.scalatest.tools.Runner should be included in any such usage, however,
 * so client software can count on them being available.
 * 
 *
 * 
 * Specifying "members-only" and "wildcard" Suite paths
 * 
 *
 * 
 * If you specify Suite path names with -m or -w, Runner will automatically
 * discover and execute accessible Suites in the runpath that are either a member of (in the case of -m)
 * or enclosed by (in the case of -w) the specified path. As used in this context, a path is a portion of a fully qualified name.
 * For example, the fully qualifed name com.example.webapp.MySuite contains paths com, com.example, and com.example.webapp.
 * The fully qualifed name com.example.webapp.MyObject.NestedSuite contains paths com, com.example,
 * com.example.webapp, and com.example.webapp.MyObject.
 * An accessible Suite is a public class that extends org.scalatest.Suite
 * and defines a public no-arg constructor. Note that Suites defined inside classes and traits do not have no-arg constructors,
 * and therefore won't be discovered. Suites defined inside singleton objects, however, do get a no-arg constructor by default, thus
 * they can be discovered.
 * 
 *
 * 
 * For example, if you specify -m com.example.webapp
 * on the command line, and you've placed com.example.webapp.RedSuite and com.example.webapp.BlueSuite
 * on the runpath, then Runner will instantiate and execute both of those Suites. The difference
 * between -m and -w is that for -m, only Suites that are direct members of the named path
 * will be discovered. For -w, any Suites whose fully qualified
 * name begins with the specified path will be discovered. Thus, if com.example.webapp.controllers.GreenSuite
 * exists on the runpath, invoking Runner with -w com.example.webapp will cause GreenSuite
 * to be discovered, because its fully qualifed name begins with "com.example.webapp". But if you invoke Runner
 * with -m com.example.webapp, GreenSuite will not be discovered because it is directly
 * a member of com.example.webapp.controllers, not com.example.webapp.
 * 
 *
 * 
 * If you specify no -s, -m, or -w arguments on the command line to Runner, it will discover and execute all accessible Suites
 * in the runpath.
 * 
 *
 * 
 * Specifying chosen styles
 * 
 *
 * 
 * You can optionally specify chosen styles for a ScalaTest run. ScalaTest supports different styles of
 * testing so that different teams can use the style or styles that best suits their situation and culture. But
 * in any one project, it is recommended you decide on one main style for unit testing, and
 * consistently use only that style for unit testing throughout the project. If you also have integration
 * tests in your project, you may wish to pick a different style for them than you are using for unit testing.
 * You may want to allow certain styles to be used in special testing situations on a project, but in general,
 * it is best to minimize the styles used in any given project to a few, or one.
 * 
 *
 * 
 * To facilitate the communication and enforcement of a team's style choices for a project, you can
 * specify the chosen styles in your project build. If chosen styles is defined, ScalaTest style traits that are
 * not among the chosen list will abort with a message complaining that the style trait is not one of the
 * chosen styles. The style name for each ScalaTest style trait is its fully qualified name. For example,
 * to specify that org.scalatest.FunSpec as your chosen style you'd pass this to
 * Runner:
 * 
 *
 * -y org.scalatest.FunSpec
 *
 * 
 * If you wanted org.scalatest.FunSpec as your main unit testing style, but also wanted to
 * allow PropSpec for test matrixes and FeatureSpec for
 * integration tests, you would write:
 * 
 *
 * -y org.scalatest.FunSpec -y org.scalatest.PropSpec -y org.scalatest.FeatureSpec
 *
 * 
 * To select org.scalatest.FlatSpec as your main unit testing style, but allow
 * org.scalatest.fixture.FlatSpec for multi-threaded unit tests, you'd write:
 * 
 *
 * -y org.scalatest.FlatSpec -y org.scalatest.fixture.FlatSpec
 *
 * 
 * The style name for a suite is obtained by invoking its styleName method. Custom style
 * traits can override this method so that a custom style can participate in the chosen styles list.
 * 
 *
 * 
 * Because ScalaTest is so customizable, a determined programmer could circumvent
 * the chosen styles check, but in practice -y should be persuasive enough tool
 * to keep most team members in line.
 * 
 *
 * 
 * Selecting suites and tests
 *
 * 
 * Runner accepts three arguments that facilitate selecting suites and tests: -i, -t, and -z.
 * The -i option enables a suite to be selected by suite ID. This argument is intended to allow tools such as IDEs or build tools to
 * rerun specific tests or suites from information included in the results of a previous run.  A -i must follow a -s
 * that specifies a class with a public, no-arg constructor. The -i parameter can be used, for example, to rerun a nested suite that
 * declares no zero-arg constructor, which was created by containing suite that does declare a no-arg constructor. In this case, -s would be
 * used to specify the class ScalaTest can instantiate directly, the containing suite that has a public, no-arg constructor, and -i would be
 * used to select the desired nested suite. One important use case for -i is to enable such a nested suite that aborted during the previous run
 * to be rerun. 
 * 
 *
 * 
 * The -t argument allows a test to be selected by its (complete) test name. Like -i, the -t argument is primarily intented
 * to be used by tools such as IDEs or build tools, to rerun selected tests based on information obtained from the results of a previous run.
 * For example, -t could be used to rerun a test that failed in the previous run.
 * The -t argument can be used directly by users, but because descriptive test names are usually rather long, the -z argument (described next), will
 * usually be a more practical choice for users. A -t must follow either -s or -i, which identifies the suite
 * to which the test belongs.
 * 
 *
 * 
 * The -z option allows tests to be selected by a simplified wildcard: any test whose name includes the substring specified after -z
 * will be selected. For example, -z popped would select tests named "An empty stack should complain when popped" and "A non-empty stack
 * should return the last-pushed value when popped, but not "An empty stack should be empty". In short, -z popped would select any
 * tests whose name includes the substring "popped", and not select any tests whose names don't include "popped". This simplified
 * approach to test name wildcards, which was suggested by Mathias Doenitz, works around the difficulty of finding an actual wildcard character that will work
 * reliably on different operating systems. 
 * 
 *
 * 
 * Specifying a span scale factor
 *
 * 
 * If you specify a integer or floating point span scale factor with -F, trait ScaledTimeSpans
 * trait will  return the specified value from its implementation of spanScaleFactor. This allows you to tune the "patience" of a run (how long to wait
 * for asynchronous operations) from the command line. For more information, see the documentation for trait ScaledTimeSpans.
 * 
 *
 * 
 * Specifying TestNG XML config file paths
 *
 * 
 * If you specify one or more file paths with -b (b for Beust, the last name of TestNG's creator), Runner will create a org.scalatest.testng.TestNGWrapperSuite,
 * passing in a List of the specified paths. When executed, the TestNGWrapperSuite will create one TestNG instance
 * and pass each specified file path to it for running. If you include -b arguments, you must include TestNG's jar file on the class path or runpath.
 * The -b argument will enable you to run existing TestNG tests, including tests written in Java, as part of a ScalaTest run.
 * You need not use -b to run suites written in Scala that extend TestNGSuite. You can simply run such suites with 
 * -s, -m, or -w parameters.
 * 
 *
 * 
 * Specifying JUnit tests
 *
 * 
 * JUnit tests, including ones written in Java, may be run by specifying
 * -j classname, where the classname is a valid JUnit class
 * such as a TestCase, TestSuite, or a class implementing a static suite()
 * method returning a TestSuite. 
 * 
 * To use this option you must include a JUnit jar file on your classpath.
 * 
 *
 * @author Bill Venners
 * @author George Berger
 * @author Josh Cough
 */
object Runner {

  private val RUNNER_JFRAME_START_X: Int = 150
  private val RUNNER_JFRAME_START_Y: Int = 100
  
  private val SELECTED_TAG = "org.scalatest.Selected"
  
  @volatile private[scalatest] var spanScaleFactor: Double = 1.0

  private final val DefaultNumFilesToArchive = 2

  @volatile private[scalatest] var testSortingReporterTimeout = Span(2, Seconds)
  
  //                     TO
  // We always include a PassFailReporter on runs in order to determine
  // whether or not all tests passed.
  //
  // The thread that calls Runner.run() will either start a GUI, if a graphic
  // reporter was requested, or just run the tests itself. If a GUI is started,
  // an event handler thread will get going, and it will start a RunnerThread,
  // which will actually do the running. The GUI can repeatedly start RunnerThreads
  // and RerunnerThreads, until the GUI is closed. If -P is specified, that means
  // the tests should be run concurrently, which in turn means a Distributor will
  // be passed to the execute method of the Suites, which will in turn populate
  // it with its nested suites instead of executing them directly in the same
  // thread. The Distributor works in conjunction with a pool of threads that
  // will take suites out of the distributor queue and execute them. The DispatchReporter
  // will serialize all reports via an actor, which because that actor uses receive
  // not react, will have its own thread. So the DispatchReporter actor's thread will
  // be the one that actually invokes TestFailed, RunAborted, etc., on this PassFailReporter.
  // The thread that invoked Runner.run(), will be the one that calls allTestsPassed.
  //
  // The thread that invoked Runner.run() will be the one to instantiate the PassFailReporter
  // and in its primary constructor acquire the single semaphore permit. This permit will
  // only be released by the DispatchReporter's actor thread when a runAborted, runStopped,
  // or runCompleted is invoked. allTestsPassed will block until it can reacquire the lone
  // semaphore permit. Thus, a PassFailReporter can just be used for one run, then it is
  // spent. A new PassFailReporter is therefore created each time the Runner.run() method is invoked.
  //
  private class PassFailReporter extends Reporter {

    @volatile private var failedAbortedOrStopped = false
    private val runDoneSemaphore = new Semaphore(1)
    runDoneSemaphore.acquire()

    override def apply(event: Event) {
      event match {
        case _: TestFailed =>
          failedAbortedOrStopped = true

        case _: RunAborted =>
          failedAbortedOrStopped = true
          runDoneSemaphore.release()

        case _: SuiteAborted =>
          failedAbortedOrStopped = true

        case _: RunStopped =>
          failedAbortedOrStopped = true
          runDoneSemaphore.release() 

        case _: RunCompleted =>
          runDoneSemaphore.release()

        case _ =>
      }
    }

    def allTestsPassed = {
      runDoneSemaphore.acquire()
      !failedAbortedOrStopped
    }
  }

  // TODO: I don't think I'm enforcing that properties can't start with "org.scalatest"
  // TODO: I don't think I'm handling rejecting multiple -f/-C with the same arg. -f fred.txt -f fred.txt should
  // fail, as should -C MyReporter -C MyReporter. I'm failing on -o -o, -g -g, and -e -e, but the error messages
  // could indeed be nicer.
  /**
   * Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details.
   */
  def main(args: Array[String]) {
    // println("FOR DEBUGGING, THESE ARE THE ARGS PASSED TO main(): " + args.mkString(" "))
    val result = 
      if (args.contains("-v") || args.contains("--version")) {
        val version = Resources("AppVersion")
        val scalaVersion = Resources("ScalaVersion")
        println("ScalaTest " + version + " (Built for Scala " + scalaVersion + ")")
        runOptionallyWithPassFailReporter(args.filter(arg => arg != "-v" && arg != "--version"), true)
      }
      else
        runOptionallyWithPassFailReporter(args, true)

    if (result)
      exit(0)
    else
      exit(1)
  }

  /**
   * Runs a suite of tests, with optional GUI. See the main documentation for this singleton object for the details.
   * The difference between this method and main is simply that this method will block until the run
   * has completed, aborted, or been stopped, and return true if all tests executed and passed. In other
   * words, if any test fails, or if any suite aborts, or if the run aborts or is stopped, this method will
   * return false. This value is used, for example, by the ScalaTest ant task to determine whether
   * to continue the build if haltOnFailure is set to true.
   *
   * @return true if all tests were executed and passed.
   */
  def run(args: Array[String]): Boolean = {
    // println("FOR DEBUGGING, THESE ARE THE ARGS PASSED TO run(): " + args.mkString(" "))
    runOptionallyWithPassFailReporter(args, true)
  }
  
  def parseFriendlyParams(friendlyArgs:String): Array[String] = {
    parseFriendlyParams(friendlyArgs.split(" "))
  }

  def parseFriendlyParams(friendlyArgs:Array[String]): Array[String] = {
    val (propsList, includesList, excludesList, repoArgsList, concurrentList, memberOnlyList, wildcardList, suiteList, junitList, testngList) = 
      new FriendlyParamsTranslator().parsePropsAndTags(friendlyArgs)
    val arrayBuffer = new ArrayBuffer[String]()
    arrayBuffer ++= propsList ::: includesList ::: excludesList ::: repoArgsList ::: concurrentList ::: memberOnlyList ::: wildcardList :::
                    suiteList ::: junitList ::: testngList
    arrayBuffer.toArray
  }

  private def runOptionallyWithPassFailReporter(args: Array[String], runWithPassFailReporter: Boolean): Boolean = {

    checkArgsForValidity(args) match {
      case Some(s) => {
        println(s)
        exit(1) // TODO: Shouldn't this be returning false?
      }
      case None =>
    }

    val (
      runpathArgsList,
      reporterArgsList,
      suiteArgsList,
      junitArgsList,
      propertiesArgsList,
      includesArgsList,
      excludesArgsList,
      concurrentList,
      membersOnlyArgsList,
      wildcardArgsList,
      testNGArgsList,
      suffixes, 
      chosenStyles, 
      spanScaleFactors, 
      testSortingReporterTimeouts
    ) = parseArgs(args)

    val fullReporterConfigurations: ReporterConfigurations =
      if (reporterArgsList.isEmpty)
        // If no reporters specified, just give them a graphic reporter
        new ReporterConfigurations(Some(GraphicReporterConfiguration(Set())), Nil, Nil, Nil, Nil, None, None, Nil, Nil, Nil)
      else
        parseReporterArgsIntoConfigurations(reporterArgsList)

    val suitesList: List[SuiteParam] = parseSuiteArgsIntoSuiteParam(suiteArgsList, "-s")
    val junitsList: List[String] = parseSuiteArgsIntoNameStrings(junitArgsList, "-j")
    val runpathList: List[String] = parseRunpathArgIntoList(runpathArgsList)
    val propertiesMap: Map[String, Object] = parsePropertiesArgsIntoMap(propertiesArgsList)
    val tagsToInclude: Set[String] = parseCompoundArgIntoSet(includesArgsList, "-n")
    val tagsToExclude: Set[String] = parseCompoundArgIntoSet(excludesArgsList, "-l")
    val concurrent: Boolean = !concurrentList.isEmpty
    val concurrentConfig: ConcurrentConfig = parseConcurrentConfig(concurrentList)
    val membersOnlyList: List[String] = parseSuiteArgsIntoNameStrings(membersOnlyArgsList, "-m")
    val wildcardList: List[String] = parseSuiteArgsIntoNameStrings(wildcardArgsList, "-w")
    val testNGList: List[String] = parseSuiteArgsIntoNameStrings(testNGArgsList, "-b")
    val chosenStyleSet: Set[String] = parseChosenStylesIntoChosenStyleSet(chosenStyles, "-y")
    spanScaleFactor = parseDoubleArgument(spanScaleFactors, "-F", 1.0)
    testSortingReporterTimeout = Span(parseDoubleArgument(testSortingReporterTimeouts, "-T", 2.0), Seconds)

    // If there's a graphic reporter, we need to leave it out of
    // reporterSpecs, because we want to pass all reporterSpecs except
    // the graphic reporter's to the RunnerJFrame (because RunnerJFrame *is*
    // the graphic reporter).
    val reporterConfigs: ReporterConfigurations =
      fullReporterConfigurations.graphicReporterConfiguration match {
        case None => fullReporterConfigurations
        case Some(grs) => {
          new ReporterConfigurations(
            None,
            fullReporterConfigurations.fileReporterConfigurationList,
            fullReporterConfigurations.junitXmlReporterConfigurationList,
            fullReporterConfigurations.dashboardReporterConfigurationList,
            fullReporterConfigurations.xmlReporterConfigurationList,
            fullReporterConfigurations.standardOutReporterConfiguration,
            fullReporterConfigurations.standardErrReporterConfiguration,
            fullReporterConfigurations.htmlReporterConfigurationList,
            fullReporterConfigurations.customReporterConfigurationList, 
            fullReporterConfigurations.socketReporterConfigurationList
          )
        }
      }

    val passFailReporter = if (runWithPassFailReporter) Some(new PassFailReporter) else None
    
    if (propertiesMap.isDefinedAt("org.scalatest.ChosenStyles"))
      throw new IllegalArgumentException("Property name 'org.scalatest.ChosenStyles' is used by ScalaTest, please choose other property name.")
    val configMap = 
      if (chosenStyleSet.isEmpty)
        propertiesMap
      else
        propertiesMap + ("org.scalatest.ChosenStyles" -> chosenStyleSet)

    fullReporterConfigurations.graphicReporterConfiguration match {
      case Some(GraphicReporterConfiguration(configSet)) => {
        val graphicEventsToPresent: Set[EventToPresent] = EventToPresent.allEventsToPresent filter
          (if (configSet.contains(FilterTestStarting)) {_ != PresentTestStarting} else etp => true) filter
          (if (configSet.contains(FilterTestSucceeded)) {_ != PresentTestSucceeded} else etp => true) filter
          (if (configSet.contains(FilterTestIgnored)) {_ != PresentTestIgnored} else etp => true) filter
          (if (configSet.contains(FilterTestPending)) {_ != PresentTestPending} else etp => true) filter
          (if (configSet.contains(FilterSuiteStarting)) {_ != PresentSuiteStarting} else etp => true) filter
          (if (configSet.contains(FilterSuiteCompleted)) {_ != PresentSuiteCompleted} else etp => true) filter
          (if (configSet.contains(FilterInfoProvided)) {_ != PresentInfoProvided} else etp => true) 

        val abq = new ArrayBlockingQueue[RunnerJFrame](1)
        usingEventDispatchThread {
          val rjf = new RunnerJFrame(
            graphicEventsToPresent,
            reporterConfigs,
            suitesList,
            junitsList,
            runpathList,
            tagsToInclude,
            tagsToExclude,
            configMap,
            concurrent,
            membersOnlyList,
            wildcardList,
            testNGList,
            passFailReporter,
            concurrentConfig,
            suffixes,
            chosenStyleSet
          )
          rjf.setLocation(RUNNER_JFRAME_START_X, RUNNER_JFRAME_START_Y)
          rjf.setVisible(true)
          rjf.prepUIForRunning()
          rjf.runFromGUI()
          abq.put(rjf)
        }
        // To get the Ant task to work, the main thread needs to block until
        // The GUI window exits.
        val rjf = abq.take()
        rjf.blockUntilWindowClosed()
      }
      case None => { // Run the test without a GUI
        withClassLoaderAndDispatchReporter(runpathList, reporterConfigs, None, passFailReporter) {
          (loader, dispatchReporter) => {
            doRunRunRunDaDoRunRun(
              dispatchReporter,
              suitesList,
              junitsList,
              Stopper.default,
              tagsToInclude,
              tagsToExclude,
              configMap,
              concurrent,
              membersOnlyList,
              wildcardList,
              testNGList,
              runpathList,
              loader,
              new RunDoneListener {},
              1,
              concurrentConfig,
              suffixes,
              chosenStyleSet
            )
          }
        }
      }
    }
    
    passFailReporter match {
      case Some(pfr) => pfr.allTestsPassed
      case None => false
    }
  }

  // Returns an Option[String]. Some is an error message. None means no error.
  private[scalatest] def checkArgsForValidity(args: Array[String]) = {

    val lb = new ListBuffer[String]
    val it = args.iterator.buffered
    while (it.hasNext) {
      val s = it.next
      if (
        s.startsWith("-p") ||
        s.startsWith("-R") ||
        s.startsWith("-f") ||
        s.startsWith("-u") ||
        s.startsWith("-d") ||
        s.startsWith("-a") ||
        s.startsWith("-r") ||
        s.startsWith("-C") ||
        s.startsWith("-n") ||
        /* s.startsWith("-x") || */
        s.startsWith("-l") ||
        s.startsWith("-s") ||
        s.startsWith("-i") ||
        s.startsWith("-j") ||
        s.startsWith("-m") ||
        s.startsWith("-w") ||
        s.startsWith("-b") ||
        s.startsWith("-y") ||
        s.startsWith("-t") ||
        s.startsWith("-z") ||
        s.startsWith("-q") ||
        s.startsWith("-Q") ||
        s.startsWith("-F") ||
        s.startsWith("-T")
      ) {
        if (it.hasNext)
          it.next
      }
      else if (s.startsWith("-k")) {
        if (it.hasNext)
          it.next
        if (it.hasNext)
          it.next
      }
      else if (s.startsWith("-h")) {
        if (it.hasNext)
          it.next
        if (it.hasNext && it.head == "-Y") {
          it.next
          it.next
        }
      }
      else if (!s.startsWith("-D") && !s.startsWith("-g") && !s.startsWith("-o") && !s.startsWith("-e") && !s.startsWith("-c") && !s.startsWith("-P")) {
        lb += s
      }
    }
    val argsList = lb.toList
    if (argsList.length != 0)
      Some("Unrecognized argument" + (if (argsList.isEmpty) ": " else "s: ") + argsList.mkString("", ", ", "."))
    else
      None
  }

  //
  // Examines concurrent option arg to see if it contains an optional numeric
  // value representing the number of threads to use, e.g. -P10 for 10 threads.
  // 
  // It also examines for the 'S' argument, e.g. -PS or -PS10, which when specified,
  // will enable the SuiteSortingReporter.
  //
  // It's possible for user to specify the -P option multiple times on the
  // command line, although it isn't particularly useful.  This method scans
  // through multiples until it finds one with a number appended and uses
  // that.  If none have a number it just returns 0.  If anyone of the -P comes 
  // with the 'S' option, SuiteSortingReporter will be enabled.
  //
  private[scalatest] def parseConcurrentConfig(concurrentList: List[String]): ConcurrentConfig = {
    val threadOpt = concurrentList.find(s => s.matches("-c\\d+") || s.matches("-cS\\d+"))
    val numThreads = threadOpt match {
      case Some(arg) => 
        if (arg.startsWith("-cS"))
          arg.substring(3).toInt
        else
          arg.substring(2).toInt
      case None      => 0
    }
    
    val enableSuiteSortingReporter = concurrentList.find(_.startsWith("-cS")).isDefined
    
    ConcurrentConfig(numThreads, enableSuiteSortingReporter)
  }

  //
  // Generates a Pattern based on suffixes passed in by user.  Pattern
  // matches class names that end with one of the specified suffixes.
  //
  private def genSuffixesPattern(suffixesList: List[String]): Option[Pattern] = {
    if (suffixesList.isEmpty)
      None
    else
      Some(Pattern.compile(".*(" + suffixesList.mkString("|") + ")$"))
  }

  private[scalatest] def parseArgs(args: Array[String]) = {

    val runpath = new ListBuffer[String]()
    val reporters = new ListBuffer[String]()
    val suites = new ListBuffer[String]()
    val junits = new ListBuffer[String]()
    val props = new ListBuffer[String]()
    val includes = new ListBuffer[String]()
    val excludes = new ListBuffer[String]()
    val concurrent = new ListBuffer[String]()
    val membersOnly = new ListBuffer[String]()
    val wildcard = new ListBuffer[String]()
    val testNGXMLFiles = new ListBuffer[String]()
    val suffixes = new ListBuffer[String]()
    val chosenStyles = new ListBuffer[String]()
    val spanScaleFactor = new ListBuffer[String]()
    val testSortingReporterTimeout = new ListBuffer[String]()

    val it = args.iterator.buffered
    while (it.hasNext) {

      val s = it.next

      if (s.startsWith("-D")) {
         props += s
      }
      else if (s.startsWith("-p")) {
        println("WARNING: -p has been deprecated and will be reused for a different (but still very cool) purpose in ScalaTest 2.0. Please change all uses of -p to -R.")
        runpath += s
        if (it.hasNext)
          runpath += it.next
      }
      else if (s.startsWith("-R")) {
        runpath += "-p" + s.substring(2)
        if (it.hasNext)
          runpath += it.next
      }
      else if (s.startsWith("-g")) {
        reporters += s
      }
      else if (s.startsWith("-o")) {
        reporters += s
      }
      else if (s.startsWith("-e")) {
        reporters += s
      }
      else if (s.startsWith("-f")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-u")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-d")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-a")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-x")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-h")) {
        reporters += s
        if (it.hasNext)
          reporters += it.next
        if (it.hasNext && it.head == "-Y") {
          reporters += it.next
          if (it.hasNext)
            reporters += it.next
        }
      }
      else if (s.startsWith("-n")) {
        includes += s
        if (it.hasNext)
          includes += it.next
      }
      else if (s.startsWith("-l")) {
        excludes += s
        if (it.hasNext)
          excludes += it.next
      }
      else if (s.startsWith("-r")) {
        println("WARNING: -r has been deprecated and will be reused for a different (but still very cool) purpose in ScalaTest 2.0. Please change all uses of -r to -C.")
        reporters += s
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-C")) {

        reporters += "-r" + s.substring(2)
        if (it.hasNext)
          reporters += it.next
      }
      else if (s.startsWith("-s")) {

        suites += s
        if (it.hasNext)
          suites += it.next
      }
      else if (s.startsWith("-i")) {
        
        suites += s
        if (it.hasNext)
          suites += it.next
      }
      else if (s.startsWith("-t")) {

        suites += s
        if (it.hasNext)
          suites += it.next
      }
      else if (s.startsWith("-z")) {

        suites += s
        if (it.hasNext)
          suites += it.next
      }
      else if (s.startsWith("-j")) {

        junits += s
        if (it.hasNext)
          junits += it.next
      }
      else if (s.startsWith("-m")) {

        membersOnly += s
        if (it.hasNext)
          membersOnly += it.next
      }
      else if (s.startsWith("-w")) {

        wildcard += s
        if (it.hasNext)
          wildcard += it.next
      }
      else if (s.startsWith("-c")) {
        println("WARNING: -c has been deprecated and will be reused for a different (but still very cool) purpose in ScalaTest 2.0. Please change all uses of -c to -P.")
        concurrent += s
      }
      else if (s.startsWith("-P")) {

        concurrent += "-c" + s.substring(2)
      }
      else if (s.startsWith("-b")) {

        testNGXMLFiles += s
        if (it.hasNext)
          testNGXMLFiles += it.next
      }
      else if (s.startsWith("-q")) {
        if (it.hasNext)
          suffixes += it.next()
      }
      else if (s.startsWith("-Q")) {
        suffixes += "Spec|Suite"
      }
      else if (s.startsWith("-k")) {
        
        reporters += s
        if (it.hasNext && !it.head.startsWith("-")) // for host
          reporters += it.next
        if (it.hasNext && !it.head.startsWith("-")) // for port
          reporters += it.next
      }
      else if (s.startsWith("-y")) {
        chosenStyles += s
        if (it.hasNext)
          chosenStyles += it.next()
      }
      else if (s.startsWith("-F")) {
        spanScaleFactor += s
        if (it.hasNext)
          spanScaleFactor += it.next()
      }
      else if (s.startsWith("-T")) {

        testSortingReporterTimeout += s
        if (it.hasNext)
          testSortingReporterTimeout += it.next
      }
      else {
        throw new IllegalArgumentException("Unrecognized argument: " + s)
      }
    }

    (
      runpath.toList,
      reporters.toList,
      suites.toList,
      junits.toList,
      props.toList,
      includes.toList,
      excludes.toList,
      concurrent.toList,
      membersOnly.toList,
      wildcard.toList,
      testNGXMLFiles.toList,
      genSuffixesPattern(suffixes.toList), 
      chosenStyles.toList, 
      spanScaleFactor.toList, 
      testSortingReporterTimeout.toList
    )
  }


  /**
   * Returns a possibly empty ConfigSet containing configuration
   * objects specified in the passed reporterArg. Configuration
   * options are specified immediately following
   * the reporter option, as in:
   *
   * -oFA
   *
   * If no configuration options are specified, this method returns an
   * empty ConfigSet. This method never returns null.
   */
  private def parseConfigSet(reporterArg: String): Set[ReporterConfigParam] = {

    if (reporterArg == null)
      throw new NullPointerException("reporterArg was null")

    if (reporterArg.length < 2)
      throw new IllegalArgumentException("reporterArg < 2")

    // The reporterArg passed includes the initial -, as in "-oFI",
    // so the first config param will be at index 2
    val configString = reporterArg.substring(2)
    val it = configString.iterator
    var set = Set[ReporterConfigParam]()
    while (it.hasNext) 
      it.next match {
        case 'Y' =>  throw new IllegalArgumentException("Use of Y was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'Z' => throw new IllegalArgumentException("Use of Z was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'T' => // Use for Dots
        case 'P' =>throw new IllegalArgumentException("Use of P was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'B' =>throw new IllegalArgumentException("Use of B was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'I' =>throw new IllegalArgumentException("Use of I was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        // case 'S' => // Use for Short Stack Traces
        case 'A' =>throw new IllegalArgumentException("Use of A was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'R' =>throw new IllegalArgumentException("Use of R was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'G' =>throw new IllegalArgumentException("Use of G was deprecated in ScalaTest 1.0 and removed in 1.5. Please check the Scaladoc documentation of org.scalatest.Runner for information on valid Reporter config parameters.")
        case 'N' => set += FilterTestStarting
        case 'C' => set += FilterTestSucceeded
        case 'X' => set += FilterTestIgnored
        case 'E' => set += FilterTestPending
        case 'H' => set += FilterSuiteStarting
        case 'L' => set += FilterSuiteCompleted
        case 'O' => set += FilterInfoProvided
        case 'W' => set += PresentWithoutColor
        case 'F' => set += PresentFullStackTraces
        case 'S' => set += PresentShortStackTraces
        case 'D' => set += PresentAllDurations
        case 'U' => set += PresentUnformatted
        case c: Char => { 

          // this should be moved to the checker, and just throw an exception here with a debug message. Or allow a MatchError.
          val msg1 = Resources("invalidConfigOption", String.valueOf(c)) + '\n'
          val msg2 =  Resources("probarg", reporterArg) + '\n'

          throw new IllegalArgumentException(msg1 + msg2)
        }
      }
    set
  }

  private[scalatest] def parseReporterArgsIntoConfigurations(args: List[String]) = {
    //
    // Checks to see if any args are smaller than two characters in length.
    // Allows a one-character arg if it's a directory-name parameter, to
    // permit use of "." for example.  Allows a one-character arg if it's
    // a number.
    //
    def argTooShort(args: List[String]): Boolean = {
      args match {
        case Nil => false

        case "-u" :: directory :: list => argTooShort(list)
        case "-d" :: directory :: list => argTooShort(list)
        case "-a" :: number    :: list => argTooShort(list)
        case "-x" :: directory :: list => argTooShort(list)

        case x :: list =>
          if (x.length < 2) true
          else              argTooShort(list)
      }
    }

    if (args == null)
      throw new NullPointerException("args was null")

    if (args.exists(_ == null))
      throw new NullPointerException("an arg String was null")

    if (argTooShort(args)) // TODO: check and print out a user friendly message for this
      throw new IllegalArgumentException("an arg String was less than 2 in length: " + args)

    for (dashX <- List("-g", "-o", "-e")) {
      if (args.toList.count(_.startsWith(dashX)) > 1) // TODO: also check and print a user friendly message for this
        throw new IllegalArgumentException("Only one " + dashX + " allowed")
    }

    // TODO: also check and print a user friendly message for this
    // again here, i had to skip some things, so I had to use an iterator.
    val it = args.iterator.buffered
    while (it.hasNext)
      it.next.take(2).toString match {
        case "-g" =>
        case "-o" =>
        case "-e" =>
        case "-f" =>
          if (it.hasNext)
            it.next // scroll past the filename
          else
            throw new IllegalArgumentException("-f needs to be followed by a file name arg: ")
        case "-u" =>
          if (it.hasNext) {
            val directoryName = it.next
            val directory = new File(directoryName)
            if (!directory.isDirectory) {
              try {
                directory.mkdirs()
                if (!directory.exists)
                  throw new IllegalArgumentException("Unable to create directory: " + directory.getAbsolutePath)
              }
              catch {
                case se: SecurityException => 
                  throw new IllegalArgumentException("Unable to create directory: " + directory.getAbsolutePath)
              }
            }
            else if (directory.isFile)
              throw new IllegalArgumentException(directory.getAbsolutePath + " is a file, directory expected.")
          }
          else {
            throw new IllegalArgumentException("-u needs to be followed by a directory name arg: ")
          }
        case "-d" =>
          if (it.hasNext) {
            val directory = it.next
            if (!(new File(directory).isDirectory))
              throw new IllegalArgumentException(
                "arg for -d option is not a directory [" + directory + "]")
            else {}
          }
          else {
            throw new IllegalArgumentException("-d needs to be followed by a directory name arg: ")
          }
        case "-a" =>
          if (it.hasNext) {
            def isValidInt(text: String): Boolean = 
              try { text.toInt; true } catch { case _: Throwable => false }

            val number = it.next
            if (!(isValidInt(number)))
              throw new IllegalArgumentException(
                "arg for -a option is not a number [" + number + "]")
            else {}
          }
          else {
            throw new IllegalArgumentException("-a needs to be followed by a number arg: ")
          }
        case "-x" =>
          if (it.hasNext) {
            val directory = it.next
            if (!(new File(directory).isDirectory))
              throw new IllegalArgumentException(
                "arg for -x option is not a directory [" + directory + "]")
            else {}
          }
          else {
            throw new IllegalArgumentException("-x needs to be followed by a directory name arg: ")
          }
        case "-h" =>
          if (it.hasNext) {
            it.next // scroll past the filename
            if (it.hasNext && it.head == "-Y") {
              it.next // scroll past the -Y
              if (it.hasNext)
                it.next // scroll past the css file name
              else
                throw new IllegalArgumentException("-Y needs to be followed by a file name arg: ")
            }
          }
          else
            throw new IllegalArgumentException("-h needs to be followed by a directory name arg: ")
        case "-r" =>
          if (it.hasNext)
            it.next // scroll past the reporter class
          else
            throw new IllegalArgumentException("-r needs to be followed by a reporter class name arg: ")
        case "-k" =>
          if (it.hasNext && !it.head.startsWith("-")) {
            it.next
            if (it.hasNext && !it.head.startsWith("-")) {
              try { 
                it.next.toInt 
              } 
              catch { 
                case _: Throwable => 
                  throw new IllegalArgumentException("port number must be an integer")
              }
            }
            else
              throw new IllegalArgumentException("-k needs to be followed by a host name and port number" )
          }
          else
            throw new IllegalArgumentException("-k needs to be followed by a host name and port number" )
        case "-C" =>
          if (it.hasNext)
            it.next // scroll past the reporter class
          else
            throw new IllegalArgumentException("-C needs to be followed by a reporter class name arg: ")
        case arg: String =>
          throw new IllegalArgumentException("An arg started with an invalid character string: " + arg)
      }

    val graphicReporterConfigurationOption =
      args.find(arg => arg.startsWith("-g")) match {
        case Some(dashGString) =>
          val configSet = parseConfigSet(dashGString)
          if (configSet.contains(PresentShortStackTraces))
            throw new IllegalArgumentException("Cannot specify an S (present short stack traces) configuration parameter for the graphic reporter (because it shows them anyway): " + dashGString)
          if (configSet.contains(PresentFullStackTraces))
            throw new IllegalArgumentException("Cannot specify an F (present full stack traces) configuration parameter for the graphic reporter (because it shows them anyway): " + dashGString)
          if (configSet.contains(PresentWithoutColor))
            throw new IllegalArgumentException("Cannot specify a W (present without color) configuration parameter for the graphic reporter: " + dashGString)
          if (configSet.contains(PresentAllDurations))
            throw new IllegalArgumentException("Cannot specify a D (present all durations) configuration parameter for the graphic reporter (because it shows them all anyway): " + dashGString)
          if (configSet.contains(PresentUnformatted))
            throw new IllegalArgumentException("Cannot specify a U (present unformatted) configuration parameter for the graphic reporter: " + dashGString)
          Some(new GraphicReporterConfiguration(configSet))
        case None => None
      }

    def buildFileReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[FileReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-f"))
          lb += new FileReporterConfiguration(parseConfigSet(arg), it.next)
      }
      lb.toList
    }
    val fileReporterConfigurationList = buildFileReporterConfigurationList(args)

    def buildJunitXmlReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[JunitXmlReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-u"))
          lb += new JunitXmlReporterConfiguration(Set[ReporterConfigParam](),
                                                  it.next)
      }
      lb.toList
    }
    val junitXmlReporterConfigurationList =
      buildJunitXmlReporterConfigurationList(args)

    def buildDashboardReporterConfigurationList(args: List[String]) = {
      def fetchNumFilesArg: Int = {
        var numFiles: Option[Int] = None
        val it = args.iterator

        while (!numFiles.isDefined && it.hasNext) {
          val arg = it.next
          if (arg.startsWith("-a"))
            numFiles = Some(it.next.toInt)
        }
        numFiles.getOrElse(DefaultNumFilesToArchive)
      }

      val numFilesToArchive = fetchNumFilesArg
      val it = args.iterator
      val lb = new ListBuffer[DashboardReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-d"))
          lb += new DashboardReporterConfiguration(Set[ReporterConfigParam](),
                                                   it.next, numFilesToArchive)
      }
      lb.toList
    }
    val dashboardReporterConfigurationList =
      buildDashboardReporterConfigurationList(args)

    def buildXmlReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[XmlReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-x"))
          lb += new XmlReporterConfiguration(Set[ReporterConfigParam](),
                                             it.next)
      }
      lb.toList
    }
    val xmlReporterConfigurationList = buildXmlReporterConfigurationList(args)

    def buildHtmlReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[HtmlReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-h") && it.hasNext) {
          if (it.hasNext) {
            val configSet = parseConfigSet(arg)
            val directory = it.next
            val cssFile = 
              if (it.hasNext && it.next == "-Y") {
                if (it.hasNext)
                  Some(new File(it.next).toURI.toURL)
                else
                  throw new IllegalArgumentException("-Y cannot be last, expected CSS file name to follow.")
              }
              else
                None
            lb += new HtmlReporterConfiguration(configSet, directory, cssFile)
          }
          else
            throw new IllegalArgumentException("-h cannot be last, expected HTML output directory name to follow.")
        }
      }
      lb.toList
    }
    val htmlReporterConfigurationList = buildHtmlReporterConfigurationList(args)

    val standardOutReporterConfigurationOption =
      args.find(arg => arg.startsWith("-o")) match {
        case Some(dashOString) => Some(new StandardOutReporterConfiguration(parseConfigSet(dashOString)))
        case None => None
      }

    val standardErrReporterConfigurationOption =
      args.find(arg => arg.startsWith("-e")) match {
        case Some(dashEString) => Some(new StandardErrReporterConfiguration(parseConfigSet(dashEString)))
        case None => None
      }

    def buildCustomReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[CustomReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-r")) {
          val dashRString = arg
          val customReporterClassName = it.next
          val configSet = parseConfigSet(dashRString)
          if (configSet.contains(PresentShortStackTraces))
            throw new IllegalArgumentException("Cannot specify an S (present short stack traces) configuration parameter for a custom reporter: " + dashRString + " " + customReporterClassName)
          if (configSet.contains(PresentFullStackTraces))
            throw new IllegalArgumentException("Cannot specify an F (present full stack traces) configuration parameter for a custom reporter: " + dashRString + " " + customReporterClassName)
          if (configSet.contains(PresentWithoutColor))
            throw new IllegalArgumentException("Cannot specify a W (without color) configuration parameter for a custom reporter: " + dashRString + " " + customReporterClassName)
          if (configSet.contains(PresentAllDurations))
            throw new IllegalArgumentException("Cannot specify a D (present all durations) configuration parameter for a custom reporter: " + dashRString + " " + customReporterClassName)
          if (configSet.contains(PresentUnformatted))
            throw new IllegalArgumentException("Cannot specify a U (present unformatted) configuration parameter for a custom reporter: " + dashRString + " " + customReporterClassName)
          lb += new CustomReporterConfiguration(configSet, customReporterClassName)
        }
      }
      lb.toList
    }
    val customReporterConfigurationList = buildCustomReporterConfigurationList(args)

    def buildSocketReporterConfigurationList(args: List[String]) = {
      val it = args.iterator
      val lb = new ListBuffer[SocketReporterConfiguration]
      while (it.hasNext) {
        val arg = it.next
        if (arg.startsWith("-k")) {
          if (!it.hasNext)
            throw new IllegalArgumentException("-k must be followed by host and port")
          val host = it.next
          if (!it.hasNext)
            throw new IllegalArgumentException("-k must be followed by host and port")
          val port = it.next.toInt
          lb += new SocketReporterConfiguration(host, port)
        }
      }
      lb.toList
    }
    val socketReporterConfigurationList = buildSocketReporterConfigurationList(args)
    
    // Here instead of one loop, i go through the loop several times.
    new ReporterConfigurations(
      graphicReporterConfigurationOption,
      fileReporterConfigurationList,
      junitXmlReporterConfigurationList,
      dashboardReporterConfigurationList,
      xmlReporterConfigurationList,
      standardOutReporterConfigurationOption,
      standardErrReporterConfigurationOption,
      htmlReporterConfigurationList,
      customReporterConfigurationList, 
      socketReporterConfigurationList
    )
  }

  // Used to parse -s, -j, -m, and -w args, one of which will be passed as a String as dashArg
  private[scalatest] def parseSuiteArgsIntoNameStrings(args: List[String], dashArg: String) = {

    if (args == null)
      throw new NullPointerException("args was null")

    if (args.exists(_ == null))
      throw new NullPointerException("an arg String was null")

    if (dashArg != "-j" && dashArg != "-s" && dashArg != "-w" && dashArg != "-m" && dashArg != "-b")
      throw new IllegalArgumentException("dashArg invalid: " + dashArg)
/*
<<<<<<< .working TODOCS: Is the above the correct way to merge these?
    if (dashArg != "-j" && dashArg != "-w" && dashArg != "-m" && dashArg != "-b")
      throw new IllegalArgumentException("dashArg invalid: " + dashArg)
=======
    if (dashArg != "-j" && dashArg != "-s" && dashArg != "-w" && dashArg != "-m" && dashArg != "-b")
      throw new NullPointerException("dashArg invalid: " + dashArg)
>>>>>>> .merge-right.r3653
*/

    val lb = new ListBuffer[String]
    val it = args.iterator
    while (it.hasNext) {
      val dashS = it.next
      if (dashS != dashArg)
        throw new IllegalArgumentException("Every other element, starting with the first, must be " + dashArg)
      if (it.hasNext) {
        val suiteName = it.next
        if (!suiteName.startsWith("-"))
          lb += suiteName
        else
          throw new IllegalArgumentException("Expecting a Suite class name or package name to follow " + dashArg + ", but got: " + suiteName)
      }
      else
        throw new IllegalArgumentException("Last element must be a Suite class name or package name, not a " + dashArg + ".")
    }
    lb.toList
  }
  
  private[scalatest] def parseSuiteArgsIntoSuiteParam(args: List[String], dashArg: String) = {
    if (args == null)
      throw new NullPointerException("args was null")

    if (args.exists(_ == null))
      throw new NullPointerException("an arg String was null")
    
    if (dashArg != "-s")
      throw new IllegalArgumentException("dashArg invalid: " + dashArg)
    
    val lb = new ListBuffer[SuiteParam]
    val it = args.iterator.buffered
    while (it.hasNext) {
      val dashS = it.next
      if (dashS != dashArg)
        throw new IllegalArgumentException("Starting element must be " + dashArg)
      if (it.hasNext) {
        val className = it.next
        if (!className.startsWith("-")) {
          
          val (testNames, wildcardTestNames) = 
            if (it.hasNext && (it.head == "-t" || it.head == "-z")) {            
              val testNamesBuffer = new ListBuffer[String]()
              val wildcardTestNamesBuffer = new ListBuffer[String]()
              while (it.hasNext && (it.head == "-t" || it.head == "-z")) {
                val dashTest = it.next
                if (dashTest == "-t")
                  testNamesBuffer += it.next
                else
                  wildcardTestNamesBuffer += it.next
              }
              (testNamesBuffer.toArray, wildcardTestNamesBuffer.toArray)
            }
            else
              (Array.empty[String], Array.empty[String])
          
          val nestedSuites = 
            if (it.hasNext && it.head == "-i") {
              val nestedLb = new ListBuffer[NestedSuiteParam]()
            
              while (it.hasNext && it.head == "-i") {
                val dashI = it.next()
                val suiteId = it.next
                val suiteIdTestNamesBuffer = new ListBuffer[String]()
                val suiteIdWildcardTestNamesBuffer = new ListBuffer[String]()
              
                while (it.hasNext && (it.head == "-t" || it.head == "-z")) {
                  val dashTest = it.next
                  if (dashTest == "-t")
                    suiteIdTestNamesBuffer += it.next
                  else
                    suiteIdWildcardTestNamesBuffer += it.next
                }
                nestedLb += new NestedSuiteParam(suiteId, suiteIdTestNamesBuffer.toArray, suiteIdWildcardTestNamesBuffer.toArray)
              }
              nestedLb.toArray
            }
            else
              Array.empty[NestedSuiteParam]
          
          lb += SuiteParam(className, testNames, wildcardTestNames, nestedSuites)
        }
        else
          throw new IllegalArgumentException("Expecting a Suite class name to follow " + dashArg + ", but got: " + className)
      }
      else
        throw new IllegalArgumentException("Last element must be a Suite class name, not a " + dashArg + ".")
    }
    lb.toList
  }

  private[scalatest] def parseCompoundArgIntoSet(args: List[String], expectedDashArg: String): Set[String] = 
      Set() ++ parseCompoundArgIntoList(args, expectedDashArg)

  private[scalatest] def parseRunpathArgIntoList(args: List[String]): List[String] = parseCompoundArgIntoList(args, "-p")

  private[scalatest] def parseCompoundArgIntoList(args: List[String], expectedDashArg: String): List[String] = {

    if (args == null)
      throw new NullPointerException("args was null")

    if (args.exists(_ == null))
      throw new NullPointerException("an arg String was null")

    if (args.length == 0) {
      List()
    }
    else if (args.length == 2) {
      val dashArg = args(0)
      val runpathArg = args(1)

      if (dashArg != expectedDashArg)
        throw new IllegalArgumentException("First arg must be " + expectedDashArg + ", but was: " + dashArg)

      if (runpathArg.trim.isEmpty)
        throw new IllegalArgumentException("The runpath string must actually include some non-whitespace characters.")

      splitPath(runpathArg)
    }
    else {
      throw new IllegalArgumentException("Runpath must be either zero or two args: " + args)
    }
  }
  
  private[scalatest] def parseChosenStylesIntoChosenStyleSet(args: List[String], dashArg: String) = {
    val it = args.iterator
    val lb = new ListBuffer[String]()
    while (it.hasNext) {
      val dash = it.next
      if (dash != dashArg) 
        throw new IllegalArgumentException("Every other element, starting with the first, must be " + dashArg)
      if (it.hasNext) {
        lb += it.next
      }
      else
        throw new IllegalArgumentException("Last element must be a style name, not a " + dashArg + ".")
    }
    lb.toSet
  }
  
  private[scalatest] def parseDoubleArgument(args: List[String], dashArg: String, defaultValue: Double): Double = {
    val it = args.iterator
    val lb = new ListBuffer[Double]()
    while (it.hasNext) {
      val dash = it.next
      if (dash != dashArg)
        throw new IllegalArgumentException("Every other element, starting with the first, must be " + dashArg)
      if (it.hasNext) {
        val spanString = it.next
        try {
          lb += spanString.toDouble
        }
        catch {
          case e: NumberFormatException =>
            throw new IllegalArgumentException(dashArg + " must be followed by a number, but '" + spanString + "' is not a number.")
        }
      }
      else
        throw new IllegalArgumentException("Last element must be a number, not a " + dashArg + ".")
    }
    if (lb.size == 0)
      defaultValue
    else if (lb.size == 1)
      lb(0)
    else
      throw new IllegalArgumentException("Only one " + dashArg + " can be specified.")
  }

  //
  // Splits a space-delimited path into its component parts.
  //
  // Spaces within path elements may be escaped with backslashes, e.g.
  // "c:\Documents\ And\ Settings c:\Program\ Files"
  //
  // See comments for isCompleteToken() below for exceptions.
  //
  private val START_TOKEN_PATTERN = Pattern.compile("""^\s*(.*?)(\s|$)""")
  private val FULL_TOKEN_PATTERN  = Pattern.compile("""^\s*(.+?)(((?<=[^\\])\s)|$)""")
  private def splitPath(pathArg: String): List[String] = {
    val path = pathArg.trim

    if (path.isEmpty) Nil
    else {
      val startMatcher = START_TOKEN_PATTERN.matcher(path)

      if (!startMatcher.find())
        throw new RuntimeException("unexpected startMatcher path [" +
                                   path + "]")
      val token = startMatcher.group(1)

      if (isCompleteToken(token)) {
        token :: splitPath(path.substring(startMatcher.end))
      }
      else {
        val fullMatcher = FULL_TOKEN_PATTERN.matcher(path)

        if (!fullMatcher.find())
          throw new RuntimeException("unexpected fullMatcher path [" +
                                     path + "]")
        val fullToken = fullMatcher.group(1).replaceAll("""\\(\s)""", "$1")

        fullToken :: splitPath(path.substring(fullMatcher.end))
      }
    }
  }

  //
  // Determines whether specified token is complete or partial.
  //
  // Tokens are considered partial if they end with a backslash, since
  // backslash is used to escape spaces that would otherwise be
  // treated as delimiters within the path string.
  //
  // Exceptions are cases where the token ends in a backslash
  // but is still considered a complete token because it constitutes
  // a valid representation of a root directory on a windows system,
  // e.g. "c:\" or just "\".
  //
  private val ROOT_DIR_PATTERN = Pattern.compile("""(?i)\\|[a-z]:\\""")
  private def isCompleteToken(token: String): Boolean = {
    val matcher = ROOT_DIR_PATTERN.matcher(token)

    matcher.matches() || (token(token.length - 1) != '\\')
  }

  private[scalatest] def parsePropertiesArgsIntoMap(args: List[String]) = {

    if (args == null)
      throw new NullPointerException("args was null")

    if (args.exists(_ == null))
      throw new NullPointerException("an arg String was null")

    if (args.exists(_.indexOf('=') == -1))
      throw new IllegalArgumentException("A -D arg does not contain an equals sign.")

    if (args.exists(!_.startsWith("-D")))
      throw new IllegalArgumentException("A spice arg does not start with -D.")

    if (args.exists(_.indexOf('=') == 2))
      throw new IllegalArgumentException("A spice arg does not have a key to the left of the equals sign.")

    if (args.exists(arg => arg.indexOf('=') == arg.length - 1))
      throw new IllegalArgumentException("A spice arg does not have a value to the right of the equals sign.")

    val tuples = for (arg <- args) yield {
      val keyValue = arg.substring(2) // Cut off the -D at the beginning
      val equalsPos = keyValue.indexOf('=')
      val key = keyValue.substring(0, equalsPos)
      val value = keyValue.substring(equalsPos + 1)
      (key, value)
    }

    scala.collection.immutable.Map() ++ tuples
  }

  // For debugging.
/*
  private[scalatest] def printOpts(opt: EventToPresent.Set32) {
    if (opt.contains(EventToPresent.PresentRunStarting))
      println("PresentRunStarting")
    if (opt.contains(EventToPresent.PresentTestStarting))
      println("PresentTestStarting")
    if (opt.contains(EventToPresent.PresentTestSucceeded))
      println("PresentTestSucceeded")
    if (opt.contains(EventToPresent.PresentTestFailed))
      println("PresentTestFailed")
    if (opt.contains(EventToPresent.PresentTestIgnored))
      println("PresentTestIgnored")
    if (opt.contains(EventToPresent.PresentSuiteStarting))
      println("PresentSuiteStarting")
    if (opt.contains(EventToPresent.PresentSuiteCompleted))
      println("PresentSuiteCompleted")
    if (opt.contains(EventToPresent.PresentSuiteAborted))
      println("PresentSuiteAborted")
    if (opt.contains(EventToPresent.PresentInfoProvided))
      println("PresentInfoProvided")
    if (opt.contains(EventToPresent.PresentRunStopped))
      println("PresentRunStopped")
    if (opt.contains(EventToPresent.PresentRunCompleted))
      println("PresentRunCompleted")
    if (opt.contains(EventToPresent.PresentRunAborted))
      println("PresentRunAborted")
  }
*/

  private[scalatest] def mergeMap[A, B](ms: List[Map[A, B]])(f: (B, B) => B): Map[A, B] =
    (Map[A, B]() /: (for (m <- ms; kv <- m) yield kv)) { (a, kv) =>
    a + (if (a.contains(kv._1)) kv._1 -> f(a(kv._1), kv._2) else kv)
  }
  
  private[scalatest] def doRunRunRunDaDoRunRun(
    dispatch: DispatchReporter,
    suitesList: List[SuiteParam],
    junitsList: List[String],
    stopRequested: Stopper,
    tagsToIncludeSet: Set[String], 
    tagsToExcludeSet: Set[String], 
    configMap: Map[String, Object],
    concurrent: Boolean,
    membersOnlyList: List[String],
    wildcardList: List[String],
    testNGList: List[String],
    runpath: List[String],
    loader: ClassLoader,
    doneListener: RunDoneListener,
    runStamp: Int,
    concurrentConfig: ConcurrentConfig,
    suffixes: Option[Pattern],
    chosenStyleSet: Set[String]
  ) = { // TODO, either put a type on here or do procedure style if Unit

    // TODO: add more, and to RunnerThread too
    if (dispatch == null)
      throw new NullPointerException
    if (suitesList == null)
      throw new NullPointerException
    if (junitsList == null)
      throw new NullPointerException
    if (stopRequested == null)
      throw new NullPointerException
    if (tagsToIncludeSet == null)
      throw new NullPointerException
    if (tagsToExcludeSet == null)
      throw new NullPointerException
    if (configMap == null)
      throw new NullPointerException
    if (membersOnlyList == null)
      throw new NullPointerException
    if (wildcardList == null)
      throw new NullPointerException
    if (testNGList == null)
      throw new NullPointerException
    if (runpath == null)
      throw new NullPointerException
    if (loader == null)
      throw new NullPointerException
    if (doneListener == null)
      throw new NullPointerException
    if (chosenStyleSet == null)
      throw new NullPointerException

    var tracker = new Tracker(new Ordinal(runStamp))

    val runStartTime = System.currentTimeMillis
    
    try {
      val loadProblemsExist =
        try {
          val unrunnableList = suitesList.filter{ suiteParam => 
            val className = suiteParam.className
            loader.loadClass(className) // Check if the class exist, so if not we get the nice cannot load suite error message.
            !isAccessibleSuite(className, loader) && !isRunnable(className, loader)
          }
          if (!unrunnableList.isEmpty) {
            val names = for (suiteParam <- unrunnableList) yield " " + suiteParam.className
            dispatch(RunAborted(tracker.nextOrdinal(), Resources("nonSuite") + names.mkString(", "), None))
            true
          }
          else {
            false
          }
        }
        catch {
          case e: ClassNotFoundException => {
            dispatch(RunAborted(tracker.nextOrdinal(), Resources("cannotLoadSuite", e.getMessage), Some(e)))
            true
          }
        }

      if (!loadProblemsExist) {

        case class SuiteConfig(suite: Suite, dynaTags: DynaTags, requireSelectedTag: Boolean, excludeNestedSuites: Boolean)

        try {
          val namedSuiteInstances: List[SuiteConfig] =
            for (suiteParam <- suitesList)
              yield {
                val suiteClassName = suiteParam.className
                val clazz = loader.loadClass(suiteClassName)
                val wrapWithAnnotation = clazz.getAnnotation(classOf[WrapWith])
                val suiteInstance = 
                if (wrapWithAnnotation == null) 
                  clazz.newInstance.asInstanceOf[Suite]
                else {
                  val suiteClazz = wrapWithAnnotation.value
                  val constructorList = suiteClazz.getDeclaredConstructors()
                  val constructor = constructorList.find { c => 
                    val types = c.getParameterTypes
                    types.length == 1 && types(0) == classOf[java.lang.Class[_]]
                  }
                  constructor.get.newInstance(clazz).asInstanceOf[Suite]
                }
                
                if (suiteParam.testNames.length == 0 && suiteParam.wildcardTestNames.length == 0 && suiteParam.nestedSuites.length == 0)
                  SuiteConfig(suiteInstance, new DynaTags(Map.empty, Map.empty), false, false) // -s suiteClass, no dynamic tagging required.
                else {
                  val nestedSuites = suiteParam.nestedSuites
                  
                  val (selectSuiteList, selectTestList) = nestedSuites.partition(ns => ns.testNames.length == 0 || ns.wildcardTestNames.length == 0)
                  val suiteDynaTags: Map[String, Set[String]] = Map() ++ selectSuiteList.map(ns => (ns.suiteId -> Set(SELECTED_TAG)))
                  
                  val suiteExactTestDynaTags: Map[String, Map[String, Set[String]]] = 
                    if (suiteParam.testNames.length > 0) 
                      Map(suiteInstance.suiteId -> (Map() ++ suiteParam.testNames.map(tn => (tn -> Set(SELECTED_TAG)))))
                    else 
                      Map.empty
                  
                  val suiteWildcardTestDynaTags: Map[String, Map[String, Set[String]]] = 
                    if (suiteParam.wildcardTestNames.length > 0) {
                      val wildcardTestNames = suiteParam.wildcardTestNames
                      val allTestNames = suiteInstance.testNames
                      val wildcardTestTags = Map() ++ allTestNames.filter(tn => wildcardTestNames.find(wc => tn.contains(wc)).isDefined)
                                                          .map(tn => (tn -> Set(SELECTED_TAG)))
                      Map(suiteInstance.suiteId -> wildcardTestTags)
                    }
                    else
                      Map.empty
                      
                  def getNestedSuiteSelectedTestNames(nestedSuite: NestedSuiteParam): Array[String] = {
                    if (nestedSuite.wildcardTestNames.length == 0)
                      nestedSuite.testNames
                    else {
                      val wildcardTestNames = nestedSuite.wildcardTestNames
                      val allTestNames = suiteInstance.testNames
                      nestedSuite.testNames ++ allTestNames.filter(tn => wildcardTestNames.find(wc => tn.contains(wc)).isDefined)
                    }
                  }
                  
                  val nestedSuitesTestDynaTags: Map[String, Map[String, Set[String]]] 
                    = Map() ++ selectTestList.map(ns => (ns.suiteId -> (Map() ++ getNestedSuiteSelectedTestNames(ns).map(tn => (tn, Set(SELECTED_TAG))))))
                    
                  val testDynaTags = mergeMap[String, Map[String, Set[String]]](List(suiteExactTestDynaTags, suiteWildcardTestDynaTags, nestedSuitesTestDynaTags)) { (suiteTestMap1, suiteTestMap2) => 
                                       mergeMap[String, Set[String]](List(suiteTestMap1, suiteTestMap2)) { (tagSet1, tagSet2) =>
                                         tagSet1 ++ tagSet2
                                       }
                                     }
                  // Only exclude nested suites when using -s XXX -t XXXX, or -s XXX -z XXX
                  val excludeNestedSuites = suiteParam.testNames.length > 0 && nestedSuites.length == 0
                  SuiteConfig(suiteInstance, new DynaTags(suiteDynaTags, testDynaTags), true, excludeNestedSuites)
                }
              }
          
          val requireSelectedTag = suitesList.find(suiteParam => suiteParam.testNames.length > 0)
          
          val emptyDynaTags = DynaTags(Map.empty[String, Set[String]], Map.empty[String, Map[String, Set[String]]])

          val junitSuiteInstances: List[SuiteConfig] =
            for (junitClassName <- junitsList)
              yield SuiteConfig(new JUnitWrapperSuite(junitClassName, loader), emptyDynaTags, false, true) // JUnit suite should exclude nested suites

          val testNGWrapperSuiteList: List[SuiteConfig] =
            if (!testNGList.isEmpty)
              List(SuiteConfig(new TestNGWrapperSuite(testNGList), emptyDynaTags, false, true)) // TestNG suite should exclude nested suites
            else
              Nil

          val (membersOnlySuiteInstances, wildcardSuiteInstances) = {

            val membersOnlyAndWildcardListsAreEmpty = membersOnlyList.isEmpty && wildcardList.isEmpty // They didn't specify any -m's or -w's on the command line

            if (membersOnlyAndWildcardListsAreEmpty && (!suitesList.isEmpty || !junitsList.isEmpty || !testNGList.isEmpty)) {
              (Nil, Nil) // No DiscoverySuites in this case. Just run Suites named with -s or -j or -b
            }
            else {
              println("DEBUG: Discovery Starting")
              // dispatch(DiscoveryStarting)
              val discoveryStartTime = System.currentTimeMillis
              val accessibleSuites = discoverSuiteNames(runpath, loader, suffixes)
              val discoveryDuration = System.currentTimeMillis - discoveryStartTime
              println("DEBUG: Discovery Completed: " + discoveryDuration + " milliseconds")
              // dispatch(DiscoveryCompleted(discoveryDuration))

              if (membersOnlyAndWildcardListsAreEmpty && suitesList.isEmpty && junitsList.isEmpty && testNGList.isEmpty) {
                // In this case, they didn't specify any -w, -m, -s, -j or -b on the command line, so the default
                // is to run any accessible Suites discovered on the runpath
                (Nil, List(SuiteConfig(new DiscoverySuite("", accessibleSuites, true, loader), emptyDynaTags, false, false)))
              }
              else {
                val membersOnlyInstances =
                  for (membersOnlyName <- membersOnlyList)
                    yield SuiteConfig(new DiscoverySuite(membersOnlyName, accessibleSuites, false, loader), emptyDynaTags, false, false)

                val wildcardInstances =
                  for (wildcardName <- wildcardList)
                    yield SuiteConfig(new DiscoverySuite(wildcardName, accessibleSuites, true, loader), emptyDynaTags, false, false)

                (membersOnlyInstances, wildcardInstances)
              }
            }
          }

          val suiteInstances: List[SuiteConfig] = namedSuiteInstances ::: junitSuiteInstances ::: membersOnlySuiteInstances ::: wildcardSuiteInstances ::: testNGWrapperSuiteList

          val testCountList =
            for (suiteConfig <- suiteInstances)
              yield { 
              val tagsToInclude = if (suiteConfig.requireSelectedTag) tagsToIncludeSet ++ Set(SELECTED_TAG) else tagsToIncludeSet
              val filter = Filter(if (tagsToInclude.isEmpty) None else Some(tagsToInclude), tagsToExcludeSet, suiteConfig.excludeNestedSuites, suiteConfig.dynaTags)
              suiteConfig.suite.expectedTestCount(filter)
            }
  
          def sumInts(list: List[Int]): Int =
            list match {
              case Nil => 0
              case x :: xs => x + sumInts(xs)
            }

          val expectedTestCount = sumInts(testCountList)

          dispatch(RunStarting(tracker.nextOrdinal(), expectedTestCount, configMap))
          
          if (concurrent) {

            // Because some tests may do IO, will create a pool of 2 times the number of processors reported
            // by the Runtime's availableProcessors method.
            val poolSize =
              if (concurrentConfig.numThreads > 0) concurrentConfig.numThreads
              else Runtime.getRuntime.availableProcessors * 2

            val distributedSuiteSorter = 
              if (concurrentConfig.enableSuiteSortingReporter)
                Some(new SuiteSortingReporter(dispatch, Span(testSortingReporterTimeout.millisPart + 1000, Millis), System.err))
              else
                None
              
            val concurrentDispatch = 
              distributedSuiteSorter match {
                case Some(dss) => dss
                case None => dispatch
              }
                
            val execSvc: ExecutorService = Executors.newFixedThreadPool(poolSize)
            try {

              val distributor = new ConcurrentDistributor(Args(dispatch, stopRequested, Filter(if (tagsToIncludeSet.isEmpty) None else Some(tagsToIncludeSet), tagsToExcludeSet), configMap, None, tracker, chosenStyleSet), execSvc)
              if (System.getProperty("org.scalatest.tools.Runner.forever", "false") == "true") {

                while (true) {
                  for (suiteConfig <- suiteInstances) {
                    val tagsToInclude = if (suiteConfig.requireSelectedTag) tagsToIncludeSet ++ Set(SELECTED_TAG) else tagsToIncludeSet
                    val filter = Filter(if (tagsToInclude.isEmpty) None else Some(tagsToInclude), tagsToExcludeSet, suiteConfig.excludeNestedSuites, suiteConfig.dynaTags)
                    val runArgs = Args(concurrentDispatch, stopRequested, filter, configMap, Some(distributor), tracker.nextTracker, chosenStyleSet, false, None, distributedSuiteSorter)
                    distributor.apply(suiteConfig.suite, runArgs)
                  }
                  distributor.waitUntilDone()
                }
              }
              else {
                for (suiteConfig <- suiteInstances) {
                  val tagsToInclude = if (suiteConfig.requireSelectedTag) tagsToIncludeSet ++ Set(SELECTED_TAG) else tagsToIncludeSet
                  val filter = Filter(if (tagsToInclude.isEmpty) None else Some(tagsToInclude), tagsToExcludeSet, suiteConfig.excludeNestedSuites, suiteConfig.dynaTags)
                  val runArgs = Args(concurrentDispatch, stopRequested, filter, configMap, Some(distributor), tracker.nextTracker, chosenStyleSet, false, None, distributedSuiteSorter)
                  distributor.apply(suiteConfig.suite, runArgs)
                }
                distributor.waitUntilDone()
              }
            }
            finally {
              execSvc.shutdown()
            }
          }
          else {
            for (suiteConfig <- suiteInstances) {
              val tagsToInclude = if (suiteConfig.requireSelectedTag) tagsToIncludeSet ++ Set(SELECTED_TAG) else tagsToIncludeSet
              val filter = Filter(if (tagsToInclude.isEmpty) None else Some(tagsToInclude), tagsToExcludeSet, suiteConfig.excludeNestedSuites, suiteConfig.dynaTags)
              val runArgs = Args(dispatch, stopRequested, filter, configMap, None, tracker, chosenStyleSet)
              val status = new ScalaTestStatefulStatus()
              val suiteRunner = new SuiteRunner(suiteConfig.suite, runArgs, status)
              suiteRunner.run()
            }
          }

          val duration = System.currentTimeMillis - runStartTime
          if (stopRequested()) {
            dispatch(RunStopped(tracker.nextOrdinal(), Some(duration)))
          }
          else {
            dispatch(RunCompleted(tracker.nextOrdinal(), Some(duration)))
          }
        }
        catch {
          case e: InstantiationException =>
            dispatch(RunAborted(tracker.nextOrdinal(), Resources("cannotInstantiateSuite", e.getMessage), Some(e), Some(System.currentTimeMillis - runStartTime)))
          case e: IllegalAccessException =>
            dispatch(RunAborted(tracker.nextOrdinal(), Resources("cannotInstantiateSuite", e.getMessage), Some(e), Some(System.currentTimeMillis - runStartTime)))
          case e: NoClassDefFoundError =>
            dispatch(RunAborted(tracker.nextOrdinal(), Resources("cannotLoadClass", e.getMessage), Some(e), Some(System.currentTimeMillis - runStartTime)))
          case e: Throwable =>
            dispatch(RunAborted(tracker.nextOrdinal(), Resources.bigProblems(e), Some(e), Some(System.currentTimeMillis - runStartTime)))
        }
      }
    }
    finally {
      dispatch.dispatchDisposeAndWaitUntilDone()
      doneListener.done()
    }
  }

  private[scalatest] def excludesWithIgnore(excludes: Set[String]) = excludes + "org.scalatest.Ignore"

  private[scalatest] def withClassLoaderAndDispatchReporter(runpathList: List[String], reporterSpecs: ReporterConfigurations,
      graphicReporter: Option[Reporter], passFailReporter: Option[Reporter])(f: (ClassLoader, DispatchReporter) => Unit): Unit = {

    val loader: ClassLoader = getRunpathClassLoader(runpathList)
    try {
      Thread.currentThread.setContextClassLoader(loader)
      try {
        val dispatchReporter = ReporterFactory.getDispatchReporter(reporterSpecs, graphicReporter, passFailReporter, loader, None)
        try {
          f(loader, dispatchReporter)
        }
        finally {
          dispatchReporter.dispatchDisposeAndWaitUntilDone()
        }
      }
      catch {
        // getDispatchReporter may complete abruptly with an exception, if there is an problem trying to load
        // or instantiate a custom reporter class.
        case ex: Throwable => {
          System.err.println(Resources("bigProblemsMaybeCustomReporter"))
          ex.printStackTrace(System.err)
        }
      }
    }
    finally {
      // eventually call close on the RunpathClassLoader
    }
  }

  private[scalatest] def getRunpathClassLoader(runpathList: List[String]): ClassLoader = {

    if (runpathList == null)
      throw new NullPointerException
    if (runpathList.isEmpty) {
      classOf[Suite].getClassLoader // Could this be null technically?
    }
    else {
      val urlsList: List[URL] =
        for (raw <- runpathList) yield {
          try {
            new URL(raw)
          }
          catch {
            case murle: MalformedURLException => {
  
              // Assume they tried to just pass in a file name
              val file: File = new File(raw)
  
              // file.toURL may throw MalformedURLException too, but for now
              // just let that propagate up.
              file.toURI.toURL // If a dir, comes back terminated by a slash
            }
          }
        }
  
      // Here is where the Jini preferred class loader stuff went.

      // Tell the URLConnections to not use caching, so that repeated runs and reruns actually work
      // on the latest binaries.
      for (url <- urlsList) {
        try {
          url.openConnection.setDefaultUseCaches(false)
        }
        catch {
          case e: IOException => // just ignore these
        }
      }

      new URLClassLoader(urlsList.toArray, classOf[Suite].getClassLoader)
    }
  }

  private[scalatest] def usingEventDispatchThread(f: => Unit): Unit = {
    SwingUtilities.invokeLater(
      new Runnable() {
        def run() {
          f
        }
      }
    )
  }
}

/*
Runner command line arguments:

a
b - sbt reporter (only used inside ScalaTestFramework)
c - parallel execution
d
e - standard error reporter
f - file reporter
g - graphical reporter
h - HTML Reporter
i
j - run a JUnit tests class
k
l - tags to exclude
m - members only path
n - tags to include
o - standard out reporter
p - space-separated runpath
q
r - custom reporter
s - suite class name
t - testNG XML config file
u - junit xml reporter
v
w - wildcard path
x (saving this for a native xml reporter)
y
z

D - configMap pair, key=value

 * 
*/
argument	description	example
`-Dkey=value`	defines a key/value pair for the config map	`-DmaxConnections=100`
`-R <runpath elements>`	the specifies the runpath from which tests classes will be discovered and loaded (Note: only one `-R` allowed)	Unix: `-R target/classes:target/generated/classes` Windows: `-R target\classes;target\generated\classes`
`-n <tag name>`	specifies a tag to include (Note: only one tag name allowed per `-n`)	`-n UnitTests -n FastTests`
`-l <tag name>`	specifies a tag to exclude (Note: only one tag name allowed per `-l`)	`-l SlowTests -l PerfTests`
`-P[S][integer thread count]`	specifies a parallel run, with optional suite sorting and thread count (Note: only one `-P` allowed)	`-P`, `-PS`, `-PS 8`, or `-P8`
`-s <suite class name>`	specifies a suite class to run	`-s com.company.project.StackSpec`
`-m <members-only package>`	requests that suites that are direct members of the specified package be discovered and run	`-m com.company.project`
`-w <wildcard package>`	requests that suites that are members of the specified package or its subpackages be discovered and run	`-w com.company.project`
`-q <suffixes>`	specify suffixes to discover	`-q Spec -q Suite`
`-Q`	discover only classes whose names end with `Spec` or `Suite` (or other suffixes specified by `-q`)	`-Q`
`-j <JUnit class name>`	instantiate and run a JUnit test class	`-j StackTestClass`
`-b <TestNG XML file>`	run TestNG tests using the specified TestNG XML file	`-b testng.xml`
`-F <span scale factor>`	a factor by which to scale time spans (Note: only one `-F` is allowed)	`-F 10` or `-F 2.5`
`-T <sorting timeout>`	specifies a integer timeout (in seconds) for sorting the events of parallel runs back into sequential order	`-T 5`
`-y <chosen styles>`	specifies chosen styles for your project	`-y org.scalatest.FlatSpec`
`-i <suite ID>`	specifies a suite to run by ID (Note: must follow `-s`, and is intended to be used primarily by tools such as IDEs.)	`-i com.company.project.FileSpec-file1.txt`
`-t <test name>`	select the test with the specified name (Note: must follow `-s` or `-i`)	`-t "An empty Stack should complain when popped"`
`-z <test name substring>`	select tests whose names include the specified substring (Note: must follow `-s` or `-i`)	`-z "popped"`
`-g[NCXEHLOPQMD]`	select the graphical reporter	`-g`
`-f[NCXEHLOPQMDWSFU] <filename>`	select the file reporter	`-f output.txt`
`-u <directory name>`	select the JUnit XML reporter	`-u target/junitxmldir`
`-h <directory name> [-Y <css file name>]`	select the HTML reporter, optionally including the specified CSS file	`-h target/htmldir -Y src/main/html/customStyles.css`
`-v`	print the ScalaTest version	`-v` or, also `-version`
`-o[NCXEHLOPQMDWSFU]`	select the standard output reporter	`-o`
`-e[NCXEHLOPQMDWSFU]`	select the standard error reporter	`-e`
`-C[NCXEHLOPQMD] <reporter class>`	select a custom reporter	`-C com.company.project.BarReporter`
org.scalatest.tools.Runner.scala Maven / Gradle / Ivy

Executing suites

Specifying the config map

Specifying a runpath

Specifying reporters

Configuring Reporters

Specifying tags to include and exclude

Specifying suffixes to discover

Executing `Suite`s in parallel

Specifying `Suite`s

Specifying "members-only" and "wildcard" `Suite` paths

Specifying chosen styles

Selecting suites and tests

Specifying a span scale factor

Specifying TestNG XML config file paths

Specifying JUnit tests
