net.grinder.script.Statistics Maven / Gradle / Ivy
// Copyright (C) 2003 - 2008 Philip Aston
// All rights reserved.
//
// This file is part of The Grinder software distribution. Refer to
// the file LICENSE which is part of The Grinder distribution for
// licensing details. The Grinder distribution is available on the
// Internet at http://grinder.sourceforge.net/
//
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS
// FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE
// COPYRIGHT HOLDERS OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT,
// INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES
// (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
// SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
// HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT,
// STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
// ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED
// OF THE POSSIBILITY OF SUCH DAMAGE.
package net.grinder.script;
import net.grinder.common.GrinderException;
/**
* Script statistics query and reporting API.
*
*
* An implementation of this interface can be obtained by calling {@link
* Grinder.ScriptContext#getStatistics getStatistics} on the
* {@link Grinder#grinder grinder} context object.
*
*
* Statistics
*
*
* The following table lists the statistics provided by The Grinder. Each
* statistic has a unique name. Basic statistics either hold long
* integer values or double floating-point values. Sample statistics
* are a special type of statistic that hold aggregate information about a
* series of long or double sample values; specifically count (number
* of samples), sum (total of all sample values), and sample
* variance.
*
*
*
*
* Name
* Type
* Description
*
*
*
* errors
* basic long
* The number of tests performed that resulted in an error.
*
*
*
* timedTests
* sample long
* Sample statistic that records successful tests.
* A test is considered successful if it is not marked as an error.
*
*
*
* userLong0, userLong1, userLong2,
* userLong3, userLong4
* basic long
* Statistics that scripts and custom plug-ins can use for their own
* purposes.
*
*
*
* userDouble0, userDouble1, userDouble2,
* userDouble3, userDouble4
* basic double
* Statistics that scripts and custom plug-ins can use for their own
* purposes.
*
*
*
* untimedTests
* basic long
* The number of successful tests performed that have no timing
* information.
*
This statistic only used when reporting statistics to the console,
* it has no meaning in a worker process.
*
*
*
*
* period
* basic long
* The sampling period duration, in milliseconds.
This statistic is
* used to define views for evaluation in the console and for the overall
* time used to calculate rates in the Totals line in the final summary
* table.
*
*
*
* peakTPS
* basic double
* The highest Tests Per Second figure in the current sampling period.
*
This statistic is used to define views for evaluation in the
* console and has no meaning in a worker process.
*
*
*
*
*
* The {@link #getForCurrentTest()} and {@link #getForLastTest()} methods allow
* scripts to query or update statistics for a single test. Consequentially,
* the statistics should be interpreted as follows when using this interface.
*
*
*
*
* Name
* Type
* Meaning for a single test
*
*
*
*
* errors
* basic long
* 1 if the test resulted in an error, otherwise
* 0.
*
*
* timedTests
* sample long
* If the test was successful, the count is 1 and the sum is
* the test time in milliseconds, otherwise the sum and the count are zero.
* The variance is always 0.
*
*
*
*
* userLong0, userLong1, userLong2,
* userLong3, userLong4
* basic long
* Depends on how the script or custom plug-in chooses to use the
* statistic.
*
*
*
* userDouble0, userDouble1, userDouble2,
* userDouble3, userDouble4
* basic double
* Depends on how the script or custom plug-in chooses to use the
* statistic.
*
*
*
* untimedTests
* basic long
* Not relevant.
*
*
*
* period
* basic long
* Not relevant.
*
*
*
* peakTPS
* basic double
* Not relevant.
*
*
*
*
*
* The Grinder updates the statistics for each test just before it is recorded
* as follows:
*
*
*
* - If errors is
0, the elapsed time of the test is
* added to the timedTests sample statistic.
* - If errors is not
0, the timedTests
* and untimedTests statistics are reset to zero, and errors
* is set to 1.
*
*
* If the grinder.reportTimesToConsole property (see The Grinder manual
* ) is false, the statistics sent to the console are further
* modified by setting untimedTests to the count of the
* timedTests statistic, and resetting timedTests.
*
* HTTP Plug-in Statistics
*
*
* The HTTP plug-in adds a number of basic long statistics. Although these will
* be updated for a single test, many HTTP requests might be wrapped in that
* test. Where there is more than one HTTP request, the statistic values will
* reflect the sum for all requests unless otherwise stated.
*
*
*
*
* Name
* Type
* Description
*
*
*
* httpplugin.responseStatus
* basic long
* The HTTP status code of the response to the last HTTP request wrapped in
* the test.
*
*
*
* httpplugin.responseLength
* basic long
* The length of the HTTP response in bytes.
*
*
*
* httpplugin.responseErrors
* basic long
* 1 if the HTTP response status code was greater or equal
* to 400, otherwise 0.
*
*
* httpplugin.dnsTime
* basic long
* The time taken to resolve the host name in milliseconds.
*
*
*
* httpplugin.connectTime
* basic long
* The time taken to establish the HTTP connection in milliseconds. (This
* includes time to resolve the host name).
*
*
*
* httpplugin.firstByteTime
* basic long
* The time taken to receive the first response byte in milliseconds. (This
* includes time to resolve the host name and establish the connection).
*
*
*
*
*
* Querying statistics
*
*
* {@link StatisticsForTest#getDouble(String)} and
* {@link StatisticsForTest#getLong(String)} provide basic
* statistics about the current or last test performed by the calling worker
* thread. There is also {@link StatisticsForTest#getSuccess()}, which is
* equivalent to getLong("errors") != 0.
*
*
*
* Example of querying the result of a test:
*
*
*
*
*
* result1 = test1.doSomething()
*
* statisticsForTest = grinder.statistics.forLastTest
*
* if statisticsForTest.success and \
* statisticsForTest.getLong("httpplugin.responseStatus") == 200:
* # ...
*
*
*
*
*
* Calling query methods on the result of {@link #getForLastTest} provides
* information about the last test performed by the calling worker thread.
*
*
*
* Calling query methods on the result of {@link #getForCurrentTest} from within
* code wrapped by a {@link Test} proxy provides information about the test in
* progress. This information may be partially complete.
*
*
*
* There are no general methods to access the count, sum, and variance of sample
* statistics; these terms aren't that meaningful for a single test. Instead,
* there are specific methods which influence the only sample statistic used
* by The Grinder - the timedTests statistic. The time of the last test
* can be obtained with {@link StatisticsForTest#getTime()} (or the elapsed time
* of the current test when used with {@link #getForCurrentTest()}}, and
* {@link StatisticsForTest#getSuccess()} returns whether the test was
* successful or not.
*
*
* There's a subtle difference between the sum of timedTests and the
* result of {@link StatisticsForTest#getTime()}.
* {@link StatisticsForTest#getTime()} always returns the time
* taken by the test, even if the test was an error and the time will not be
* added to timedTests. This allows the script to access the time
* taken by a failed test, even though it's not recorded anywhere else.
*
*
* Updating statistics
*
*
* The following methods allow basic statistics to be updated.
*
*
*
* - {@link StatisticsForTest#setSuccess(boolean)}
* - {@link StatisticsForTest#setLong(String, long)}
* - {@link StatisticsForTest#setDouble(String, double)}
* - {@link StatisticsForTest#addLong(String, long)}
*
- {@link StatisticsForTest#addDouble(String, double)}
*
*
*
* The only way to influence the timedTests sample statistic through
* this interface is to mark the test as an error.
*
*
*
* By default, test statistics reports are automatically sent to the console and
* data log when the test proxy call completes, so the script cannot modify the
* test statistics after the call. Scripts can turn off this automatic reporting
* for the current worker thread by using {@link #setDelayReports}. Having done
* so, the script can modify or set the statistics before they are sent to the
* log and the console. The statistics reports are sent at a later time as
* described in {@link #setDelayReports}. For example:
*
*
*
*
* grinder.statistics.delayReports = 1
*
* result1 = test1.doSomething()
*
* if isFailed(result1):
* # Mark test as failure. The appropriate failure detection
* # depends on the type of test.
* grinder.statistics.forLastTest.success = 0
*
*
*
*
*
* It is also possible to set the statistics from within test implementation
* itself using {@link #getForCurrentTest()}. Changes to the standard statistics
* will be modified by The Grinder engine when the test finishes as described
* above.
*
*
* Registering new expressions
*
* New statistics expressions for the console and the data log can be registered
* using {@link #registerSummaryExpression(String, String)} and
* {@link #registerDataLogExpression(String, String)}.
*
*
* @author Philip Aston
*/
public interface Statistics {
/**
* Access to the statistics for the current test.
*
*
* This is only valid when called from code wrapped within a Test. If this is
* not the case, {@link InvalidContextException} will be thrown. E.g.
*
*
*
* def foo():
* statistics = grinder.statistics.getForCurrentTest()
* t = statistics.time # Time since foo() was called.
* statistics.success = 0 # Mark test as bad.
*
* instrumentedFoo = Test(1, "My Test").wrap(foo)
*
* instrumentedFoo() # OK.
* foo() # The statistics.getForCurrentTest() call in
* # foo() will throw exception as there is no
* # current test.
* statistics.getForCurrentTest() # Will throw exception, no current test.
*
*
* @return The statistics for the current test.
* @throws InvalidContextException
* If not called from a worker thread.
* @throws InvalidContextException
* If there is no test in progress.
* @see #getForLastTest()
* @see #isTestInProgress()
*/
StatisticsForTest getForCurrentTest() throws InvalidContextException;
/**
* Access the statistics for the last completed test. These can only
* be updated if {@link #setDelayReports(boolean) reporting has been
* delayed}.
*
* @return The statistics for the last test.
* @throws InvalidContextException
* If not called from a worker thread.
* @throws InvalidContextException
* If called before the first test has started.
* @see #getForCurrentTest
*/
StatisticsForTest getForLastTest() throws InvalidContextException;
/**
* Returns whether there is a test in progress.
*
* @return true {@link #getForCurrentTest} will
* not throw {@link InvalidContextException}.
*/
boolean isTestInProgress();
/**
* Use to delay reporting of the last test statistics to the log and the
* console so that the script can modify them. Normally test statistics are
* reported automatically when the code wrapped by the test completes.
*
*
* With this set to true the statistics for a completed test
* will be reported at the following times:
*
* - When the next test begins.
* - When an enclosing test ends.
* - When the current run completes.
* - When the script calls {@link #report}.
* - When the script calls
setDelayReports(false).
*
*
*
*
* This method only changes reporting for the calling worker thread.
*
*
* @param b
* false => enable automatic reporting when the code
* wrapped by a test completes (the default behaviour);
* true => delay reporting.
* @throws InvalidContextException
* If not called from a worker thread.
* @see #report
*/
void setDelayReports(boolean b) throws InvalidContextException;
/**
* Send any pending statistics for the last completed test to the data log and
* the console.
*
*
* Calling this does nothing if there are no pending statistics to report.
* This will be the case if {@link #setDelayReports} has not been called to
* delay reporting.
*
*
* @throws InvalidContextException
* If not called from a worker thread.
*/
void report() throws InvalidContextException;
/**
* Register a new "summary" statistic expression. This expression will appear
* in the worker process output log summaries and the console.
*
*
* Statistic expressions are composed of statistic names (see
* {@link Statistics}) in a simple post-fix format using the symbols
* +, -, / and *,
* which have their usual meanings, in conjunction with simple statistic names
* or sub-expressions. Precedence is controlled by grouping expressions in
* parentheses. For example, the error rate is
* (* (/ errors period) 1000) errors per second. The symbol
* sqrt can be used to calculate the square root of an
* expression.
*
*
* Sample statistics, such as timedTests, must be introduced with
* one of sum, count, or variance,
* depending on the attribute of interest.
*
*
*
* For example, the statistic expression (/ (sum timedTests)
* (count timedTests))
* represents the mean test time in milliseconds.
*
*
* @param displayName
* A display name. In the console, this is converted to a key for an
* internationalised resource bundle look up by prefixing the string
* with statistic. and replacing any whitespace with
* underscores.
* @param expression
* The expression string.
* @throws GrinderException
* If the expression could not be registered.
*/
void registerSummaryExpression(String displayName, String expression)
throws GrinderException;
/**
* Register a new "detail" statistic expression. This expression will appear
* in the worker process data log. Each test invocation will have an value
* displayed for the detail statistic expression.
*
*
* You should call registerDataLogExpression from the top level
* of your script. It cannot be called from a worker thread - the data logs
* are initialised by the time the worker threads start.
*
*
* @param displayName
* A display name.
* @param expression
* The expression string.
* @throws GrinderException
* If the expression could not be registered.
* @throws InvalidContextException
* If called from a worker thread.
* @see #registerSummaryExpression(String, String) for details of the
* expression format.
*/
void registerDataLogExpression(String displayName, String expression)
throws GrinderException, InvalidContextException;
/**
* Query and update methods for the statistics relating to a particular call
* of a test.
*
* @see Statistics#getForLastTest()
* @see Statistics#getForCurrentTest()
*/
interface StatisticsForTest {
/**
* Return the Test that the statistics are for.
*
* @return The test.
*/
net.grinder.common.Test getTest();
/**
* Sets the long statistic statisticName to the specified
* value.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @param value
* The value.
* @throws InvalidContextException
* If called when the statistics have already been sent for the
* last test performed by this thread - see
* {@link Statistics#setDelayReports}.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known basic
* long statistic.
*/
void setLong(String statisticName, long value)
throws InvalidContextException, NoSuchStatisticException;
/**
* Sets the double statistic statisticName to the specified
* value.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @param value
* The value.
* @throws InvalidContextException
* If called when the statistics have already been sent for the
* last test performed by this thread - see
* {@link Statistics#setDelayReports(boolean)}.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known basic
* double statistic.
*/
void setDouble(String statisticName, double value)
throws InvalidContextException, NoSuchStatisticException;
/**
* Add value to the long statistic statisticName.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @param value
* The value.
* @throws InvalidContextException
* If called when the statistics have already been sent for the
* last test performed by this thread - see
* {@link Statistics#setDelayReports(boolean)}.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known basic
* long statistic.
*/
void addLong(String statisticName, long value)
throws InvalidContextException, NoSuchStatisticException;
/**
* Add value to the double statistic
* statisticName.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @param value
* The value.
* @throws InvalidContextException
* If called when the statistics have already been sent for the
* last test performed by this thread - see
* {@link Statistics#setDelayReports(boolean)}.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known basic
* double statistic.
*/
void addDouble(String statisticName, double value)
throws InvalidContextException, NoSuchStatisticException;
/**
* Return the value of long statistic statisticName.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @return The value.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known
* basic long statistic.
*/
long getLong(String statisticName) throws NoSuchStatisticException;
/**
* Return the value of the double statistic statisticName.
*
* @param statisticName
* The statistic name. See {@link Statistics} for a list of valid
* names.
* @return The value.
* @throws NoSuchStatisticException
* If statisticName does not refer to a known basic
* double statistic.
*/
double getDouble(String statisticName) throws NoSuchStatisticException;
/**
* Convenience method that sets whether the last test should be considered a
* success or not.
*
* @param success
* If true errors is set to
* 0, otherwise errors is set to
* 1.
* @throws InvalidContextException
* If called when the statistics have already been sent for the
* last test performed by this thread - see
* {@link Statistics#setDelayReports(boolean)}.
*/
void setSuccess(boolean success) throws InvalidContextException;
/**
* Convenience method that returns whether the test was a success
* (errors is zero) or not.
*
* @return Whether the test was a success.
*/
boolean getSuccess();
/**
* Returns the elapsed time for the test.
*
* If this {@link Statistics.StatisticsForTest StatisticsForTest} was
* obtained with {@link Statistics#getForCurrentTest}, the result will be
* the elapsed time since the test in progress was started. If it was
* obtained with {@link Statistics#getForLastTest}, the result will be the
* time taken by the last test.
*
*
* {@link #getTime()} always returns the time taken by the test, even if the
* test was an error and the time will not be added to timedTests.
*
*
* @return The elapsed time for the test.
*/
long getTime();
}
}