robotframework-2.7.7.doc.userguide.src.ExecutingTestCases.OutputFiles.rst Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of robotframework Show documentation
Show all versions of robotframework Show documentation
High level test automation framework
The newest version!
Created outputs
---------------
Several output files are created when tests are executed, and all of
them are somehow related to test results. This section discusses what
outputs are created, how to configure where they are created, and how
to fine-tune their contents.
.. contents::
:depth: 2
:local:
Different output files
~~~~~~~~~~~~~~~~~~~~~~
This section explains what different output files can be created and
how to configure where they are created. Output files are configured
using command line options, which get the path to the output file in
question as an argument. A special value :opt:`NONE`
(case-insensitive) can be used to disable creating a certain output
file.
Output directory
''''''''''''''''
All output files can be set using an absolute path, in which case they
are created to the specified place, but in other cases, the path is
considered relative to the output directory. The default output
directory is the directory where the execution is started from, but it
can be altered with the :opt:`--outputdir (-d)` option. The path
set with this option is, again, relative to the execution directory,
but can naturally be given also as an absolute path. Regardless of how
a path to an individual output file is obtained, its parent directory
is created automatically, if it does not exist already.
Output file
'''''''''''
Output files contain all the test execution results in XML
format. Log_ and report_ files are generated based on
output files, and output files can also be `combined and otherwise
post-processed`__ after the test execution.
The command line option :opt:`--output (-o)` determines the path where
the output file is created relative to the `output directory`_. The default
name for the output file, when tests are run, is :path:`output.xml`.
When post-processing outputs, new output files are not created unless
:opt:`--output (-o)` option is explicitly used.
Starting from Robot Framework 2.6, it is possible to disable creation of
the output file also when running tests with special value :opt:`NONE`. In this
case also other output files, except for the `debug file`_, are disabled.
__ `Post-processing outputs`_
Log file
''''''''
Log files contain details about the executed test cases in HTML
format. They have a hierarchical structure showing test suite, test
case and keyword details. Log files are needed nearly every time when
test results are to be investigated in detail. Even though log files
also have statistics, reports are better for
getting an higher-level overview.
The command line option :opt:`--log (-l)` determines where log
files are created. Unless the special value :opt:`NONE` is used,
log files are always created and their default name is
:path:`log.html`.
.. figure:: src/ExecutingTestCases/log_passed.png
:target: src/ExecutingTestCases/log_passed.html
:width: 500
An example of beginning of a log file
.. figure:: src/ExecutingTestCases/log_failed.png
:target: src/ExecutingTestCases/log_failed.html
:width: 500
An example of a log file with keyword details visible
Report file
'''''''''''
Report files contain an overview of the test execution results in HTML
format. They have statistics based on tags and executed test suites,
as well as a list of all executed test cases. When both reports and
logs are generated, the report has links to the log file for easy
navigation to more detailed information. It is easy to see the
overall test execution status from report, because its background
color is green, if all `critical tests`_ pass, and bright red
otherwise.
The command line option :opt:`--report (-r)` determines where
report files are created. Similarly as log files, reports are always
created unless :opt:`NONE` is used as a value, and their default
name is :path:`report.html`.
.. figure:: src/ExecutingTestCases/report_passed.png
:target: src/ExecutingTestCases/report_passed.html
:width: 500
An example report file of successful test execution
.. figure:: src/ExecutingTestCases/report_failed.png
:target: src/ExecutingTestCases/report_failed.html
:width: 500
An example report file of failed test execution
XUnit compatible result file
''''''''''''''''''''''''''''
XUnit result files contain the test run summary in XUnit compatible XML format.
This file can be used as input for tools that process XUnit data. For example,
Hudson continuous integration server has built-in support for this output format
and can be configured to generate test history based on this file.
XUnit output files are not created unless the command line option
:opt:`--xunitfile (-x)` is used explicitly.
Debug file
''''''''''
Debug files are plain text files that are written during the test
execution. All messages got from test libraries are written to them,
as well as information about started and ended test suites, test cases
and keywords. Debug files can be used for monitoring the test
execution. This can be done using, for example, a separate `fileviewer.py`_
tool, or in UNIX-like systems, simply with the :prog:`tail -f` command.
Debug files are not created unless the command line option
:opt:`--debugfile (-b)` is used explicitly.
Timestamping output files
'''''''''''''''''''''''''
All output files listed in this section can be automatically timestamped
with the option :opt:`--timestampoutputs (-T)`. When this option is used,
a timestamp in the format :opt:`YYYYMMDD-hhmmss` is placed between
the extension and the base name of each file. The example below would,
for example, create such output files as
:path:`output-20080604-163225.xml` and :path:`mylog-20080604-163225.html`::
pybot --timestampoutputs --log mylog.html --report NONE tests.html
Setting titles
''''''''''''''
The default titles for logs_ and reports_ are generated by prefixing
the name of the top-level test suite with :name:`Test Log` or
:name:`Test Report`. Custom titles can be given from the command line
using the options :opt:`--logtitle` and :opt:`--reporttitle`,
respectively. Underscores in the given titles are converted to spaces
automatically.
Example::
pybot --logtitle Smoke_Test_Log --reporttitle Smoke_Test_Report --include smoke my_tests/
Setting background colors
'''''''''''''''''''''''''
By default the `report file`_ has a green background when all the
`critical tests`_ pass and a red background otherwise. These colors
can be customized by using the :opt:`--reportbackground` command line
option, which takes two or three colors separated with a colon as an
argument::
--reportbackground blue:red
--reportbackground green:yellow:red
--reportbackground #00E:#E00
If you specify two colors, the first one will be used instead of the
default green color and the second instead of the default red. This
allows, for example, using blue instead of green to make backgrounds
easier to separate for color blind people.
If you specify three colors, the first one will be used when all the
test succeed, the second when only non-critical tests have failed, and
the last when there are critical failures. This feature thus allows
using a separate background color, for example yellow, when
non-critical tests have failed.
The specified colors are used as a value for the :code:`body`
element's :code:`background` CSS property. The value is used as-is and
can be a HTML color name (e.g. :code:`red`), a hexadecimal value
(e.g. :code:`#F00` or :code:`#FF0000`), or an RGB value
(e.g. :code:`rgb(255,0,0)`). The default green and red colors are
specified using hexadecimal values :code:`#9F6` and :code:`#F33`,
respectively.
Log levels
~~~~~~~~~~
Available log levels
''''''''''''''''''''
Messages in `log files`_ can have different log levels. Some of the
messages are written by Robot Framework itself, but also executed
keywords can `log information`__ using different levels. The available
log levels are:
:msg:`FAIL`
Used when a keyword fails. Can be used only by Robot Framework itself.
:msg:`WARN`
Used to display warnings. They shown also in `the console and in
the Test Execution Errors section in log files`__, but they
do not affect the test case status.
:msg:`INFO`
The default level for normal messages. By default,
messages below this level are not shown in the log file.
:msg:`DEBUG`
Used for debugging purposes. Useful, for example, for
logging what libraries are doing internally. When a keyword fails,
a traceback showing where in the code the failure occurred is
logged using this level automatically.
:msg:`TRACE`
More detailed debugging level. The keyword arguments and return values
are automatically logged using this level.
__ `Logging information`_
__ `Errors and warnings during execution`_
Setting log level
'''''''''''''''''
By default, log messages below the :msg:`INFO` level are not logged, but this
threshold can be changed from the command line using the
:opt:`--loglevel (-L)` option. This option takes any of the
available log levels as an argument, and that level becomes the new
threshold level. A special value :msg:`NONE` can also be used to
disable logging altogether.
Starting from Robot Framework 2.5.2, it is possible to use the
:opt:`--loglevel` option also when `post-processing outputs`_ with
:prog:`rebot`. This allows, for example, running tests initially with
the :msg:`TRACE` level, and generating smaller log files for normal
viewing later with the :msg:`INFO` level. By default all the messages
included during execution will be included also with :prog:`rebot`.
Messages ignored during the execution cannot be recovered.
Another possibility to change the log level is using the `BuiltIn
keyword`_ :name:`Set Log Level` in the test data. It takes the same
arguments as the :opt:`--loglevel` option, and it also returns the
old level so that it can be restored later, for example, in a `test
teardown`_.
Visible log level
'''''''''''''''''
Starting from Robot Framework 2.7.2, if the log file contains messages at
:msg:`DEBUG` or :msg:`TRACE` levels, a visible log level drop down is shown
in the upper right corner. This allows users to remove messages below chosen
level from the view. This can be useful especially when running test at
:msg:`TRACE` level.
.. figure:: src/ExecutingTestCases/visible_log_level.png
:target: src/ExecutingTestCases/visible_log_level.html
:width: 500
An example log showing the visible log level drop down
By default the drop down will be set at the lowest level in the log file, so
that all messages are shown. The default visible log level can be changed using
:opt:`--loglevel` option by giving the default after the normal log level
separated by a colon::
--loglevel DEBUG:INFO
In the above example, tests are run using level :msg:`DEBUG`, but
the default visible level in the log file is :msg:`INFO`.
Splitting logs
~~~~~~~~~~~~~~
Normally the log file is just a single HTML file. When the amount of he test
cases increases, the size of the file can grow so large that opening it into
a browser is inconvenient or even impossible. Starting from Robot Framework
2.6, it is possible to use the :opt:`--splitlog` option to split parts of
the log into external files that are loaded transparently into the browser
when needed.
The main benefit of splitting logs is that individual log parts are so small
that opening and browsing the log file is possible even if the amount
of the test data is very large. A small drawback is that the overall size taken
by the log file increases.
Technically the test data related to each test case is saved into
a JavaScript file in the same folder as the main log file. These files have
names such as :path:`log-42.js` where :path:`log` is the base name of the
main log file and :path:`42` is an incremented index.
.. note:: When copying the log files, you need to copy also all the
:path:`log-*.js` files or some information will be missing.
Configuring statistics
~~~~~~~~~~~~~~~~~~~~~~
There are several command line options that can be used to configure
and adjust the contents of the :name:`Statistics by Tag`, :name:`Statistics
by Suite` and :name:`Test Details by Tag` tables in different output
files. All these options work both when executing test cases and when
post-processing outputs.
Configuring displayed suite statistics
''''''''''''''''''''''''''''''''''''''
When a deeper suite structure is executed, showing all the test suite
levels in the :name:`Statistics by Suite` table may make the table
somewhat difficult to read. Bt default all suites are shown, but you can
control this with the command line option :opt:`--suitestatlevel` which
takes the level of suites to show as an argument::
--suitestatlevel 3
Including and excluding tag statistics
''''''''''''''''''''''''''''''''''''''
When many tags are used, the :name:`Statistics by Tag` table can become
quite congested. If this happens, the command line options
:opt:`--tagstatinclude` and :opt:`--tagstatexclude` can be
used to select which tags to display, similarly as
:opt:`--include` and :opt:`--exclude` are used to `select test
cases`__::
--tagstatinclude some-tag --tagstatinclude another-tag
--tagstatexclude owner-*
--tagstatinclude prefix-* --tagstatexclude prefix-13
__ `By tag names`_
Generating combined tag statistics
''''''''''''''''''''''''''''''''''
The command line option :opt:`--tagstatcombine` can be used to
generate aggregate tags that combine statistics from multiple
tags. There are three somewhat different ways for giving arguments for
this option:
One tag as a `simple pattern`_
All tags matching the given pattern are combined together.
Two or more tags separated by :code:`AND` or :code:`&`
The combined statistics contain tests that have all the listed tags.
Tags can be given as simple patterns.
Two or more tags separated by :code:`NOT`
The combined statistics contain tests that have the first tag but not
any of the others. Also in this case tags may be patterns.
The following examples illustrate these usages, and the figure below shows a snippet
of the resulting :name:`Statistics by Tag` table when the example test data is
executed with these options::
--tagstatcombine owner-*
--tagstatcombine smokeANDmytag
--tagstatcombine smokeNOTowner-janne*
.. figure:: src/ExecutingTestCases/tagstatcombine.png
:width: 550
Examples of combined tag statistics
As the above example shows, the name of the added combined statistic
is, by default, generated from the given pattern. In certain
situations this name can look pretty cryptic and it is possible to
specify a more descriptive name. This name is given after the pattern
separating it with a colon (:code:`:`). Example below generates
combined tag so that the name shown in reports and logs is
:name:`Critical Tests`::
--tagstatcombine *NOTnon-critical:Critical_Tests
Creating links from tag names
'''''''''''''''''''''''''''''
You can add external links to the :name:`Statistics by Tag` table by
using the command line option :opt:`--tagstatlink`. Arguments to this
option are given in the format :opt:`tag:link:name`, where :opt:`tag`
specifies the tags to assign the link to, :opt:`link` is the link to
be created, and :opt:`name` is the name to give to the link.
:opt:`tag` may be a single tag, but more commonly a `simple pattern`_
where :code:`*` matches anything and :code:`?` matches any single
character. When :opt:`tag` is a pattern, the matches to wildcards may
be used in :opt:`link` and :opt:`title` with the syntax :code:`%N`,
where "N" is the index of the match starting from 1.
The following examples illustrate the usage of this option, and the
figure below shows a snippet of the resulting :name:`Statistics by
Tag` table when example test data is executed with these options::
--tagstatlink mytag:http://www.google.com:Google
--tagstatlink jython-bug-*:http://bugs.jython.org/issue_%1:Jython-bugs
--tagstatlink owner-*:mailto:%[email protected]?subject=Acceptance_Tests:Send_Mail
.. figure:: src/ExecutingTestCases/tagstatlink.png
:width: 550
Examples of links from tag names
Adding documentation to tags
''''''''''''''''''''''''''''
Tags can be given a documentation with the command line option
:opt:`--tagdoc`, which takes an argument in the format
:opt:`tag:doc`. :opt:`tag` is the name of the tag to assign the
documentation to, and it can also be a `simple pattern`_ matching
multiple tags. :opt:`doc` is the assigned documentation. Underscores
in the documentation are automatically converted to spaces and it
can also contain `HTML formatting`_.
The given documentation is shown with matching tags in the :name:`Test
Details by Tag` table, and as a tool tip for these tags in the
:name:`Statistics by Tag` table. If one tag gets multiple documentations,
they are combined together and separated with an ampersand.
Examples::
--tagdoc mytag:My_documentation
--tagdoc regression:*See*_http://info.html
--tagdoc owner-*:Original_author
Removing keywords from outputs
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Most of the content of `output files`_ comes from keywords and especially their
log messages. When creating higher level reports, log files are not necessarily
needed at all, and then keywords and their messages just take space
unnecessarily. Log files themselves can also grow overly large if they contain
`for loops`_ or other constructs that repeat certain keywords multiple times.
In these situations, the command line option :opt:`--removekeywords` can be
used to dispose of unnecessary keywords. It can be used both when executing
tests and with :prog:`rebot`, but in the former case keywords are not removed
from the output file. Keywords that contain warnings are not removed except
in :opt:`ALL` mode.
The option has the following modes of operation:
:opt:`ALL`
Remove data from all keywords unconditionally.
:opt:`PASSED`
Remove keyword data from test cases that have passed and do not
contain warnings_. In most cases, log files created after this contain
enough information to investigate possible failures.
:opt:`FOR`
Remove passed iterations from `for loops`_. Starting from Robot Framework
2.7.5, the last iteration is always kept.
:opt:`WUKS`
Remove all but last failing keyword inside BuiltIn_ keyword
:name:`Wait Until Keyword Succeeds`.
Examples::
rebot --removekeywords all output.xml
pybot --removekeywords passed --removekeywords for tests.txt
.. Note::
The support for using :opt:`--removekeywords` when executing tests as well
as :opt:`FOR` and :opt:`WUKS` modes were added in Robot Framework 2.7.
Setting start and end time of execution
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
When `combining outputs`_ using :prog:`rebot`, it is possible to set the start
and end time of the combined test suite using the options :opt:`--starttime`
and :opt:`--endtime`, respectively. This is convenient, because by default,
combined suites do not have these values. When both the start and end time are
given, the elapsed time is also calculated based on them. Otherwise the elapsed
time is got by adding the elapsed times of the child test suites together.
Starting from Robot Framework 2.5.6, it is also possible to use the above
mentioned options to set start and end times for a single suite when using
:prog:`rebot`. Using these options with a single output always affects the
elapsed time of the suite.
Times must be given as timestamps in the format :code:`YYYY-MM-DD
hh:mm:ss.mil`, where all separators are optional and the parts from
milliseconds to hours can be omitted. For example, :code:`2008-06-11
17:59:20.495` is equivalent both to :code:`20080611-175920.495` and
:code:`20080611175920495`, and also mere :code:`20080611` would work.
Examples::
rebot --starttime 20080611-17:59:20.495 output1.xml output2.xml
rebot --starttime 20080611-175920 --endtime 20080611-180242 *.xml
rebot --starttime 20110302-1317 --endtime 20110302-11418 myoutput.xml
System log
~~~~~~~~~~
Robot Framework has its own plain-text system log where it writes
information about
- Processed and skipped test data files
- Imported test libraries, resource files and variable files
- Executed test suites and test cases
- Created outputs
Normally users never need this information, but it can be
useful when investigating problems with test libraries or Robot Framework
itself. A system log is not created by default, but it can be enabled
by setting the environment variable :opt:`ROBOT_SYSLOG_FILE` so
that it contains a path to the selected file.
A system log has the same `log levels`_ as a normal log file, with the
exception that instead of :msg:`FAIL` it has the :msg:`ERROR`
level. The threshold level to use can be altered using the
:opt:`ROBOT_SYSLOG_LEVEL` environment variable like shown in the
example below. Possible `unexpected errors and warnings`__ are
written into the system log in addition to the console and the normal
log file.
.. sourcecode:: bash
#!/bin/bash
export ROBOT_SYSLOG_FILE=/tmp/syslog.txt
export ROBOT_SYSLOG_LEVEL=DEBUG
pybot --name Syslog_example path/to/tests
__ `Errors and warnings during execution`_