com.github.dakusui.thincrest.metamor.package-info.adoc Maven / Gradle / Ivy
Go to download
Show more of this group Show more artifacts with this name
Show all versions of thincrest-pcond Show documentation
Show all versions of thincrest-pcond Show documentation
Yet another assertion library powered by "pcond"
The newest version!
A package that holds classes for "Metamorphic Testing".
Metamorphic testing is a technique to alleviate "oracle problem" by verifying known relationship among IOs of multiple executions of a target function under test.
For instance, let's look at a situation, where you are testing an implementation of a mathematical function `sin(double x)`.
If you are going to check the function if the function gives a value equal to a value you calculated without using the function under test itself, you will need to calculate it by hand, which is error prone and very expensive.
However, we know some characteristics, which a `sin(x)` function must satisfy if it is properly implemented.
For instance, this equation holds for any real `x`.
----
sin(x) = sin(Math.PI - x)
----
Also, we can think of following:
----
(sin(x))^2 + (cos(x))^2 = 1
≡(sin(x))^2 + (sin(Math.PI/2 - x))^2 = 1
----
With this knowledge, we can verify the correctness of the implementation of a sine function even if we don't know the value _sin(x)_ should return for a given _x_.
== Design
Forllowing is a diagram that illustrates the pipeline of the metamorphic testing support by `thincrest-pcond`.
.Metamorphic Testing Support Package's Pipeline Design
[ditaa]
----
+------------+
|SOURCE VALUE|
+------------+
^
|
|
/-------------------------+--------------------------\
| /--------------\ /--------------\ /--------------\ |
| |input resolver| |input resolver| |input resolver+---------------------------+
| \--------------/ \--------------/ \--------------/ | |
\----------------------------------------------------/ |
: |
| |
V |
+---------------------+---------------------+ |
|Dataset | |
| +-----------+ +-----------+ +-----------+ | |
| |input value| |input value| |input value|<--------------------------+ |
| +-----------+ +-----------+ +-----------+ | | |
+---------------------^---------------------+ | |
| ^ | |
| | | |
| | | V
/-----------+---------------+-----------\ +-------------|------------------------+
| /---------+---------\ | | +----*--+ +-------+ +-------+ |
| |function under test| FUT controller| |Dataset |IO pair| |IO pair| |IO pair| |
| \---------+---------/ | | +----*--+ +-------+ +-------+ |
\-----------|---------------------------/ +-------------|------------------------+
: : ^ | ^
| | | | |
| +----------------------+ | |
+----------------------|-----------------------+ | |
|Dataset V | | |
| +------------+ +------------+ +------------+ | | |
| |output value| |output value| |output value|<------------------------+ |
| +------------+ +------------+ +------------+ | |
+----------------------------------------------+ |
|
|
|
/-----------+--------+-+----+-----------\
| /---------\ |
| |preformer| preformer controller |
| \----+----/ |
\------|--------------------------------/
: :
| |
| V
+---------------------|---------------------------------+
|Dataset V |
| +---------------+ +---------------+ +---------------+ |
| |preformed value| |preformed value| |preformed value| |
| +---------------+ +---------------+ +---------------+ |
+-------------------------------------------------------+
^
|
+---------|---------+
| | |proposition
| /------+------\ |
| | reducer | |
| \------+------/ |
| : |
| | |
| V |
| +------+------+ |
| |reduced value| |
| +------+------+ |
| ^ |
| | |
| /------+------\ |
| | checker | |
| \------+------/ |
| : |
+---------|---------+
V
+------+------+
| TEST RESULT |
+------+------+
----
For the detail of each component in the diagram, refer to <>.
[[componentsInPipeline]]
[cols="1,3,3,3"]
.Components in the Metamorphic Testing Support Package's Pipeline
|===
| |Description |Example: _sin(x)_ |Example: _search(query)_
|*SOURCE VALUE*
|An original value from which metamorphic input values are created through *InputResolver*.
|1.0
|"Hello, world"
|*InputResolver*
|A function that creates an input value from *SOURCE VALUE*.
|x, π-x
|query, query + (and "lang:en")
|*Dataset*
|A module to hold a set of data items.
This module is used for input values, output values, and IoPairs
|*Input Values:* `[1.0, 2.1415926535...]`
*Output Values:* `[0.01745..., 0.01745...]`
*IoPairs:* `[(1.0, 0.01745...), (2.1415926535..., 0.01745...)]`
|*Input Values:* `["Hello, world", "Hello, world" and "lang:en")]`
*Output Values:* `[(d1, d2, d3, d4, d5), (d1, d3, d5)]`
*IoPairs:* `[("Hello, world", (d1, d2, d3, d4, d5)), ("Hello, world" and "lang:en", (d1, d3, d5))]`
|*InputValue*
|A value to be given to FUT.
Usually, its type is the same as *SOURCE VALUE* ("endomorphic"), but it is not mandatory.
|1.0, 2.1415926535...
|"Hello, world", "Hello, world" AND lang:en
|*Function Under Test (FUT)*
|The function under test.
|_sin(x)_
|_search(query)_
|*FUT Controller*
|A module to control FUT's execution, input, and output.
This passes an input value to FUT and writes its result to IoPair Dataset.
|-
|-
|*IoPair*
|A module to hold a pair of an input value given to FUT and an output value from the FUT.
|(1.0, 0.01745...), (2.1415926535..., 0.01745...)
|("Hello, world", (d1, d2, d3, d4, d5)), ("Hello, world" + and "lang:en"), (d1, d3, d5))
|*Preformer*
|A function that converts *IoPair* into a value that can be processed by *Reducer*.
Normally just extracts output side of the *IoPair*.
|-
|-
|*Preformed Value*
|A value converted by *Preformer* function from *IoPair*.
|`(IoPair p) -> p.output()`
|`(IoPair p) -> p.output()`
|*Reducer*
|A function that converts preformed values into one value that can be examined by *Checker* predicate.
|`(Dataset ds) -> ds.get(0) - ds.get(1)`
|`ds.get(1).stream()
.filter(e -> ds.get(0).contains(e))
.collect(toList())`
|*Checker*
|A predicate that examines if the value produced by *Reducer* satisfies the specification of *FUT*.
|`v -> Objects.equals(v, 0)`, `Predicates.equalTo(0)`, etc.
|`Collection::isEmpty`, `Predicates.isEmpty()`, etc.
|*Proposition*
|A predicate that examines IoPair Dataset directly.
|`Objects.equals(ds.get(0), ds.get(1))`
|`ds.get(0).containsAll(ds.get(1))`
|*TEST RESULT*
|A boolean value that represents the result of the test (`true` - success / `false` - fail).
|`true`
|`true`
|===
You may think that a reducer can be a predicate whose parameter is a dataset of preformed values.
However, this approach sometimes results in a not helpful error message.
For instance, if you are testing a _sin_ function's implementation, and it has non-zero error, you may want to see how much big the error was.
If you directly check if the values of _sin(x)_ and _sin(π-x)_ are equal, you will just see the two values and they are not equal.
Not seeing the magnitude of the error.
To address it, you may want to compute the difference of them (_sin(x)_ - _sin(π-x)_) by the *Reducer* and check if it was zero or not by the *Checker*.
With this approach you will be able to see what you need in the error message on a failure.
Also, you can skip the `Preformer` because you can integrate the step in `Reducer` or `Checker`, which also may result in less informative message.
== Example
The entry-point of the metamorphic testing functionality of `pcond` is `MetamorphicTestCaseFactory` class.
It has several static method which return an instance of `MetamorphicTestCaseFactory.Builder` class.
Following is an example that illustrates the usage through `thincrest` library.
[%nowrap,java]
----
public class MetamorphicExample {
@BeforeClass
public static void beforeAll() {
Validator.reconfigure(Validator.Configuration.Builder::enableMetamorphicTesting);
}
@Test
public void testMetamorphicTest2a() {
TestAssertions.assertThat(
1.23,
// Intentionally add 0.0001 to make the implementation incomplete and make the test fail.
MetamorphicTestCaseFactory.forFunctionUnderTest("Math::sin", (Double x) -> Math.sin(x + 0.0001))
.makeInputResolversEndomorphic()
.addInputResolver((x) -> String.format("πー%s", x), x -> Math.PI - x)
.outputOnly()
.proposition("{0}={1}", (Dataset ds) -> Objects.equals(ds.get(0), ds.get(1)))
.toMetamorphicTestPredicate());
}
}
----
The call to `Validator.reconfigure(...)` method optimizes the report readability for the `metamor` package.
Following is a matrix that illustrates how the report looks like.
.Test Failure Report
[cols=">1,<20,<20"]
|===
|Line|Expected|Actual
a|
[%nowrap]
----
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
----
a|
[%nowrap]
----
1.23 ->transform ->io:[[1.23]=>[0.94252222...245537]]
-> [x,πーx] ->input:[1.23, πー1.23]
input:[1.23, πー1.23] -> begin:fut ->(context:fut:input=>io)
[0] 1.23 -> Math::sin(input[0]) ->[1.23]=>[0.94252222]
[1] πー1.23 -> Math::sin(input[1]) ->[1.911592653589]=>[0.942455373446]
-> end:fut ->(context:fut:input=>io)
(context:fut:input=>io) -> output(fut) ->io:[[1.23]=>[0.942522...24553734]]
io:[[1.23]=>[0.942522...24553734]]->check:transform ->true
-> begin:preform ->(context:preform:io=>io)
[2] [1.23]=>[0.942522220991] -> outputOnly(io[0]) ->0.942522220991
[3] [1.911592653589]=>[0.942455373446]-> outputOnly(io[1]) ->0.942455373446
-> end:preform ->(context:preform:io=>io)
(context:preform:io=>io) -> output(preform) ->io:[0.942522220991,...42455373446]
io:[0.9425222209919102,...4465968]-> reduce:out[0]=out[1] ->0.943=0.942
[4] 0.943=0.942 -> check:evaluate ->true
----
a|
[%nowrap]
----
1.23 ->transform ->io:[[1.23]=>[0.9425...245537]]
-> [x,πーx] ->input:[1.23, πー1.23]
input:[1.23, πー1.23] -> begin:fut ->(context:fut:input=>io)
[0] 1.23 -> Math::sin(input[0]) ->[1.23]=>[0.9425222209919102]
[1] πー1.23 -> Math::sin(input[1]) ->[1.9115926535897931]=>[0.942455373446]
-> end:fut ->(context:fut:input=>io)
(context:fut:input=>io) -> output(fut) ->io:[[1.23]=>[0.94252222...2455373446]]
io:[[1.23]=>[0.9425...2455373446]]->check:transform ->false
-> begin:preform ->(context:preform:io=>io)
[2] [1.23]=>[0.94252222] -> outputOnly(io[0]) ->0.942522220991
[3] [1.911592653589]=>[0.942455373446]-> outputOnly(io[1]) ->0.942455373446
-> end:preform ->(context:preform:io=>io)
(context:preform:io=>io) -> output(preform) ->io:[0.942522220991,...42455373446]
io:[0.942522220991,...42455373446]-> reduce:out[0]=out[1] ->0.943=0.942
[4] 0.943=0.942 -> check:evaluate ->false
----
|
a|
.Detail of failure [0]
----
Math::sin(input[0])
----
a|
.Detail of failure [0]
----
in: <1.23>
out:<[1.23]=>[0.9425222209919102]>
----
|
a|
.Detail of failure [1]
----
Math::sin(input[1])
----
a|
.Detail of failure [1]
----
in: <πー1.23>
out:<[1.9115926535897931]=>[0.9424553734465968]>
----
|
a|
.Detail of failure [2]
----
preform:outputOnly(io[0])
----
a|
.Detail of failure [2]
----
in: <[1.23]=>[0.9425222209919102]>
out:<0.9425222209919102>
----
|
a|
.Detail of failure [3]
----
preform:outputOnly(io[1])
----
a|
.Detail of failure [3]
----
in: <[1.9115926535897931]=>[0.9424553734465968]>
out:<0.9424553734465968>
----
|
a|
.Detail of failure [4]
----
evaluate
----
a|
.Detail of failure [4]
----
0.943=0.942
----
|===
© 2015 - 2025 Weber Informatics LLC | Privacy Policy