Skip to content

Commit

Permalink
Merge branch 'feature/304_forward_case_and_step_results' into feature…
Browse files Browse the repository at this point in the history
…/247-sakuli-se
  • Loading branch information
Tobias Schneck committed Apr 22, 2018
2 parents a86d658 + d039b44 commit b5c2d3e
Show file tree
Hide file tree
Showing 179 changed files with 2,979 additions and 1,239 deletions.
2 changes: 1 addition & 1 deletion docs/manual/testdefinition/sakuli-api-application.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ Application Class - Represents an application.

*Params*

* applicationNameOrPath `String` - Path to the application file. *Example:* `C:\Windows\system32\notepad.exe`
* applicationNameOrPath `String` - Path to the application file. Example: `C:\Windows\system32\notepad.exe`
* optResumeOnException `Boolean` - Determines whether to ignore exceptions from this class. If this parameter is
undefined, it will be false.

Expand Down
2 changes: 1 addition & 1 deletion docs/manual/testdefinition/sakuli-api-environment.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -107,7 +107,7 @@ Takes a screenshot of the current screen and add the current timestamp in the fi
* optFormat `string` - optional format, for the screenshot (currently supported: jpg and png)
If null or empty use property `sakuli.screenshot.format`

*Returns*: `String` - file path to the created screenshot OR null on errors
*Returns*: `Path` - to the created screenshot OR null on errors
*Example*
[source,js]
----
Expand Down
26 changes: 20 additions & 6 deletions docs/manual/testdefinition/sakuli-api-testcase.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -12,9 +12,10 @@ TestCase - initializes the Sakuli object and sets the warning and critical time

*Params*

* warningTime `number` - threshold in seconds. If the threshold is set to 0,
* optCaseId `String` - optional ID to set the testcase ID to some specifc value. Default is the folder name.
* optWarningTime `number` - threshold in seconds. If the threshold is not set or is set to 0,
the execution time will never exceed, so the state will be always OK!
* criticalTime `number` - threshold in seconds. If the threshold is set to 0,
* optCriticalTime `number` - threshold in seconds. If the threshold is not set or is set to 0,
the execution time will never exceed, so the state will be always OK!
* optImagePathArray `Array.<String>` - (optional) Path or Array of Paths to the folder containing the image patterns
for these test cases.
Expand All @@ -23,16 +24,20 @@ TestCase - initializes the Sakuli object and sets the warning and critical time
*Example*
[source,js]
----
var testCase = new TestCase(20,30, "path-to/image-folder-name");
//new syntex since v1.2.0
var testCase = new TestCase(["own-case-id", 20, 30, "path-to/image-folder-name"]);
//old syntax < v1.2.0
var testCase = new TestCase(20, 30, ["path-to/image-folder-name"]);
----

*Members*

* link:#TestCase[TestCase]
* link:#TestCase.addImagePaths[TestCase.addImagePaths(imagePaths)]
* link:#TestCase.endOfStep[TestCase.endOfStep(stepName, optWarningTime)]
* link:#TestCase.endOfStep[TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)]
* link:#TestCase.handleException[TestCase.handleException(e)]
* link:#TestCase.saveResult[TestCase.saveResult()]
* link:#TestCase.saveResult[TestCase.saveResult(optForward)]
* link:#TestCase.getID[TestCase.getID()]
* link:#TestCase.getLastURL[TestCase.getLastURL()]
* link:#TestCase.getTestCaseFolderPath[TestCase.getTestCaseFolderPath()]
Expand All @@ -52,7 +57,7 @@ If a relative path is assigned, the current testcase folder will be used as curr


[[TestCase.endOfStep]]
.TestCase.endOfStep(stepName, optWarningTime)
.TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)

A step allows to sub-divide a case to measure logical units, such as "login", "load report" etc. in its
particular runtime. When a case starts, Sakuli starts a "step" timer. It gets read out, stored with the
Expand All @@ -65,6 +70,11 @@ If the step runtime exceeds the step threshold (second parameter, optional), the
* stepName `String`
* optWarningTime `number` - (optional) threshold in seconds, default = 0. If the threshold is set to 0,
the execution time will never exceed, so the state will be always OK!
* optCriticalTime `number` - (optional) threshold in seconds, default = 0. If the threshold is set to 0,
the execution time will never exceed, so the state will be always OK!
* optForward `boolean` - (optional) indicate whether the result of the test step shall be immediately
processed by the enabled forwarders. This means before the test suite has been executed to
the end. If not specified in another way, this option is disabled! Default: `false`


[[TestCase.handleException]]
Expand Down Expand Up @@ -97,6 +107,10 @@ Saves the results of the current test case for further processing.

Should be called in finally-block of the test case:

*Params*

* optForward `boolean` - indicate whether the result of the test case shall be immediately processed by the enabled forwarders. This means before the test suite has been executed to the end. If not specified in another way, this option is disabled! Default: `false`

*Example*
[source,js]
----
Expand Down
4 changes: 2 additions & 2 deletions docs/manual/testdefinition/sakuli-api.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -26,9 +26,9 @@ IMPORTANT: All classes and methods are implemented identical for Java and JavaSc
* link:#Sahi-API[Sahi-API]
* link:#TestCase[TestCase]
** link:#TestCase.addImagePaths[TestCase.addImagePaths(imagePaths)]
** link:#TestCase.endOfStep[TestCase.endOfStep(stepName, optWarningTime)]
** link:#TestCase.endOfStep[TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)]
** link:#TestCase.handleException[TestCase.handleException(e)]
** link:#TestCase.saveResult[TestCase.saveResult()]
** link:#TestCase.saveResult[TestCase.saveResult(optForward)]
** link:#TestCase.getID[TestCase.getID()]
** link:#TestCase.getLastURL[TestCase.getLastURL()]
** link:#TestCase.getTestCaseFolderPath[TestCase.getTestCaseFolderPath()]
Expand Down
34 changes: 24 additions & 10 deletions docs/manual/testdefinition/sakuli-api.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
* [Sahi-API](#Sahi-API)
* [TestCase](#TestCase)
* [TestCase.addImagePaths(imagePaths)](#TestCase.addImagePaths)
* [TestCase.endOfStep(stepName, optWarningTime)](#TestCase.endOfStep)
* [TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)](#TestCase.endOfStep)
* [TestCase.handleException(e)](#TestCase.handleException)
* [TestCase.saveResult()](#TestCase.saveResult)
* [TestCase.saveResult(optForward)](#TestCase.saveResult)
* [TestCase.getID()](#TestCase.getID)
* [TestCase.getLastURL()](#TestCase.getLastURL)
* [TestCase.getTestCaseFolderPath()](#TestCase.getTestCaseFolderPath)
Expand Down Expand Up @@ -123,26 +123,31 @@ TestCase - initializes the Sakuli object and sets the warning and critical time

**Params**

- warningTime `number` - threshold in seconds. If the threshold is set to 0,
- optCaseId `String` - optional ID to set the testcase ID to some specifc value. Default is the folder name.
- optWarningTime `number` - threshold in seconds. If the threshold is not set or is set to 0,
the execution time will never exceed, so the state will be always OK!
- criticalTime `number` - threshold in seconds. If the threshold is set to 0,
- optCriticalTime `number` - threshold in seconds. If the threshold is not set or is set to 0,
the execution time will never exceed, so the state will be always OK!
- optImagePathArray `Array.<String>` - (optional) Path or Array of Paths to the folder containing the image patterns
for these test cases.

**Returns**: - an initialized Sakuli object.
**Example**
```
var testCase = new TestCase(20,30, "path-to/image-folder-name");
//new syntex since v1.2.0
var testCase = new TestCase(["own-case-id", 20, 30, "path-to/image-folder-name"]);
//old syntax < v1.2.0
var testCase = new TestCase(20, 30, ["path-to/image-folder-name"]);
```

**Members**

* [TestCase](#TestCase)
* [TestCase.addImagePaths(imagePaths)](#TestCase.addImagePaths)
* [TestCase.endOfStep(stepName, optWarningTime)](#TestCase.endOfStep)
* [TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)](#TestCase.endOfStep)
* [TestCase.handleException(e)](#TestCase.handleException)
* [TestCase.saveResult()](#TestCase.saveResult)
* [TestCase.saveResult(optForward)](#TestCase.saveResult)
* [TestCase.getID()](#TestCase.getID)
* [TestCase.getLastURL()](#TestCase.getLastURL)
* [TestCase.getTestCaseFolderPath()](#TestCase.getTestCaseFolderPath)
Expand All @@ -159,18 +164,23 @@ If a relative path is assigned, the current testcase folder will be used as curr
- imagePaths `string` - one or more path strings

<a name="TestCase.endOfStep"></a>
##TestCase.endOfStep(stepName, optWarningTime)
##TestCase.endOfStep(stepName, optWarningTime, optCriticalTime, optForward)
A step allows to sub-divide a case to measure logical units, such as "login", "load report" etc. in its
particular runtime. When a case starts, Sakuli starts a "step" timer. It gets read out, stored with the
step name, and the timer will set to `0` each time endOfStep() is called.
If the step runtime exceeds the step threshold (second parameter, optional), the step is saved with state
If the step runtime exceeds the step threshold (second parameter, optional), the step is saved with state
"WARNING" (there is no CRITICAL state).

**Params**

- stepName `String`
- optWarningTime `number` - (optional) threshold in seconds, default = 0. If the threshold is set to 0,
the execution time will never exceed, so the state will be always OK!
- optCriticalTime `number` - (optional) threshold in seconds, default = 0. If the threshold is set to 0,
the execution time will never exceed, so the state will be always OK!
- optForward `boolean` - (optional) indicate whether the result of the test step shall be immediately
processed by the enabled forwarders. This means before the test suite has been executed to
the end. If not specified in another way, this option is disabled! Default: `false`

<a name="TestCase.handleException"></a>
##TestCase.handleException(e)
Expand All @@ -193,11 +203,15 @@ try {
```

<a name="TestCase.saveResult"></a>
##TestCase.saveResult()
##TestCase.saveResult(optForward)
Saves the results of the current test case for further processing.

Should be called in finally-block of the test case:

**Params**

- optForward `boolean` - indicate whether the result of the test case shall be immediately processed by the enabled forwarders. This means before the test suite has been executed to the end. If not specified in another way, this option is disabled! Default: `false`

**Example**
```
try {
Expand Down
28 changes: 25 additions & 3 deletions pom.xml
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,7 @@
<test.group.unit>unit</test.group.unit>
<test.group.ui>ui</test.group.ui>
<test.group.sakuliDSL>sakuli-test</test.group.sakuliDSL>
<dependencies.checker.version>2.5.0</dependencies.checker.version>
</properties>

<!-- Plugin CONFIG -->
Expand Down Expand Up @@ -403,6 +404,16 @@

<!-- general dependencies -->
<dependencies>
<!-- Annotations from the Checker Framework: nullness, interning, locking, ... -->
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
</dependency>
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>jdk8</artifactId>
</dependency>

<!-- Testing frameworks -->
<dependency>
<groupId>org.springframework</groupId>
Expand All @@ -415,7 +426,7 @@
<!--Mockito -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<artifactId>mockito-core</artifactId>
</dependency>
</dependencies>

Expand Down Expand Up @@ -630,8 +641,8 @@
<!--Mockito -->
<dependency>
<groupId>org.mockito</groupId>
<artifactId>mockito-all</artifactId>
<version>1.9.5</version>
<artifactId>mockito-core</artifactId>
<version>2.18.0</version>
<scope>test</scope>
</dependency>
<dependency>
Expand All @@ -649,6 +660,17 @@
<artifactId>gson</artifactId>
<version>2.8.1</version>
</dependency>

<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>checker-qual</artifactId>
<version>${dependencies.checker.version}</version>
</dependency>
<dependency>
<groupId>org.checkerframework</groupId>
<artifactId>jdk8</artifactId>
<version>${dependencies.checker.version}</version>
</dependency>
</dependencies>
</dependencyManagement>
<repositories>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
{% spaceless %}
{% import 'error_message_creator.twig' as errorMessageCreator %}
<<<local>>>$newline$
{% include 'performance_data.twig' %}
{% include 'short_summary.twig' %}
{% include 'detailed_summary.twig' %}
{% include 'performance_data.twig' with { testsuite: testDataEntity}%}
{% include 'short_summary.twig' with { testsuite: testDataEntity}%}
{% include 'detailed_summary.twig' with { testsuite: testDataEntity}%}
$newline$
{% endspaceless %}
Original file line number Diff line number Diff line change
@@ -1,14 +1,10 @@
{% for testCaseId, testCase in testsuite.testCasesAsSortedSet %}
{% if (testCase.state.warning || testCase.state.critical || testCase.state.error) %}
, case "{{ testCase.name }}"$whitespace$
{% if (testCase.state.error) %}
EXCEPTION: {{ errorMessageCreator.exceptionMessageTestCase(testCase) }}
{% else %}
{% if (testCase.state.critical) %}
over runtime ( {{ testCase.duration | number_format(2, '.') }}s/crit at {{ testCase.criticalTime }}s)
{% else %}
over runtime ( {{ testCase.duration | number_format(2, '.') }}s/warn at {{ testCase.warningTime }}s)
{% endif %}
{% endif %}
{% if (testCase.state.warning || testCase.state.critical || testCase.state.error) %}
, case "{{ testCase.id }}"$whitespace$
{% if (testCase.state.error) %}
EXCEPTION: {{ errorMessageCreator.exceptionMessageTestCase(testCase) }}
{% elseif (testCase.state.critical) %}
over runtime ( {{ testCase.duration | number_format(2, '.') }}s/crit at {{ testCase.criticalTime }}s)
{% else %}
over runtime ( {{ testCase.duration | number_format(2, '.') }}s/warn at {{ testCase.warningTime }}s)
{% endif %}
{% endfor %}
{% endif %}
Original file line number Diff line number Diff line change
@@ -1,78 +1,8 @@
{# This template renders the detailed summary for the current test suite. #}
{% include 'detailed_summary_css_styles.twig' %}
<table style="border-collapse: collapse;">
<tr valign="top">
<td class="service{{ getOutputState(testsuite.state) }}">
{{ getOutputState(testsuite.state).shortState }} Sakuli suite "{{ testsuite.id }}"$whitespace$
{% if (testsuite.state.error) %}
{% else %}
{{ testsuite.state.nagiosStateDescription }}$whitespace$
{% endif %}
{% if (testsuite.state.ok) %}
({{ testsuite.duration | number_format(2, '.') }}s)$whitespace$
{% elseif (testsuite.state.name == 'WARNING_IN_SUITE') %}
({{ testsuite.duration | number_format(2, '.') }}s/warn at {{ testsuite.warningTime }}s)$whitespace$
{% elseif (testsuite.state.name == 'CRITICAL_IN_SUITE') %}
({{ testsuite.duration | number_format(2, '.') }}s/crit at {{ testsuite.criticalTime }}s)$whitespace$
{% elseif (testsuite.state.name == 'WARNING_IN_STEP') %}
{% for testCaseId, testCase in testsuite.testCasesAsSortedSet %}
{% if (testCase.state.warningInStep) %}
{% for testStep in testCase.steps %}
{% if (testStep.state.warning) %}
, step "{{ testStep.id }}" over runtime ({{ testStep.duration | number_format(2, '.') }}s/warn at {{ testStep.warningTime }}s)
{% endif %}
{% endfor %}
{% endif %}
{% endfor %}
{% elseif (testsuite.state.name == 'WARNING_IN_CASE') %}
{% for testCaseId, testCase in testsuite.testCasesAsSortedSet %}
{% if (testCase.state.warning) %}
, case "{{ testCase.id }}" over runtime ({{ testCase.duration | number_format(2, '.') }}s/warn at {{ testCase.warningTime }}s)
{% endif %}
{% endfor %}
{% elseif (testsuite.state.name == 'CRITICAL_IN_CASE') %}
{% for testCaseId, testCase in testsuite.testCasesAsSortedSet %}
{% if (testCase.state.critical) %}
, case "{{ testCase.id }}" over runtime ({{ testCase.duration | number_format(2, '.') }}s/crit at {{ testCase.criticalTime }}s)
{% endif %}
{% endfor %}
{% endif %}
{% if (testsuite.state.error) %}
({{ testsuite.duration | number_format(2, '.') }}s)$whitespace$
{{ testsuite.state.nagiosStateDescription }}:$whitespace$
{{ errorMessageCreator.exceptionMessageTestSuite(testsuite) }}.$whitespace$
{% endif %}
. (Last suite run:$whitespace$
{% if (empty(testsuite.stopDate)) %}
xx
{% else %}
{{ testsuite.stopDate|date("dd.MM.YY HH:mm:ss") }}
{% endif %})
{% include 'exception_screenshot.twig' with { testDataEntity: testsuite } %}
</td>
</tr>
{% for testCase in testsuite.testCasesAsSortedSet %}
<tr valign="top">
<td class="service{{ getOutputState(testCase.state) }}">
{{ getOutputState(testCase.state).shortState }} case "{{ testCase.id }}"$whitespace$
{% if (testCase.state.ok) %}
ran in {{ testCase.duration | number_format(2, '.') }}s - {{ testCase.state.nagiosStateDescription }}
{% elseif (testCase.state.warning) %}
over runtime ({{ testCase.duration | number_format(2, '.') }}s/warn at {{ testCase.warningTime }}s)
{% include 'step_information.twig' with { testCase: testCase } %}$whitespace$
{% elseif (testCase.state.critical) %}
{% set errorMessage = errorMessageCreator.exceptionMessageTestCase(testCase) %}
{% if (empty(errorMessage)) %}
over runtime ({{ testCase.duration | number_format(2, '.') }}s/crit at {{ testCase.criticalTime }}s)
{% else %}
EXCEPTION: {{ errorMessageCreator.exceptionMessageTestCase(testCase) }}
{% include 'exception_screenshot.twig' with { testDataEntity: testCase } %}
{% for testStep in testCase.stepsAsSortedSet %}
{% include 'exception_screenshot.twig' with { testDataEntity: testStep } %}
{% endfor %}
{% endif %}
{% endif %}
</td>
</tr>
{% endfor %}
</table>
{% set testDataEntityType = getTestDataEntityType(testDataEntity) %}
{% if ('TestSuite' == testDataEntityType) %}
{% include 'detailed_summary_suite.twig' with { testsuite: testDataEntity} %}
{% elseif ('TestCase' == testDataEntityType) %}
{% include 'detailed_summary_case.twig' with { testCase: testDataEntity } %}
{% elseif ('TestCaseStep' == testDataEntityType) %}
{% include 'detailed_summary_step.twig' with { testStep: testDataEntity } %}
{% endif %}
Loading

0 comments on commit b5c2d3e

Please sign in to comment.