From d9f1786b5bac86a42bb33ccb2a1afe72d7106193 Mon Sep 17 00:00:00 2001 From: Ken Wills Date: Tue, 28 May 2024 07:57:13 -0500 Subject: [PATCH 1/4] Add initial test plan guide --- design-doc-template.adoc | 17 +++++++++++++++++ 1 file changed, 17 insertions(+) diff --git a/design-doc-template.adoc b/design-doc-template.adoc index 4156a0aa..4d12a9f6 100644 --- a/design-doc-template.adoc +++ b/design-doc-template.adoc @@ -112,6 +112,23 @@ or a confirmation that there are no security implications to consider. == Test Plan +Depending on the selected stability level, the appropriate section below should be completed, including a brief description of how testing is to be performed in accordance with the selected stability level. The non-relevant sections may be removed as needed. +//// +Depending on the stability level, the test plan required may vary. see below: +//// + +** Experimental - no test plan is required. Unit / integration tests may still be added as needed during development. + +** Preview - a brief high-level description of the testing approach should be added here, including types of tests added (unit, integration, smoke, component, subsystem, etc.) Note that not all test types are required for a particular feature, so include a description of what is being tested and the approach chosen to perform the testing. + +** Community - this level should include everything in the 'Preview' stability level, plus the following additional testing as relevant: +*** Manual tests: for features that may require manual testing (such as user interface changes etc), provide a description of what was tested and any methods employed (browsers, CLI clients etc.) If manual testing is required, the person performing the manual tests should be identified here. +*** Miscellaneous checks: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. +*** Integration tests - at the 'Community' stability level, complete integration tests should be provided. +*** Compatibility tests - if backwards compatibility is relevant to the feature, then describe how the testing is performed. + +** Default - This stability level is reserved for a more formal process, which may be found [here] // note add a link or remove this? + == Community Documentation //// Generally a feature should have documentation as part of the PR to wildfly master, or as a follow up PR if the feature is in wildfly-core. In some cases though the documentation belongs more in a component, or does not need any documentation. Indicate which of these will happen. From e762fb9d8dc5de33808fe57b61da80eacfa3e9c8 Mon Sep 17 00:00:00 2001 From: Ken Wills Date: Tue, 23 Jul 2024 06:25:24 -0500 Subject: [PATCH 2/4] Update with feedback from discussions --- design-doc-template.adoc | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/design-doc-template.adoc b/design-doc-template.adoc index 4d12a9f6..ea4e8cbd 100644 --- a/design-doc-template.adoc +++ b/design-doc-template.adoc @@ -122,12 +122,12 @@ Depending on the stability level, the test plan required may vary. see below: ** Preview - a brief high-level description of the testing approach should be added here, including types of tests added (unit, integration, smoke, component, subsystem, etc.) Note that not all test types are required for a particular feature, so include a description of what is being tested and the approach chosen to perform the testing. ** Community - this level should include everything in the 'Preview' stability level, plus the following additional testing as relevant: -*** Manual tests: for features that may require manual testing (such as user interface changes etc), provide a description of what was tested and any methods employed (browsers, CLI clients etc.) If manual testing is required, the person performing the manual tests should be identified here. -*** Miscellaneous checks: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. +*** Manual tests: briefly describe checks to be performed during one-time exploratory testing. The purpose of this testing is to check corner cases and other cases that are not worth implementing as automated tests. Typical checks are: bad configurations are easy to reveal, attribute descriptions and error messages are clear, names are descriptive and consistent with similar resources, default values are reasonable. +*** Unit / Integration tests: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. Fully qualified test case names should be provided along with a brief description of what the test is doing. *** Integration tests - at the 'Community' stability level, complete integration tests should be provided. *** Compatibility tests - if backwards compatibility is relevant to the feature, then describe how the testing is performed. -** Default - This stability level is reserved for a more formal process, which may be found [here] // note add a link or remove this? +** Default - This stability level is reserved and requires approval by a professional Quality Engineer with subject matter expertise. == Community Documentation //// From d029e96b318acafad162bb8a34cd61e3dd07226e Mon Sep 17 00:00:00 2001 From: Ken Wills Date: Tue, 23 Jul 2024 06:40:11 -0500 Subject: [PATCH 3/4] Unit / Integation -> Misc checks --- design-doc-template.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-doc-template.adoc b/design-doc-template.adoc index ea4e8cbd..f01f31db 100644 --- a/design-doc-template.adoc +++ b/design-doc-template.adoc @@ -123,7 +123,7 @@ Depending on the stability level, the test plan required may vary. see below: ** Community - this level should include everything in the 'Preview' stability level, plus the following additional testing as relevant: *** Manual tests: briefly describe checks to be performed during one-time exploratory testing. The purpose of this testing is to check corner cases and other cases that are not worth implementing as automated tests. Typical checks are: bad configurations are easy to reveal, attribute descriptions and error messages are clear, names are descriptive and consistent with similar resources, default values are reasonable. -*** Unit / Integration tests: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. Fully qualified test case names should be provided along with a brief description of what the test is doing. +*** Miscellaneous checks: Manual checks for significant changes in server performance, memory and disk footprint should be described here. These checks are not always relevant, but consideration of these impacts, and others, are strongly encouraged and should be described here. Fully qualified test case names should be provided along with a brief description of what the test is doing. *** Integration tests - at the 'Community' stability level, complete integration tests should be provided. *** Compatibility tests - if backwards compatibility is relevant to the feature, then describe how the testing is performed. From 7f9b6e174ef4185581a11c17d3ef0f826368b674 Mon Sep 17 00:00:00 2001 From: Ken Wills Date: Tue, 17 Sep 2024 15:35:25 -0500 Subject: [PATCH 4/4] minor updates --- design-doc-template.adoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/design-doc-template.adoc b/design-doc-template.adoc index f01f31db..7b1d847b 100644 --- a/design-doc-template.adoc +++ b/design-doc-template.adoc @@ -117,7 +117,7 @@ Depending on the selected stability level, the appropriate section below should Depending on the stability level, the test plan required may vary. see below: //// -** Experimental - no test plan is required. Unit / integration tests may still be added as needed during development. +** Experimental - no test plan is required. Basic unit / integration tests should be added during development. ** Preview - a brief high-level description of the testing approach should be added here, including types of tests added (unit, integration, smoke, component, subsystem, etc.) Note that not all test types are required for a particular feature, so include a description of what is being tested and the approach chosen to perform the testing.