docs: Userguide enhancements (#890)

This PR updates the doc for how_to_use section and add the documentation for requirement tests. ref: https://splunk.atlassian.net/browse/ADDON-73384
splunk · Sep 26, 2024 · 5cf1bd4 · 5cf1bd4
1 parent a968cd1
commit 5cf1bd4
Show file tree

Hide file tree

Showing 5 changed files with 226 additions and 20 deletions.
diff --git a/docs/how_to_use.md b/docs/how_to_use.md
@@ -1,7 +1,7 @@
 # How To Use
 
 
-Create a test file in the tests folder
+1. Create a test file in the tests folder
 
 ```python
 from pytest_splunk_addon.standard_lib.addon_basic import Basic
@@ -10,6 +10,10 @@ class Test_App(Basic):
         pass
 ```
 
+2. Create sample files and pytest-splunk-addon-data.conf (Can be skipped if the events are already present in the splunk environment)
+
+- For detailed information on how to create samples and pytest-splunk-addon-data.conf, please refer [here](./sample_generator.md).
+
 ## Test Execution
 
 There are three ways to execute the tests:
@@ -34,7 +38,6 @@ There are three ways to execute the tests:
  ARG SPLUNK_APP_ID=TA_UNKNOWN
  ARG SPLUNK_APP_PACKAGE=$SPLUNK_APP_PACKAGE
  RUN echo Splunk VERSION=$SPLUNK_VERSION
- COPY deps/apps /opt/splunk/etc/apps/
  COPY $SPLUNK_APP_PACKAGE /opt/splunk/etc/apps/$SPLUNK_APP_ID
 ```
 
@@ -225,7 +228,7 @@ The tool assumes the Splunk Add-on is located in a folder "package" in the proje
 
 ______________________________________________________________________
 
-There are 3 types of tests included in pytest-splunk-addon are:
+There are 4 types of tests included in pytest-splunk-addon are:
 
  1. To generate test cases only for knowledge objects, append the following marker to pytest command:
 
@@ -244,9 +247,15 @@ There are 3 types of tests included in pytest-splunk-addon are:
      ```console
      -m  splunk_indextime --splunk-data-generator=<Path to the conf file>
      ```
-    
+
      For detailed information on index time test execution, please refer [here](./index_time_tests.md).
 
+ 4. To generate only requirement tests, append the folowing marker to pytest command:
+
+     ```console
+     -m splunk_requirements
+     ```
+
  - To execute all the searchtime tests together, i.e both Knowledge objects and CIM compatibility tests,
    append the following marker to the pytest command:
 
@@ -317,7 +326,7 @@ The following optional arguments are available to modify the default settings in
 6. Options to separate event generation, event ingestion and test execution stage
     :
 
-    ```console
+      ```console
       --tokenized-event-source=new|store_new|pregenerated
       ```
 
@@ -438,6 +447,6 @@ def splunk_setup(splunk):
 
  How can this be achieved :
 
-  - Make json representation of the data models, which satisfies this [DataModelSchema](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/standard_lib/cim_tests/DatamodelSchema.json).
+  - Make json representation of the data models, which satisfies this [DataModelSchema](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/cim_tests/DatamodelSchema.json).
   - Provide the path to the directory having all the data models by adding `--splunk_dm_path path_to_dir` to the pytest command
-  - The test cases will now be generated for the data models provided to the plugin and not for the [default data models](https://github.com/splunk/pytest-splunk-addon/tree/main/pytest_splunk_addon/standard_lib/data_models).
+  - The test cases will now be generated for the data models provided to the plugin and not for the [default data models](https://github.com/splunk/pytest-splunk-addon/tree/main/pytest_splunk_addon/data_models).
diff --git a/docs/index.md b/docs/index.md
@@ -1,28 +1,41 @@
 # Overview
 
-pytest-splunk-addon is an open-source dynamic test plugin for Splunk Apps and Add-ons
-which allows the user to test knowledge objects, CIM compatibility and index time properties.
+pytest-splunk-addon is an open-source dynamic test plugin for Splunk [Apps](https://docs.splunk.com/Splexicon:App) and [Add-ons](https://docs.splunk.com/Splexicon:Addon)
+which allows the user to test [knowledge objects](https://docs.splunk.com/Splexicon:Knowledgeobject), [CIM](https://docs.splunk.com/Splexicon:CommonInformationModel) compatibility and [index time properties](https://docs.splunk.com/Splexicon:Indexedfield).
+
+## Prerequisites
+
+- Splunk App or Add-on package
+- Splunk instance with App or Add-on installed (Not required if using Docker)
+- Docker (Not required if using external Splunk instance)
 
 ## Support
 
 - **Python**: 3.7
 - **Platforms**: Linux, Windows and MacOS
 
-## Features
-
-- Generate tests for Splunk Knowledge objects in Splunk Technology Add-ons.
-- Generate tests for checking CIM compatibility in Splunk Technology Add-ons.
-- Generate tests for checking Splunk index-time properties in Splunk Technology Add-ons.
-- Validate your add-ons using Splunk + Docker.
-
-## Release notes
-
-Find details about all the releases [here](https://github.com/splunk/pytest-splunk-addon/releases).
-
 ## Installation
 
 pytest-splunk-addon can be installed via pip from PyPI:
 
 ```console
 pip3 install pytest-splunk-addon
 ```
+
+## Features
+
+The pytest-splunk-addon works by dynamically generating different types of tests for Splunk apps and add-ons by parsing their configuration files. Specifically, it looks at the .conf files in the provided Splunk app/add-on to create:
+
+1. [Knowledge Object Tests](./field_tests.md)
+
+2. [CIM Compatibility Tests](./cim_tests.md)
+
+3. [Index-Time Field Tests](./index_time_tests.md)
+
+4. [Requirement Tests](./requirement_tests.md)
+
+Refer to the [How to use](./how_to_use.md) section for detailed instructions on running the tests.
+
+## Release notes
+
+Find details about all the releases [here](https://github.com/splunk/pytest-splunk-addon/releases).
diff --git a/docs/requirement_tests.md b/docs/requirement_tests.md
@@ -0,0 +1,93 @@
+# Requirement Tests
+
+## Overview
+
+- The tests are written with a purpose of testing the proper functioning of the fields extractions of the add-on.
+
+______________________________________________________________________
+
+To generate only requirement tests, append the folowing marker to pytest command:
+
+ ```console
+ -m  splunk_requirements
+ ```
+
+## Test Scenarios
+
+**1. Fields should be extracted as defined in the cim_fields and other_mappings.**
+
+ ```python
+ test_requirements_fields[sample_name::<stanza>::host::<host>]
+ ```
+
+ Testcase verifies that fields are getting extracted with the values defined in the cim_fields and other_mappings in sample xml.
+ Here &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and
+ &lt;host&gt; is the host of the event
+
+ **Workflow:**
+
+ - Plugin gets the list of events and the fields defined under cim_fields and other_mappings for each event
+ - For each event, plugin generates a search query to retrieve all the fields extracted by splunk.
+ - For each event, all the fields defined in the cim_fields and other_mappings should be extracted with the values defined in sample xml.
+
+**2. Event should be mapped with the datamodel defined in the sample xml**
+
+ ```python
+ test_datamodels[<datamodel>::sample_name::<stanza>::host::<host>]
+ ```
+
+ Testcase verifies that the event is mapped with the datamodel defined in the sample xml.
+ Here &lt;datamodel&gt; is the name of the datamodel that the event should be mapped with, &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and &lt;host&gt; is the host of the event
+
+ **Workflow:**
+
+ - Plugin gets the events and its expected datamodel from the sample xml
+ - Plugin generates a search query for that event and validates if the values of tags field are according to the datamodel defined in sample xml.
+
+**3. All the cim recommended field should be extracted**
+
+ ```python
+ test_cim_fields_recommended[<datamodel>::sample_name::<stanza>::host::<host>]
+ ```
+
+ Testcase verifies that all the cim recommended fields of mapped datamodel are extracted for the event defined in the sample xml.
+ Here &lt;datamodel&gt; is the name of the datamodel that the event should be mapped with, &lt;stanza&gt; is the stanza name defined in the pytest-splunk-addon-data.conf and &lt;host&gt; is the host of the event
+
+ **Workflow:**
+
+ - For each events, plugin gets the list of fields defined under cim_fields and missing_recommended_fields.
+ - Field list retrieved from the sample xml should be equal to the list of recommended fields defined in the datamodel schema.
+
+## Testcase Troubleshooting
+
+In the case of test-case failure check if:
+
+ - The add-on to be tested is installed on the Splunk instance.
+ - Splunk licence has not expired.
+ - Splunk instance is up and running.
+ - Splunk instance's management port is accessible from the test machine.
+
+If all the above conditions are satisfied, further analysis of the test is required.
+For every test case failure, there is a defined structure for the stack trace.
+
+ ```text
+ - test_requirements_fields
+ AssertionError: <<error_message>>
+     Field | Splunk Value | Expected Value
+     ------| ------------ | --------------
+      str  |      str     |      str
+ Search =  <Query>
+
+ - test_datamodels
+ AssertionError: <<error_message>>
+     Expected datamodel | Expected tags | Found datamodel | Found tags
+     -------------------| ------------- | --------------- | ----------
+            str         |      str      |       str       |    str
+ Search =  <Query>
+
+ - test_cim_fields_recommended
+ AssertionError: <<error_message>>
+ Search =  <Query>
+ ```
+
+Get the search query from the stack trace and execute it on the Splunk instance and verify which specific event is causing failure.
diff --git a/docs/sample_generator.md b/docs/sample_generator.md
@@ -5,6 +5,90 @@ The sample generation & ingestion takes place before executing the testcases.
 For index-time test cases, there are multiple metadata required about the sample file for which `pytest-splunk-addon-data.conf` must be created and provided to the pytest command.
 
 
+## Generate sample files
+Pytest splunk addon supports 2 different formats of sample files
+
+**1. Text format**
+
+- Sample events can be generated from the live events by replacing the field values with the tokens. Token name should be enclosed with ##.
+- Token replacement setting for this samples should be added to pytest-splunk-addon-data.conf to generate events
+- For example,
+     ```
+     Example live event:
+     <111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1
+
+     Sample created from live event:
+     <111> ##timestamp_token## ##src_ip_token## RT_FLOW: RT_FLOW_SESSION_CREATE: action=##action_token## transport=##transport_type_token## dest=##dest_ip_token## dest_ip=##dest_ip_token## dest_port=##dest_port_token## dest_zone=##dest_zone_token## app=##app_token##
+     ```
+
+**2. XML format**
+
+- Generate sample xml file using the format provided [here](https://github.com/splunk/pytest-splunk-addon/blob/main/pytest_splunk_addon/sample_generation/schema.xsd)
+- Using xml format for the sample events, user can also execute requirement tests for the event for fields mentioned in the cim_fields and other_mappings
+- For example,
+
+```
+Example live event:
+<111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1
+```
+
+<details>
+<summary>Sample xml created from live event:</summary>
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<device>
+     <vendor>Test</vendor>
+     <product>Test product</product>
+     <version id="8.21.0"/>
+     <event code="" name="SyslogEvent" format="syslog" >
+          <transport type="syslog" host="10.0.0.3"/>
+          <source>
+               <comment/>
+          </source>
+          <raw>
+               <![CDATA[<111> 2020-02-12T03:27:09+10:00 10.0.0.3 RT_FLOW: RT_FLOW_SESSION_CREATE: action=allowed transport=ICMP dest=10.0.0.1 dest_ip=10.0.0.1 dest_port=5048 dest_zone=DUMMY_ZONE app=app1]]>
+          </raw>
+          <cim>
+               <models>
+                    <model>Network Traffic</model>
+               </models>
+               <cim_fields>
+                    <field name="action" value="allowed"/>
+                    <field name="dest" value="10.0.0.1"/>
+                    <field name="dest_ip" value="10.0.0.1"/>
+                    <field name="dest_port" value="5048"/>
+                    <field name="dest_zone" value="DUMMY_ZONE"/>
+                    <field name="app" value="app1"/>
+                    <field name="transport" value="ICMP"/>
+               </cim_fields>
+               <missing_recommended_fields>
+                    <field>bytes</field>
+                    <field>bytes_in</field>
+                    <field>bytes_out</field>
+                    <field>dvc</field>
+                    <field>rule</field>
+                    <field>session_id</field>
+                    <field>src</field>
+                    <field>src_ip</field>
+                    <field>src_port</field>
+                    <field>src_zone</field>
+                    <field>src_interface</field>
+                    <field>user</field>
+                    <field>protocol</field>
+                    <field>vendor_product</field>
+               </missing_recommended_fields>
+               <exceptions>
+                    <field name="vendor_product" value="Incorrect vendor product" reason="testing exceptions"/>
+               </exceptions>
+          </cim>
+          <other_mappings>
+               <field name="event_name" value="RT_FLOW_SESSION_CREATE">
+          </other_mappings>
+     </event>
+</device>
+```
+</details>
+
 ## pytest-splunk-addon-data.conf.spec
 
 **Default Values**:
@@ -18,6 +102,7 @@ sourcetype = pytest-splunk-addon
 source = pytest-splunk-addon:{{input_type}}
 sourcetype_to_search = {{sourcetype}}
 sample_count = 1
+requirement_test_sample = 1
 timestamp_type = event
 count = 0
 earliest = now
@@ -78,6 +163,11 @@ host_prefix = {{host_prefix}}
 - This parameter will be used to calculate the total number of events which will be generated from the sample file.
 - If `input_type = modinput`, do not provide this parameter.
 
+**requirement_test_sample = 1**
+
+- This parameter is used to run requirement tests for the provided sample xml file
+- only supported with the xml sample file
+
 **expected_event_count = <count\>**
 
 - The no. of events this sample stanza should generate.

diff --git a/mkdocs.yml b/mkdocs.yml
@@ -63,6 +63,7 @@ nav:
   - CIM Compliance Report: "cim_compliance.md"
   - Knowledge Object Tests: "field_tests.md"
   - Index Time Tests: "index_time_tests.md"
+  - Requirement Tests: "requirement_tests.md"
   - Data Generator: "sample_generator.md"
   - API Documentation:
       - API Overview: "api_reference/api_reference.md"