Add functional smoke tests and automated testing workflow (#336)

* build out structure for initial smoke test

* get latest created GWSBaselineConformance dir

* add capability to recursively check if paths are file/directory

* add helper function to determine if all outputs exist

* refactor into smoke_test_utils

* cleanup, add run_smoke_test workflow

* build out initial workflow for smoke testing scubagoggles from a windows os

* add workflow_dispatch

* add pytest install

* run test

* change secret creds

* change secret creds

* change secret creds

* move location of cache step; add shell=True to test

* debug smoke_test.py

* adjust smoke_test workflow

* smoke_test workflow updates

* smoke_test workflow updates

* adjust smoke_test workflow

* smoke_test workflow updates

* smoke_test workflow updates

* smoke_test workflow updates

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust smoke_test workflow

* adjust pip3 -> pip

* update numpy

* update secrets, create credentials.json for service account

* add cache dependency path, not working yet; readd numpy line to prevent error

* remove utf-8 encoding

* refactor try/catch handlers into smoke test methods; remove ProviderSettingsExport and switch to ScubaResults.json

* workflow updates

* create custom action for setting up repo and python version/cache

* make macos smoke test depend on windows job

* adjust local composite action

* adjust workflow

* adjust workflow

* modify path structure for initialize-smoke-test action

* modify path structure for initialize-smoke-test action

* modify path structure for initialize-smoke-test action

* rename action to initialize-scubagoggles; move majority of steps out of workflow into action for reuse

* action modifications

* action modifications

* action changes

* create separate subactions for windows/mac setup

* fix location of shell property

* workflow adjustment

* workflow adjustment

* rename to setup python

* add -y to bypass confirmation

* try removing shell from actions and keep in workflow

* trigger workflow

* readd shell param

* fix credentials.json for macos job

* test macos

* comment pytest in mac run so cache completes

* see if credentials.json is created

* try converting workflow to matrix

* try converting workflow to matrix

* try converting workflow to matrix

* test matrix version

* lets see if refactoring into separate workflow changes things

* retry conditionals

* commit

* commit

* commit

* fairly major change to structure.. should be clear on which action to navigate to whether its windows/mac

* didnt add input in macos dependencies action

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* macos credentials.json update

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* try different approach for creating credentials.json

* switching to environment secrets.. issue is with repo secrets

* switching to environment secrets.. issue is with repo secrets

* switching to environment secrets.. issue is with repo secrets

* cleanup workflows/actions; add check for valid json; add types

* added test_scubagoggles_report pytest with selenium; testing BaselineReports.html successfully so far

* added test_scubagoggles_report pytest with selenium; testing BaselineReports.html successfully so far

* added test_scubagoggles_report pytest with selenium; testing BaselineReports.html successfully so far

* added test_scubagoggles_report pytest with selenium; testing BaselineReports.html successfully so far

* added test_scubagoggles_report pytest with selenium; testing BaselineReports.html successfully so far

* best practice improvements for how file protocol is handled before invoking selenium; adding caching for both windows/macos deps

* best practice improvements for how file protocol is handled before invoking selenium; adding caching for both windows/macos deps

* test python v3.9, bug fix

* readability improvements

* expand selenium testing of scubagoggles reports, checks parent/individual reports

* refactor smoke_test.py; add run_selenium method for handling report testing

* conditionally uninstall numpy to see if 3.8.10 passes smoke test; some additional refactoring/code improvements

* conditionally uninstall numpy to see if 3.8.10 passes smoke test; some additional refactoring/code improvements

* try simplifying types to see if 3.8.10 passes

* finish testing scubaresults for errors

* encapsulate selenium driver setup into a class

* remove 3.8 from workflow; convert some static strings to const

* improve error handling when verifying scubaresults.json

* start addressing pylint warnings for selenium_browser.py

* address pylint warnings across selenium_browser.py, smoke_test.py, smoke_test_utils.py

* addressing more of pylint warnings

* address some path issues with importing orchestrator methods

* move pytest.ini to root

* continue to address pylint errors, most should be resolved now

* address final pylinter errors, add get_package_version() method in orchestrator/utils

* address last pylinter error

.github/actions/setup-dependencies-macos/action.yml

@@ -0,0 +1,36 @@
+name: Setup Dependencies (macOS)
+inputs:
+  operating-system:
+    required: true
+    default: "macos"
+  opa-version:
+    required: true
+    default: "0.60.0"
+  python-version:
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Setup virtualenv
+      shell: bash
+      run: |
+        pip install virtualenv
+        virtualenv -p python .venv
+        source .venv/bin/activate
+    
+    - name: Install dependencies
+      shell: bash
+      run: |
+        python -m pip install .
+        pip install -r requirements.txt
+        pip install pytest
+        pip install selenium
+        pip uninstall -y numpy
+        pip install numpy==1.26.4
+    
+    - name: Download OPA executable
+      shell: bash
+      run: |
+        python download_opa.py -v ${{ inputs.opa-version }} -os ${{ inputs.operating-system }}
+        chmod +x opa_darwin_amd64

.github/actions/setup-dependencies-windows/action.yml

@@ -0,0 +1,41 @@
+name: Setup Dependencies (Windows)
+inputs:
+  operating-system:
+    required: true
+    default: "windows"
+  opa-version:
+    required: true
+    default: "0.60.0"
+  python-version:
+    required: true
+
+runs:
+  using: "composite"
+  steps:
+    - name: Setup virtualenv
+      shell: powershell
+      run: |
+        pip install virtualenv
+        python -m venv .venv
+        .venv\Scripts\activate
+    
+    - name: Install dependencies
+      shell: powershell
+      run: |
+        python -m pip install .
+        pip install -r requirements.txt
+        pip install pytest
+        pip install selenium
+        pip uninstall -y numpy
+        pip install numpy==1.26.4
+        
+        # Below python v3.9, a lower numpy v1.24.4 is used
+        #$pythonVersion = [version]${{ inputs.python-version }}
+        #if ($pythonVersion -ge [version]"3.8.18") {
+        #  pip uninstall -y numpy
+        #  pip install numpy==1.26.4
+        #} 
+    
+    - name: Download OPA executable
+      shell: powershell
+      run: python download_opa.py -v ${{ inputs.opa-version }} -os ${{ inputs.operating-system }}

.github/workflows/pylint.yml

@@ -31,7 +31,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install -r requirements.txt
-        pip install pylint
+        pip install pylint pytest selenium
     - name: Analysing the code with pylint
       run: |
         pylint -d R0913,R0914,R0915,R1702,W0718,W0719,R0801 $(git ls-files '*.py')

.github/workflows/run_smoke_test.yml

@@ -0,0 +1,67 @@
+name: Run Smoke Test 
+on: 
+  workflow_call:
+  workflow_dispatch:
+  pull_request:
+    types: [opened, reopened, synchronize]
+    branches:
+      - "main"
+  pull_request_review:
+    types: [submitted]
+  push:
+    # Uncomment when testing locally
+    #paths:
+    #  - ".github/workflows/run_smoke_test.yml"
+    #  - ".github/actions/setup-dependencies-windows/action.yml"
+    #  - ".github/actions/setup-dependencies-macos/action.yml"
+    branches:
+      - "main"
+      #- "*smoke*"
+
+jobs:
+  smoke-test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [windows-latest, macos-latest]
+        # See https://raw.githubusercontent.com/actions/python-versions/main/versions-manifest.json,
+        # ctrl + f and search "python-3.<minor>.<patch>-<darwin-arm64/win32/linux>" for supported versions
+        python-version: ["3.9", "3.12"] # "3.8 fails with numpy uninstall"
+    runs-on: ${{ matrix.os }}
+    environment: Development
+    steps:
+      - name: Checkout Repository
+        uses: actions/checkout@v4
+
+      - name: Setup Python v${{ matrix.python-version }}
+        uses: actions/setup-python@v5
+        with: 
+          python-version: ${{ matrix.python-version }}
+          cache: "pip"
+          cache-dependency-path: "requirements.txt"
+
+      - name: Setup Dependencies (Windows)
+        if: ${{ matrix.os == 'windows-latest' }}
+        uses: ./.github/actions/setup-dependencies-windows
+        with:
+          operating-system: "windows"
+          opa-version: "0.60.0"
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup Dependencies (macOS)
+        if: ${{ matrix.os == 'macos-latest' }}
+        uses: ./.github/actions/setup-dependencies-macos
+        with:
+          operating-system: "macos"
+          opa-version: "0.60.0"
+          python-version: ${{ matrix.python-version }}
+
+      - name: Setup credentials for service account
+        id: create-json
+        uses: jsdaniell/[email protected]
+        with: 
+          name: "credentials.json"
+          json: ${{ secrets.GWS_GITHUB_AUTOMATION_CREDS }}
+
+      - name: Run ScubaGoggles and check for correct output
+        run: pytest -s -vvv ./Testing/Functional/SmokeTests/ --subjectemail="${{ secrets.GWS_SUBJECT_EMAIL }}" --domain="${{ secrets.GWS_DOMAIN }}"

Testing/Functional/SmokeTests/selenium_browser.py

@@ -0,0 +1,75 @@
+"""
+selenium_browser.py declares a Browser class for use in ScubaGoggles testing.    
+"""
+
+from selenium import webdriver
+from selenium.webdriver.chrome.options import Options
+
+class Browser:
+    """
+    The Browser class encapsulates the setup, usage, and teardown of a
+    Selenium WebDriver instance for automated browser interactions.
+    """
+    def __init__(self):
+        chrome_options = Options()
+        chrome_options.add_argument("--headless")
+        chrome_options.add_argument("--no-sandbox")
+        chrome_options.add_argument("--disable-dev-shm-usage")
+
+        self.driver = webdriver.Chrome(options=chrome_options)
+
+    def get(self, url):
+        """
+        Load a new web page in the current browser window.
+
+        Args:
+            url: The URL to load. Must be a fully qualified URL
+        """
+        self.driver.get(url)
+
+    def quit(self):
+        """
+        Quits the driver, closing every associated window. 
+        """
+        self.driver.quit()
+
+    def find_element(self, by, value):
+        """
+        Find the first WebElement using the given method.
+        This method is affected by the 'implicit wait' times in force at the time and execution.
+        The find_element(...) invocation will return a matching row, or try again repeatedly
+        until the configured timeout is reached.
+
+        Args:
+            by: The locating mechanism to use, i.e. By.CLASS_NAME, By.TAG_NAME
+            value: The locator, i.e. "h1", "header"
+
+        Returns:
+            WebElement: The first matching element on the current page
+        """
+        return self.driver.find_element(by, value)
+
+    def find_elements(self, by, value):
+        """
+        Find all WebElements within the current page.
+        This method is affected by the 'implicit wait' times in force at the time and execution.
+        The find_elements(...) invocation will return as soon as there are more than 0 items in 
+        the found collection, or will return an empty list of the timeout is reached.
+
+        Args:
+            by: The locating mechanism to use, i.e. By.CLASS_NAME, By.TAG_NAME
+            value: The locator, i.e. "h1", "header"
+
+        Returns:
+            WebElement: A list of all matching WebElements, or an empty list if nothing matches
+        """
+        return self.driver.find_elements(by, value)
+
+    def current_url(self):
+        """
+        Get a string representing the current URL that the browser is looking at.
+
+        Returns:
+            The URL of the page currently loaded in the browser
+        """
+        return self.driver.current_url

Testing/Functional/SmokeTests/smoke_test.py

@@ -0,0 +1,72 @@
+"""
+smoke_test.py declares a SmokeTest class for ScubaGoggles automation testing.
+"""
+
+import subprocess
+import os
+import pytest
+from smoke_test_utils import (
+    get_output_path,
+    prepend_file_protocol,
+    get_required_entries,
+    verify_all_outputs_exist,
+    verify_output_type,
+    run_selenium,
+    verify_scubaresults,
+)
+
+SAMPLE_REPORT = "sample-report"
+SCUBA_RESULTS = "ScubaResults.json"
+BASELINE_REPORTS = "BaselineReports.html"
+
+class SmokeTest:
+    """
+    Pytest class to encapsulate the following test cases:
+
+    - Generate the correct output files (BaselineReports.html, ScubaResults.json, etc)
+    - Check the content of html files, verify href attributes are correct, etc
+    - Check if ScubaResults.json contains errors in the summary. If errors exist, then
+      either API calls or functions produced exceptions which need to be handled
+    """
+    def test_scubagoggles_output(self, subjectemail):
+        """
+        Test if the `scubagoggles gws` command generates correct output for all baselines.
+        
+        Args:
+            subjectemail: The email address of a user for the service account
+        """
+        try:
+            command: str = f"scubagoggles gws --subjectemail {subjectemail} --quiet"
+            subprocess.run(command, shell=True, check=True)
+            output_path: str = get_output_path()
+            output: list = verify_output_type(output_path, [])
+            required_entries = get_required_entries(os.path.join(os.getcwd(), SAMPLE_REPORT), [])
+            verify_all_outputs_exist(output, required_entries)
+        except (OSError, ValueError, Exception) as e:
+            pytest.fail(f"An error occurred, {e}")
+
+    def test_scubaresults(self):
+        """
+        Determine if ScubaResults.json contains API errors or exceptions.
+        """
+        try:
+            output_path: str = get_output_path()
+            scubaresults_path: str = os.path.join(output_path, SCUBA_RESULTS)
+            with open(scubaresults_path, encoding="utf-8") as jsonfile:
+                verify_scubaresults(jsonfile)
+        except (ValueError, Exception) as e:
+            pytest.fail(f"An error occurred, {e}")
+
+    def test_scubagoggles_report(self, browser, domain):
+        """
+        Test if the generated baseline reports are correct,
+        i.e. BaselineReports.html, CalendarReport.html, ChatReport.html
+        """
+        try:
+            output_path: str = get_output_path()
+            report_path: str = prepend_file_protocol(os.path.join(output_path, BASELINE_REPORTS))
+            browser.get(report_path)
+            run_selenium(browser, domain)
+        except (ValueError, AssertionError, Exception) as e:
+            browser.quit()
+            pytest.fail(f"An error occurred, {e}")