Add config file support for existing parameters (#413)

* Add pyyaml to requirements

* Add basic config file support

* Update Parameters.md with config file parameter

* Expand help string for config parameter

* Create Config.md file

* Update README.md with link to config file docs

* Document config file support

* Remove note about config not supporting aliases

* Correct link syntax

* minor grammatical tweak

* Add navigation section

* Reformat notes to make clear not part of any one example

* Pylint corrections

* Refactor ScubaConfig glass into ScubaArgumentParser

* linter tweaks

* Correct return type hint

* Clarify list parameters

* Pin pylint version

.github/workflows/pylint.yml

@@ -31,7 +31,7 @@ jobs:
       run: |
         python -m pip install --upgrade pip
         pip install -r requirements.txt
-        pip install pylint pytest selenium
+        pip install pylint==3.2.7 pytest selenium
     - name: Analysing the code with pylint
       run: |
         pylint -d R0913,R0914,R0915,R1702,W0718,W0719,R0801 $(git ls-files '*.py')

README.md

@@ -51,6 +51,7 @@ The majority of the conformance checks done by ScubaGoggles rely on [GWS Admin l
 ### Usage
 
 - [Usage: Parameters](/docs/usage/Parameters.md)
+- [Usage: Config File](/docs/usage/Config.md)
 - [Usage: Examples](/docs/usage/Examples.md)
 - [Reviewing Output](/docs/usage/ReviewOutput.md)
 - [Limitations](/docs/usage/Limitations.md)

docs/usage/Config.md

@@ -0,0 +1,26 @@
+
+# Usage: Config File
+All ScubaGoggles [parameters](/docs/usage/Parameters.md) can be placed into a configuration file in order to made execution easier. The path of the file is specified by the `--config` parameter, and its contents are expected as YAML.
+
+> [!NOTE]
+> If a parameter is specified both on the command-line and in a configuration file, the command-line parameter has precedence over the config file.
+
+## Sample Configuration Files
+[Sample config files](/sample-config-files) are available in the repo and are discussed below.
+
+### Basic Usage
+The [basic use](/sample-config-files/basic_config.yaml) example config file specifies the `outpath`, `baselines`, and `quiet` parameters.
+
+ScubaGoggles can be invokes with this config file:
+```
+scubagoggles gws --config basic_config.yaml
+```
+
+It can also be invoked while overriding the `baselines` parameter.
+```
+scubagoggles gws --config basic_config.yaml -b gmail chat
+```
+
+## Navigation
+- Continue to [Usage: Examples](/docs/usage/Examples.md)
+- Return to [Documentation Home](/README.md)

docs/usage/Examples.md

@@ -1,5 +1,8 @@
 
 # Usage: Examples
+> [!Note]
+> If you chose not install the `scubagoggles` package in a venv but do have the dependencies installed from `requirements.txt`, you may execute the tool using the `scuba.py` script located in the root directory of this repository. Replace any `scubagoggles` directions with `python scuba.py`
+
 ## Example 1: Run an assessment against all GWS products
 ```
 scubagoggles gws
@@ -33,11 +36,14 @@ See the `help` options yourself
 scubagoggles gws -h
 ```
 
-The html report should open automatically. If not, navigate to the output folder and open the `*.html` file using a browser of your choice. The json output will also be located in this folder.
+## Example 6: Run with a config file
+```
+scubagoggles gws --config sample-config-files/basic_config.yaml
+```
 
 > [!NOTE]
-> If you chose not install the `scubagoggles` package in a venv but do have the dependencies installed from `requirements.txt`, you may execute the tool using the `scuba.py` script located in the root directory of this repository. Replace any `scubagoggles` directions with `python scuba.py`
+> In all the above examples, the html report should open automatically. If not, navigate to the output folder and open the `*.html` file using a browser of your choice. The json output will also be located in this folder.
 
 ## Navigation
 - Continue to [Reviewing Output](/docs/usage/ReviewOutput.md)
-- Return to [Documentation Home](/README.md)
+- Return to [Documentation Home](/README.md)

docs/usage/Parameters.md

@@ -19,6 +19,10 @@ optional arguments:
   -c , --credentials    The relative path and name of the OAuth / service account credentials json file. Defaults to
                         "./credentials.json" which means the tool will look for the file named credentials.json in the
                         current directory.
+  --config              Local file path to a YAML formatted configuration file. Configuration file parameters can be
+                        used in place of command-line parameters. Additional parameters and variables not available
+                        on the command line can also be included in the file that will be provided to the tool for
+                        use in specific tests.
   --outjsonfilename     The name of the file that encapsulates all assessment output. Defaults to ScubaResults.
   --subjectemail        Only applicable when using a service account. The email address of a user the service account
                         should act on behalf of. This user must have the necessary privileges to run scubagoggles.
@@ -50,5 +54,5 @@ optional arguments:
 ```
 
 ## Navigation
-- Continue to [Usage: Examples](/docs/usage/Examples.md)
-- Return to [Documentation Home](/README.md)
+- Continue to [Usage: Config File](/docs/usage/Config.md)
+- Return to [Documentation Home](/README.md)

requirements.txt

@@ -6,3 +6,4 @@ dnspython==2.6.1
 pandas==2.2.0
 tqdm==4.66.5
 requests==2.32.3
+pyyaml==6.0.2

sample-config-files/basic_config.yaml

@@ -0,0 +1,14 @@
+# YAML basic configuration file with examples of how to specify various types
+# of parameters.
+
+# Example specifying a string parameter
+outputpath: example_output
+
+# Example specifying a list parameter
+# Note that list parameter values must be formatted as a list (i.e., with
+# brackets) in the config file, even if only one value is specified. For
+# example, "baselines: [gmail]" is correct but "baselines: gmail" is not.
+baselines: [gmail, calendar, groups, chat, drive, meet, sites, commoncontrols]
+
+# Example specifying a boolean paramter
+quiet: false

scubagoggles/main.py

@@ -7,6 +7,7 @@
 
 import argparse
 from scubagoggles.orchestrator import Orchestrator
+from scubagoggles.scuba_argument_parser import ScubaArgumentParser
 
 def get_gws_args(parser):
     """
@@ -39,6 +40,13 @@ def get_gws_args(parser):
     'Defaults to "./credentials.json" which means the tool will look ' +
     'for the file named credentials.json in the current directory.')
 
+    parser.add_argument('--config', type=str, required=False, metavar='',
+    help='Local file path to a YAML formatted configuration file. ' +
+    'Configuration file parameters can be used in place of command-line ' +
+    'parameters. Additional parameters and variables not available on the ' +
+    'command line can also be included in the file that will be provided to the ' +
+    'tool for use in specific tests.')
+
     parser.add_argument('--outjsonfilename', type=str,
     default=default_file_output_names['json_output_name'], metavar='',
     help='The name of the file that encapsulates all assessment output.' +
@@ -136,8 +144,8 @@ def dive():
     "check against one or more Google Workspace products"
     gws_parser = subparsers.add_parser('gws', help=gws_parser_help)
     get_gws_args(gws_parser)
-
-    args = parser.parse_args()
+    scuba_parser = ScubaArgumentParser(parser)
+    args = scuba_parser.parse_args_with_config()
 
     if args.scuba_cmd == 'gws':
         Orchestrator(args).start_automation()

scubagoggles/scuba_argument_parser.py

@@ -0,0 +1,86 @@
+"""
+Class for parsing the config file and command-line arguments.
+"""
+
+import argparse
+import yaml
+
+class ScubaArgumentParser:
+    """
+    Class for parsing the config file and command-line arguments.
+    """
+
+    # Create a mapping of the long form of parameters to their short aliases
+    _param_to_alias = {
+        "baselines": "b",
+        "outputpath": "o",
+        "credentials": "c"
+    }
+
+    def __init__(self, parser):
+        self.parser = parser
+
+    def parse_args(self) -> argparse.Namespace:
+        """
+        Parse the arguments without loading config file.
+        """
+        return self.parser.parse_args()
+
+    def parse_args_with_config(self) -> argparse.Namespace:
+        """
+        Parse the arguments and the config file, if provided, resolving any
+        differences between the two.
+        """
+        args = self.parse_args()
+
+        # Create a mapping of the short param aliases to the long form
+        alias_to_param = {
+            value: key for key, value in self._param_to_alias.items()
+        }
+
+        # Get the args explicitly specified on the command-line so we know
+        # what should override the config file
+        cli_args = self._get_explicit_cli_args(args)
+
+        # If a config file is not specified, just return the args unchanged.
+        if args.config is not None:
+            with open(args.config, 'r', encoding="utf-8") as f:
+                config = yaml.safe_load(f)
+            config_params = list(config)
+            for param in config_params:
+                # If the short form of a param was provided in the config,
+                # translate it to the long form
+                if param in alias_to_param:
+                    config[alias_to_param[param]] = config[param]
+                    param = alias_to_param[param]
+                # If the param was specified in the command-line, the
+                # command-line arg takes precedence
+                if param in cli_args:
+                    continue
+                vars(args)[param] = config[param]
+        # Return the args (argparse.Namespace) as a dictionary
+        return args
+
+    @classmethod
+    def _get_explicit_cli_args(cls, args : argparse.Namespace) -> dict:
+        """
+        Return the list of arguments that were explicitly specified on the
+        command-line.
+        """
+        # Build a secondary parser, configure the secondary parser to
+        # suppress the default values so the secondary parser will only
+        # contain the values explicitly specified on the command-line.
+        aux_parser = argparse.ArgumentParser(argument_default=argparse.SUPPRESS)
+        for arg, val in vars(args).items():
+            dests = [f"--{arg}"]
+            # If the arg has a short form alias, add the short form as well
+            if arg in cls._param_to_alias:
+                dests.append(f"-{cls._param_to_alias[arg]}")
+            # If the arg is a boolean, need to specify the store action
+            # otherwise the boolean args will cause an error
+            if isinstance(val, bool):
+                aux_parser.add_argument(*dests, action="store_false")
+            else:
+                aux_parser.add_argument(*dests)
+        cli_args, _ = aux_parser.parse_known_args()
+        return cli_args

setup.py

@@ -21,7 +21,8 @@
         'dnspython==2.6.1',
         'pandas==2.2.0',
         'tqdm==4.66.5',
-        'requests==2.32.3'
+        'requests==2.32.3',
+        'pyyaml==6.0.2'
       ],
       entry_points={
           'console_scripts': ['scubagoggles=scubagoggles.main:dive']