setup: download OPA; resolve other PR issues

.github/actions/setup-dependencies-macos/action.yml

@@ -29,5 +29,5 @@ runs:
     - name: Download OPA executable
       shell: bash
       run: |
-        scubagoggles setup -m -nc -d ~/scubagoggles -r ~/scubagoggles -c credentials.json
+        scubagoggles setup -m -nc -nd -d ~/scubagoggles -r ~/scubagoggles -c credentials.json
         scubagoggles getopa -v ${{ inputs.opa-version }}

.github/actions/setup-dependencies-windows/action.yml

@@ -29,5 +29,5 @@ runs:
     - name: Download OPA executable
       shell: powershell
       run: |
-        scubagoggles setup -m -nc -d ~/scubagoggles -r ~/scubagoggles -c credentials.json
+        scubagoggles setup -m -nc -nd -d ~/scubagoggles -r ~/scubagoggles -c credentials.json
         scubagoggles getopa -v ${{ inputs.opa-version }}

README.md

@@ -32,15 +32,6 @@ We use a three-step process:
 2. **Verify**. Compare the exported settings from the previous step with the configuration prescribed in the baselines. We do this using [OPA Rego](https://www.openpolicyagent.org/docs/latest/policy-language/#what-is-rego), a declarative query language for defining policy.
 3. **Report**. Package the results as HTML and JSON.
 
-## Limitations of the tool
-The majority of the conformance checks done by ScubaGoggles rely on
-[GWS Admin log events](https://support.google.com/a/answer/4579579?hl=en). If there is no log event corresponding to a SCuBA
-baseline policy, ScubaGoggles will indicate that the setting currently can not
-be checked on its HTML report output. In this situation, we recommend you
-manually review your GWS security configurations with the SCuBA secure
-baselines. See [Limitations](docs/usage/Limitations.md) for more
-details.
-
 ## Table of Contents
 
 ### Installation

docs/installation/DownloadAndInstall.md

@@ -19,6 +19,10 @@ Running ScubaGoggles requires Python 3.9 or higher.  If Python is not installed
 in your environment, please visit the [Python website](https://www.python.org/)
 for instructions on how to download and install Python.
 
+A 64-bit operating system is required.  While Python will run in a 32-bit
+environment, the Open Policy Agent (OPA) required for ScubaGoggles is only
+available on 64-bit platforms.
+
 Depending on the operating system, the command to invoke Python from the command
 line is either `python` (Windows) or `python3` (linux & macOS, for backward
 compatibilty with Python version 2).  You will need access to a command line,
@@ -146,16 +150,17 @@ scubagoggles setup
 ```
 
 You will be prompted to enter the output directory, location of the OPA
-executable, and the location and name of the Google credentials file.  You
-do not have to download the OPA executable, and you do not have to create the
-Google credentials file before running setup.  If either do not exist, you
-will see a warning indicating one or both is missing.  You will need to have
-both the OPA executable and credentials files before running the conformance
-assessment.
-
-These are sample outputs from running the setup utility before the required files
-are available.  Warnings are shown, but the `.scubagoggles` configuration file
-is still created.
+executable, and the location and name of the Google credentials file.  By
+default, the OPA executable will be downloaded into the directory you specify.
+You do not have to create the Google credentials file before running setup.  If
+the credentials file doesn't exist, you will see a warning indicating that it is
+missing.  You will need to have the credentials file before running the
+conformance assessment.  See the [instructions for creating a credentials
+file](../authentication/AuthenticationMethods.md).
+
+These are sample outputs from running the setup utility before the required
+files are available.  Warnings are shown, but the `.scubagoggles` configuration
+file is still created.
 
 ### Windows Example
 
@@ -164,11 +169,13 @@ is still created.
 Setup: output directory
 Scubagoggles output directory [C:\Users\userID\scubagoggles]?
 Create directory C:\Users\userID\scubagoggles [Yes/no]?
+  creating: C:\Users\userID\scubagoggles
   C:\Users\userID\scubagoggles
 Setup: OPA executable directory
 (WARNING): OPA executable not found in PATH
 Location of OPA executable [C:\Users\userID\scubagoggles]?
-(WARNING): OPA executable not found in C:\Users\userID\scubagoggles
+  downloading: opa_windows_amd64.exe
+  OPA executable: C:\Users\userID\scubagoggles\opa_windows_amd64.exe
 Setup: Google API credentials file
 Google credentials (JSON) file [C:\Users\userID\scubagoggles\credentials.json]?
 (WARNING): Google credentials file not found in C:\Users\userID\scubagoggles\credentials.json
@@ -180,11 +187,14 @@ Google credentials (JSON) file [C:\Users\userID\scubagoggles\credentials.json]?
 $ scubagoggles setup
 Setup: output directory
 Scubagoggles output directory [/home/userID/scubagoggles]?
+Create directory /home/userID/scubagoggles [Yes/no]?
+  creating: /home/userID/scubagoggles
   /home/userID/scubagoggles
 Setup: OPA executable directory
 (WARNING): OPA executable not found in PATH
 Location of OPA executable [/home/userID/scubagoggles]?
-(WARNING): OPA executable not found in /home/userID/scubagoggles
+  downloading: opa_linux_amd64_static
+  OPA executable: /home/userID/scubagoggles/opa_linux_amd64_static
 Setup: Google API credentials file
 Google credentials (JSON) file [/home/userID/scubagoggles/credentials.json]?
 (WARNING): Google credentials file not found in /home/userID/scubagoggles/credentials.json

docs/installation/OPA.md

@@ -18,11 +18,11 @@ options:
   --version <OPA-version>, -v <OPA-version>
                         Version of OPA to download (default: latest version)
   --opa_directory <directory>, -r <directory>
-                        Directory containing OPA executable (default: location established by setup
+                        Directory containing OPA executable (default: location established by setup)
 ```
 ```
 # example
-scubagoggles getopa -v v0.60.0 -c
+scubagoggles getopa -v v0.60.0
 ```
 
 If you have run the [ScubaGoggles setup utility](DownloadAndInstall.md#ScubaGoggles-Setup-Utility),

docs/prerequisites/Prerequisites.md

@@ -17,8 +17,7 @@ https://www.googleapis.com/auth/cloud-identity.policies.readonly
 When running ScubaGoggles for the first time you will be prompted to consent to
 these API scopes. Users with the Super Admin role automatically have the
 privilege to consent to these scopes. When running ScubaGoggles for the first
-time you will be prompted to consent to these API scopes. Users with the Super
-Admin role automatically have the privilege to consent to these scopes.
+time you will be prompted to consent to these API scopes.
 
 ## Create a Project
 1. If you already have a Google Cloud Project that you want to utilize skip to [Authentication Methods](../authentication/AuthenticationMethods.md)

docs/troubleshooting/Troubleshooting.md

@@ -12,6 +12,23 @@ Ensure that you consented to the following API scopes as a user with the proper
 [permissions to consent](../prerequisites/Prerequisites.md#permissions) and have
 enabled the required [APIs and Services](../authentication/OAuth.md).
 
+## Windows:  WinError 10013: Permission Error
+
+When ScubaGoggles is run and it needs to re-authorize you using your Google
+credentials, it makes a connection using port 8080.  If you receive a permission
+error with the text `An attempt was made to access a socket in a way forbidden
+by its access permissions`, it is likely that another process on your system is
+using that port.
+
+The following PowerShell command, when run as an Adminstrator, may help to
+locate the process using the port.  Once you've determined how the port is
+being used, you can evaluate whether something may be done to temporarily
+relinquish the port for ScubaGoggles use or whether you might need to try
+running ScubaGoggles on a system where the port is available.
+
+```
+Get-Process -Id (Get-NetTCPConnection -LocalPort 8080).OwningProcess
+```
 
 ## Unable to view HTML report due to environment limitations
 

scubagoggles/getopa.py

@@ -7,6 +7,7 @@
 import re
 import stat
 import subprocess
+import sys
 
 from hashlib import sha256
 from pathlib import Path
@@ -16,7 +17,7 @@
 from urllib.request import Request, urlcleanup, urlopen, urlretrieve
 
 from scubagoggles.orchestrator import UserRuntimeError
-from scubagoggles.user_setup import prompt_boolean
+from scubagoggles.utils import prompt_boolean
 
 log = logging.getLogger(__name__)
 
@@ -55,6 +56,7 @@ def getopa(arguments: argparse.Namespace):
 
     download_opa(opa_dir, version, verify, force)
 
+
 def download_opa(opa_dir: Path,
                  version: str = None,
                  verify: bool = False,
@@ -77,6 +79,9 @@ def download_opa(opa_dir: Path,
 
     # pylint: disable=too-many-branches
 
+    if sys.maxsize <= 2 ** 32:
+        raise UserRuntimeError('64-bit operating environment required for OPA')
+
     opa_base_url = 'https://github.com/open-policy-agent/opa/releases/'
     version_re = re.compile(r'v\d+(?:\.\d+){2}')
 
@@ -98,18 +103,7 @@ def download_opa(opa_dir: Path,
                                    'string - expected "v<X>.<Y>.<Z>"')
         version = f'v{version}'
 
-    os_type = platform.system().lower()
-
-    arch = platform.machine().lower()
-
-    if arch == 'x86_64':
-        arch = 'amd64'
-    elif arch.startswith('arm'):
-        arch = 'arm64'
-
-    file_name = f'opa_{os_type}_{arch}'
-
-    file_name += '.exe' if os_type == 'windows' else '_static'
+    file_name = opa_filespec()
 
     log.debug('Downloading %s to %s', file_name, str(opa_dir))
 
@@ -159,6 +153,40 @@ def download_opa(opa_dir: Path,
 
     test_opa(output_file)
 
+
+def opa_filespec(opa_dir: Path = None):
+
+    """Returns the file name for the OPA executable that is EXPECTED to be
+    the default for the current operating environment.  For example, on a
+    macOS system with the ARM architecture, the default executable should
+    be "opa_darwin_arm64_static", but the user may rename this to be
+    simply "opa" or download the AMD architecture executable, which will
+    run even in an ARM-based macOS environment.  This function simply
+    returns the name of the expected executable.
+
+    :param opa_dir: [optional] expected location of the OPA executable.
+    :return: complete file specification (as a Path), if the OPA directory
+        is provided; otherwise, the expected OPA file name (as a str) for
+        the current environment.
+    :rtype: Path or str
+    """
+
+    os_type = platform.system().lower()
+
+    arch = platform.machine().lower()
+
+    if arch == 'x86_64':
+        arch = 'amd64'
+    elif arch.startswith('arm'):
+        arch = 'arm64'
+
+    file_name = f'opa_{os_type}_{arch}'
+
+    file_name += '.exe' if os_type == 'windows' else '_static'
+
+    return opa_dir / file_name if opa_dir else file_name
+
+
 def test_opa(opa_exe_file: Path):
 
     """Runs the OPA "version" command to check that the downloaded OPA

scubagoggles/main.py

@@ -7,7 +7,6 @@
 
 import argparse
 import logging
-import os
 import sys
 
 from pathlib import Path
@@ -20,6 +19,7 @@
 from scubagoggles.scuba_argument_parser import ScubaArgumentParser
 from scubagoggles.user_setup import default_file_names, find_legacy_dir, \
     user_setup
+from scubagoggles.utils import path_parser
 from scubagoggles.version import Version
 
 EXIT_FAILURE = 1
@@ -275,6 +275,13 @@ def get_setup_args(parser: argparse.ArgumentParser, user_config: UserConfig):
                         action = 'store_true',
                         help = 'Do not check for directory or file existence')
 
+    parser.add_argument('--nodownload',
+                        '-nd',
+                        default = False,
+                        action = 'store_true',
+                        help = 'Do not download OPA executable when it does '
+                               'not exist')
+
     parser.add_argument('--noprompt',
                         '-np',
                         default = False,
@@ -377,22 +384,6 @@ def log_level(level):
     return return_level
 
 
-def path_parser(value):
-
-    """Given a string value, this function returns an absolute Path.  The
-    value may contain a leading "~" to indicate the user's home directory,
-    and may use environment variables (e.g., $HOME).
-
-    :param str value: directory or file specification
-
-    :return: absolute Path
-    """
-
-    path_value = Path(os.path.expandvars(value)).expanduser().absolute()
-
-    return path_value
-
-
 def dive():
 
     """Takes in the arguments needed to run scubagoggles

scubagoggles/orchestrator.py

@@ -420,12 +420,12 @@ def start_automation(self):
         try:
             find_opa(args.opapath)
         except FileNotFoundError as fnf:
-            raise UserRuntimeError('OPA executable missing - '
-                                   f'{see_docs}') from fnf
+            raise UserRuntimeError(f'? "{args.opapath}" - OPA executable '
+                                   f'missing - {see_docs}') from fnf
 
         if not args.credentials.exists():
-            raise UserRuntimeError('Google credentials file missing - '
-                                   f'{see_docs}')
+            raise UserRuntimeError(f'? "{args.credentials}" - Google '
+                                   f'credentials file missing - {see_docs}')
 
         # add any additional variables to args
         gws_params = self.gws_products()

scubagoggles/run_rego.py

@@ -112,7 +112,7 @@ def find_opa(opa_path: Path = None):
     os_type = platform.system().lower()
 
     architectures = [platform.machine().lower()]
-    if architectures[0] == 'x64_64':
+    if architectures[0] == 'x86_64':
         architectures.append('amd64')
 
     # An ARM-based Mac can supposedly run the AMD64 version

scubagoggles/scuba_argument_parser.py

@@ -11,6 +11,7 @@
 import yaml
 
 from scubagoggles.reporter.md_parser import MarkdownParser
+from scubagoggles.utils import path_parser
 from scubagoggles.version import Version
 
 
@@ -135,8 +136,21 @@ def validate_config(args : argparse.Namespace) -> None:
         options.
         """
 
-        if 'credentials' in args and not isinstance(args.credentials, Path):
-            args.credentials = Path(args.credentials)
+        # Options read in from the configuration file must be converted to the
+        # same data type that's defined in the corresponding command parser
+        # definition (see the main module).  The following option values are
+        # read as strings but are converted to Path.
+
+        path_value_options = ('credentials',
+                              'documentpath',
+                              'opapath',
+                              'outputpath',
+                              'regopath')
+
+        for option_name in path_value_options:
+            if (option_name in args
+                and not isinstance(getattr(args, option_name), Path)):
+                setattr(args, option_name, path_parser(args.credentials))
 
         if 'omitpolicy' in args:
             ScubaArgumentParser.validate_omissions(args)