update README for tests, add in testing for our scripts

.github/workflows/README.md

@@ -1,5 +1,7 @@
 # recover github workflows
 
+## Overview
+
 Recover ETL has four github workflows:
 
 - workflows/upload-and-deploy.yaml
@@ -14,35 +16,56 @@ Recover ETL has four github workflows:
 | codeql-analysis                | on-push from feature branch, feature branch merged into main |
 | cleanup                        | feature branch deleted                                       |
 
+
+## upload-files
+
+Copies pilot data sets from ingestion bucket to input data bucket for use in integration test. Note that this behavior assumes that there are files in the ingestion bucket. Could add updates to make this robust and throw an error if the ingestion bucket path is empty.
+
 ## upload-and-deploy
 
 Here are some more detailed descriptions and troubleshooting tips for some jobs within each workflow:
 
-### upload-files
-
-Copies pilot data sets from ingestion bucket to input data bucket for use in integration test. Note that this behavior assumes that there are files in the ingestion bucket. Could add updates to make this robust and throw an error if the ingestion bucket path is empty.
+### Current Testing Related Jobs
 
-### nonglue-unit-tests
+#### nonglue-unit-tests
 
 See [testing](/tests/README.md) for more info on the background behind these tests. Here, both the `recover-dev-input-data` and `recover-dev-processed-data` buckets' synapse folders are tested for STS access every time something is pushed to the feature branch and when the feature branch is merged to main.
 
 This is like an integration test and because it depends on connection to Synapse, sometimes the connection will be stalled, broken, etc. Usually this test will only take 1 min or less. Sometimes just re-running this job will do the trick.
 
-### pytest-docker
+#### pytest-docker
 
 This sets up and uploads the two docker images to ECR repository.
 **Note: A ECR repo called `pytest` would need to exist in the AWS account we are pushing docker images to prior to running this GH action.**
 
 Some behavioral aspects to note - there were limitations with the matrix method in Github action jobs thus had to unmask account id to pass it as an output for `glue-unit-tests` to use. The matrix method at this time [see issue thread](https://github.com/orgs/community/discussions/17245) doesn't support dynamic job outputs and the workaround seemed more complex to implement, thus we weren't able to pass the path of the uploaded docker container directly and had to use a static output. This leads us to use `steps.login-ecr.outputs.registry` which contains account id directly so the output could be passed and the docker container could be found and used.
 
-### glue-unit-tests
+#### glue-unit-tests
 
 See [testing](/tests/README.md) for more info on the background behind these tests.
 
 For the JSON to Parquet tests sometimes there may be a scenario where a github workflow gets stopped early due to an issue/gets canceled.
 
 With the current way when the `test_json_to_parquet.py` run, sometimes the glue table, glue crawler role and other resources may have been created already for the given branch (and didn’t get deleted because the test didn’t run all the way through) and will error out when the github workflow gets triggered again because it hits the `AlreadyExistsException`. This is currently resolved manually by deleting the resource(s) that has been created in the AWS account and re-running the github jobs that failed.
 
+### Adding Test Commands to Github Workflow Jobs
+
+After developing and running tests locally, you need to ensure the tests are run in the CI pipeline. To add your tests to under the `upload-and-deploy` job:
+
+Add your test commands under the appropriate job (see above for summaries on the specific testing related jobs), for example:
+
+```yaml
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      # Other steps...
+      - name: Run tests
+        run: |
+          pytest tests/
+
+```
+
 ### sceptre-deploy-develop
 
 ### integration-test-develop-cleanup

.github/workflows/upload-and-deploy.yaml

@@ -137,6 +137,8 @@ jobs:
         run: |
           pipenv run python -m pytest tests/test_s3_event_config_lambda.py -v
           pipenv run python -m pytest tests/test_s3_to_glue_lambda.py -v
+          pipenv run python -m pytest tests/test_lambda_dispatch.py -v
+          pipenv run python -m pytest tests/test_consume_logs.py -v
 
       - name: Test dev synapse folders for STS access with pytest
         run: >
@@ -258,6 +260,8 @@ jobs:
         run: >
           su - glue_user --command "cd $GITHUB_WORKSPACE &&
           python3 -m pytest tests/test_json_to_parquet.py --namespace $NAMESPACE -v"
+          su - glue_user --command "cd $GITHUB_WORKSPACE &&
+          python3 -m pytest tests/test_run_great_expectations_on_parquet.py -v"
 
   sceptre-deploy-develop:
     name: Deploys branch using sceptre

tests/README.md

@@ -1,6 +1,9 @@
-### Running tests
+### Tests in RECOVER
 Tests are defined in the `tests` folder in this project.
 
+### Running tests locally
+See here for information on how to run the tests locally especially when you are adding tests.
+
 #### Running tests using Docker
 All tests can be run inside a Docker container which includes all the necessary
 Glue/Spark dependencies and simulates the environment which the Glue jobs
@@ -17,6 +20,7 @@ These scripts needs to be run inside a Docker container:
 
 - Under AWS 4.0 (Dockerfile.aws_glue_4)
   - test_json_to_parquet.py (Note that these tests deploys test resources to aws and will take several min to run)
+  - test_run_great_expectations_on_parquet.py
 
 Run the following commands to run tests for:
 
@@ -83,6 +87,8 @@ pytest with other tests because they have to be run in a Dockerfile with the AWS
 #### Running tests for lambda
 Run the following command from the repo root to run tests for the lambda functions (in develop).
 
+Example:
+
 ```shell script
 python3 -m pytest tests/test_s3_to_glue_lambda.py -v
 ```
@@ -103,3 +109,6 @@ python3 -m pytest tests/test_setup_external_storage.py
 --namespace <put_namespace_here>
 --test_sts_permission <put_the_type_of_permission_to_test_here>
 ```
+
+#### Adding tests to Recover CI/CD
+Tests are run automatically as part of `upload-and-deploy.yaml` Github workflow. See [Github Workflows README](.github/workflows/README.md#adding-test-commands-to-github-workflow-jobs) for more details on the workflow.