RS Integration Tests in Staging (#1293)

* Added new module rs-e2e for RS Integration Tests in Staging * Added initial implementation to pull files from Azure storage container * Added local file fecther, hl7 file matcher, and first test assertion on matched files Co-authored-by: Sylvie <[email protected]> * Cleaned up and simplified logic * Clean up: remove comment * start copying rule engine stuff from etor project Co-Authored-By: Basilio Bogado <[email protected]> * first draft of assertions Co-Authored-By: Basilio Bogado <[email protected]> * Renamed 'transformation' to 'assertion', plus some cleanup * Some more refatoring to adapt the transformation engine to be used for HL7 assertions * Turned arrays in definitions into key value pairs * Updated the assertion definitions to use new syntax and added initial implementation for the hl7 expression evaluator Co-authored-by: Sylvie <[email protected]> * Further implementation for the hl7 expression validator. Got a test expression working for equal comparison between hl7 field and a literal value Co-authored-by: Sylvie <[email protected]> * Cleanup * Some refactoring and cleanup for parseAndEvaluate * Added implementation for evaluateMembership and removed Optional return value Co-authored-by: Sylvie <[email protected]> * Added method to evaluate count operation + cleanup * Use both input and output messages, add a few more statements to test Co-Authored-By: Basilio Bogado <[email protected]> * Start making AssertionRuleEngine work and some associated cleanup Co-Authored-By: Basilio Bogado <[email protected]> * Start using the rules file, doing a little cleanup and troubleshooting Co-Authored-By: Basilio Bogado <[email protected]> * Fixed loading of the definitions file and dependency injection * Fixed null pointer exception. All test assertions are now passing * Added missing assertion * Added missing catch of NumberFormatException * Trying to fix issue with sonar * Added HL7FileStream record to be able to later log the file name Co-authored-by: Sylvie <[email protected]> * Refactored, added logging and fixed dependency injection Co-authored-by: Sylvie <[email protected]> * Added unit tests for the asserion rules engine Co-authored-by: Sylvie <[email protected]> * Catched missing exceptions Co-authored-by: Sylvie <[email protected]> * Refactored to use singleton pattern and dependency injection system for consistency Co-authored-by: Sylvie <[email protected]> * Refactored to reduce code duplication and merge rule engine framework Co-authored-by: Sylvie <[email protected]> * Further refactoring to abstract and reduce code duplication among rule engine frameworks. Still pending to integrate the transformation and validation rule engines to use the new classes Co-authored-by: Sylvie <[email protected]> * Further refactoring to integrate the transformation and validation rule engines to use the new classes Co-authored-by: Sylvie <[email protected]> * Fixed tests in rs-e2e + cleanup * Refactoring to improve null handling * Added test cobverafe for HapiHL7ExpressionEvaluator and HapiHL7FileMatcher Co-authored-by: Sylvie <[email protected]> * Cleanup and move class to appropiate location * Added some of the remaining test coverage * Added test coverage * Updated assertion definitions Co-authored-by: Sylvie <[email protected]> * Added sample file to test UCSD transformations Co-authored-by: Sylvie <[email protected]> * Updated message to avoid sending to ucsd Co-authored-by: Sylvie <[email protected]> * Fixed arguments order Co-authored-by: Sylvie <[email protected]> * Implemented a simple HL7 parser given we can't get what we need from the Hapi library * Added missing error handling * Removed unused code and fixed failing tests Co-authored-by: Sylvie <[email protected]> * Added unit tests in rs-e2e to the allUnitTests task Co-authored-by: Sylvie <[email protected]> * Fixed regex Co-authored-by: Sylvie <[email protected]> * Added github workflow to run the automated test Co-authored-by: Sylvie <[email protected]> * CHanged one assertion to fail to test the workflow Co-authored-by: Sylvie <[email protected]> * Tests are still passing in the PR. Trying something else Co-authored-by: Sylvie <[email protected]> * Updated MSH-10 ids for sample messages + cleanup Co-authored-by: Sylvie <[email protected]> * Need more output from log to get information about the reason for test failure * Way too much output in the logs with --info * Reverted the change to make test fail and updated cron job to run 2 hours after the staging flow run * Removed pull_request trigger used for testing * Add additional test coverage * Added missing javadocs + cleanup * Fixed merge conflicts * Renamed 'result' to 'ORU' to avoid confusion * Moved HapiHL7FileMatcher and HL7FileStream to rs-e2e project given they are not used outside of this project * Updated to follow convention * Explicitly use UTF-8 charset * Move injections before public methods * Updated language to capture what is really the issue here * Simplified logic * Set all patterns as private static final * Missed one Pattern * Changed ArrayList for Set, which is more appropiate in this case * Moved HapiHL7ExpressionEvaluator and HapiHL7Message to rs-e2e project * Added back JavaDocs * Avoid code duplication * Need to keep inputstream open until used * Added readme files for the rs-e2e project and the sample files Co-authored-by: Sylvie <[email protected]> * Added ADRs for the assertion engine and the automated rs integration test. Some more work to be done on the integration test one Co-authored-by: Sylvie <[email protected]> * Added integration test section in the main readme Co-authored-by: Sylvie <[email protected]> * Fixed merge conflicts * Added a couple more assertions for UCSD transformations * Added more to the integration test ADR. Still some more pending Co-authored-by: Sylvie <[email protected]> * Update shared/src/test/groovy/gov/hhs/cdc/trustedintermediary/wrappers/HealthDataTest.groovy Co-authored-by: Jorge Lopez <[email protected]> * More updates to ADR * Renamed readme.md => README.MD for consistency and changed urls to use relative path from the root of the repo * Missed on last commit * Formatting * Added a few more impact points * Capitalized file extenesion by mistake * Trying to update file extension capitalization * Trying to get git to catch the file extension capitalization * Reverting workaround to get git to catch the file extension capitalization --------- Co-authored-by: Sylvie <[email protected]> Co-authored-by: Jorge Lopez <[email protected]>
CDCgov · Oct 4, 2024 · c992bc0 · c992bc0
1 parent 338be18
commit c992bc0
Show file tree

Hide file tree

Showing 101 changed files with 2,589 additions and 290 deletions.
diff --git a/.github/workflows/automated-staging-test-run.yml b/.github/workflows/automated-staging-test-run.yml
@@ -0,0 +1,19 @@
+name: Automated Staging Test - Run integration tests
+
+on:
+  schedule:
+    - cron: "0 2 * * 2-6"  # Tuesday to Saturday at 2am UTC - two hours after `automated-staging-test-submit` runs
+  workflow_dispatch:
+
+jobs:
+  test_files:
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Check out the repository
+        uses: actions/checkout@v4
+
+      - name: Run automated tests
+        env:
+          AZURE_STORAGE_CONNECTION_STRING: ${{ secrets.AUTOMATED_TEST_AZURE_STORAGE_CONNECTION_STRING }}
+        run: ./gradlew rs-e2e:clean rs-e2e:automatedTest
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -80,7 +80,7 @@ jobs:
         run: ./gradlew spotlessCheck
 
       - name: Sonar
-        run: ./gradlew allBuilds testCodeCoverageReport e2e:assemble sonar --info
+        run: ./gradlew allBuilds testCodeCoverageReport e2e:assemble rs-e2e:assemble sonar --info
         env:
           GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
           SONAR_TOKEN: ${{ secrets.SONAR_TOKEN }}

diff --git a/README.md b/README.md
@@ -3,9 +3,11 @@
 This document provides instructions for setting up the environment, running the application, and performing various tasks such as compiling, testing, and contributing to the project.
 
 ## Requirements
+
 Any distribution of the Java 17 JDK.
 
 ## Using and Running
+
 To run the application directly, execute...
 
 ```shell
@@ -17,14 +19,14 @@ This runs the web API on port 8080. The app reads/writes data to a local file (u
 You can view the API documentation at `/openapi`.
 
 ### Generating and using a token
+
 1. Run `brew install mike-engel/jwt-cli/jwt-cli`
 2. Replace `PATH_TO_FILE_ON_YOUR_MACHINE` in this command with the actual path, then run it: `jwt encode --exp='+5min' --jti $(uuidgen) --alg RS256  --no-iat -S @/PATH_TO_FILE_ON_YOUR_MACHINE/trusted-intermediary/mock_credentials/organization-trusted-intermediary-private-key-local.pem`
 3. Copy token from terminal and paste into your postman body with the key `client_assertion`
 4. Add a key to the body with the key `scope` and value of `trusted-intermediary`
 5. Body type should be `x-wwww-form-urlencoded`
 6. You should be able to run the post call against the `v1/auth/token` endpoint to receive a bearer token [to be used in this step](#submit-request-to-reportstream)
 
-
 ## Development
 
 ### Additional Requirements
@@ -49,19 +51,23 @@ creates a `.env` file in the resource folder with the required configuration
    ```bash
    ./generate_env.sh
    ```
+
 3. If you run TI using Docker rather than Gradle, update the DB and port values in the `.env` file (the alternate values are in comments)
 
 ### Using a local database
+
 Use [docker-compose.postgres.yml](docker-compose.postgres.yml) to run your local DB. In IntelliJ, you can click the play arrow to start it
 
 ![docker-postgres.png](images/docker-postgres.png)
 
 Apply all outstanding migrations:
+
 ```bash
 liquibase update --changelog-file ./etor/databaseMigrations/root.yml --url jdbc:postgresql://localhost:5433/intermediary --username intermediary --password 'changeIT!' --label-filter '!azure'
 ```
 
 If running in Windows, use double quotes instead:
+
 ```shell
 liquibase update --changelog-file ./etor/databaseMigrations/root.yml --url jdbc:postgresql://localhost:5433/intermediary --username intermediary --password "changeIT!" --label-filter "!azure"
 ```
@@ -109,6 +115,11 @@ This will start the API, wait for it to respond, run the end-to-end tests agains
 These tests are located under the `e2e` Gradle sub-project directory.  Like any Gradle project, there are the `main` and `test` directories.
 The `test` directory contains the tests.  The `main` directory contains our custom framework that helps us interact with the API.
 
+#### Automated ReportStream Integration/End-to-End Test
+
+These tests cover the integration between ReportStream and TI. They run automatically every
+weekday via Github actions. See [the rs-e2e readme](rs-e2e/README.md) for more details.
+
 #### Load Testing
 
 Load tests are completed with [Locust.io](https://docs.locust.io/en/stable/installation.html).
@@ -119,6 +130,7 @@ Run the load tests by running...
 
 ./docker-load-execute.sh
 ```
+
 Currently, we are migrating to using Azure. Local load testing is using gradle, however a docker load test is available to mimic the Azure environment settings until the azure migration is complete.
 
 This will run the API for you, so no need to run it manually.
@@ -139,11 +151,12 @@ locust -f ./operations/locustfile.py
 
 The terminal will start a local web interface, and you can enter
 the swarm parameters for the test and the local url where the app is running
-(usually http://localhost:8080).  You can also set time limits for the tests under 'Advanced Settings'.
+(usually `http://localhost:8080`).  You can also set time limits for the tests under 'Advanced Settings'.
 
 ### Debugging
 
 #### Attached JVM Config for IntelliJ
+
 The project comes with an attached remote jvm configuration for debuging the container.
 If you check your remote JVM settings, under `Run/Edit Configurations`,
 you will see the `Debug TI`. If you want to add a new remote JVM configuration, follow the steps below,
@@ -154,6 +167,7 @@ under "**Docker Container Debugging Using Java Debug Wire Protocal**"
 Go into the `Dockerfile` file and change `CMD ["java", "-jar", "app.jar"]` to `CMD ["java", "-agentlib:jdwp=transport=dt_socket,address=*:6006,server=y,suspend=n", "-jar", "app.jar"]`
 
 #### Steps
+
 1. In Intellij, click on Run and select Edit Configurations ![img.png](images/img.png)
 2. Create a new Remote JVM Debug ![img_1.png](images/img_1.png)
 3. Set up the configuration for the remote JVM debug to look like this. ![img_3.png](images/img_2.png)
@@ -163,7 +177,6 @@ Go into the `Dockerfile` file and change `CMD ["java", "-jar", "app.jar"]` to `C
 7. Select your Docker Debug that you set up in step 3 ![img_4.png](images/img_4.png)
 8. A console window will pop up that will show you that it is connected to Docker, and at that point, you can interact with your container and then step through the code at your breakpoints. ![img_5.png](images/img_5.png)
 
-
 ### Deploying
 
 #### Environments
@@ -177,21 +190,27 @@ deployed environment in a _non-CDC_ Azure Entra domain and subscription. See bel
 
 > **Before starting...**
 >
->  Remember to ping the Engineering Channel to make sure someone is not already using the enviroment.
+> Remember to ping the Engineering Channel to make sure someone is not already using the enviroment.
 
 To deploy to the Internal environment...
+
 1. Check with the team that no one is already using it.
 2. [Find the `internal` branch](https://github.com/CDCgov/trusted-intermediary/branches/all?query=internal) and delete
    it inGitHub.
 3. Delete your local `internal` branch if needed.
+
    ```shell
    git branch -D internal
    ```
+
 4. From the branch you want to test, create a new `internal` branch.
+
    ```shell
    git checkout -b internal
    ```
+
 5. Push the branch to GitHub.
+
    ```shell
    git push --set-upstream origin internal
    ```
@@ -306,6 +325,7 @@ CDC including this GitHub page may be subject to applicable federal law, includi
 ### Database
 
 For database documentation: [/docs/database.md](/docs/database.md)
+
 ### Setup with ReportStream
 
 #### CDC-TI Setup
@@ -333,7 +353,7 @@ with this option enabled.
 8. Run the `./reset.sh` script to reset the database
 9. Run the `./load-etor-org-settings.sh` to apply the ETOR organization settings
 10. Run the `./setup-local-vault.sh` script to set up the local vault secrets
-   - You can verify that the script created the secrets successfully by going to `http://localhost:8200/` in your browser, use the token in `prime-router/.vault/env/.env.local` to authenticate, and then go to `Secrets engines` > `secret/` to check the available secrets
+    - You can verify that the script created the secrets successfully by going to `http://localhost:8200/` in your browser, use the token in `prime-router/.vault/env/.env.local` to authenticate, and then go to `Secrets engines` > `secret/` to check the available secrets
 
 #### Submit request to ReportStream
 
@@ -342,23 +362,27 @@ with this option enabled.
 ###### Orders
 
 To test sending from a simulated hospital:
+
 ```
 curl --header 'Content-Type: application/hl7-v2' --header 'Client: flexion.simulated-hospital' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/orm_message.hl7' 'http://localhost:7071/api/waters'
 ```
 
 To test sending from TI:
+
 ```
 curl --header 'Content-Type: application/fhir+ndjson' --header 'Client: flexion.etor-service-sender' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oml_message.fhir' 'http://localhost:7071/api/waters'
 ```
 
 ###### Results
 
 To test sending from a simulated lab:
+
 ```
 curl --header 'Content-Type: application/hl7-v2' --header 'Client: flexion.simulated-lab' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oru_message.hl7' 'http://localhost:7071/api/waters'
 ```
 
 To test sending from TI:
+
 ```
 curl --header 'Content-Type: application/fhir+ndjson' --header 'Client: flexion.etor-service-sender' --header 'Authorization: Bearer dummy_token' --data-binary '@/path/to/oru_message.fhir' 'http://localhost:7071/api/waters'
 ```
@@ -368,14 +392,19 @@ After one or two minutes, check that hl7 files have been dropped to `prime-repor
 ##### Staging
 
 In order to submit a request, you'll need to authenticate with ReportStream using JWT auth:
+
 1. Create a JWT for the sender (e.g. `flexion.simulated-hospital`) using the sender's private key, which should be stored in Keybase. You may use [this CLI tool](https://github.com/mike-engel/jwt-cli) to create the JWT:
+
    ```
    jwt encode --exp='+5min' --jti $(uuidgen) --alg RS256 -k <sender> -i <sender> -s <sender> -a staging.prime.cdc.gov --no-iat -S @/path/to/sender_private.pem
    ```
+
 2. Use the generated JWT to authenticate with ReportStream and get the token, which will be in the `access_token` response
+
    ```
    curl --header 'Content-Type: application/x-www-form-urlencoded' --data 'scope=flexion.*.report' --data 'client_assertion=<jwt>' --data 'client_assertion_type=urn:ietf:params:oauth:client-assertion-type:jwt-bearer' --data 'grant_type=client_credentials' 'http://localhost:7071/api/token'
    ```
+
 3. Submit an Order or Result using the returned token in the `'Authorization: Bearer <token>'` header
 
 ## DORA Metrics
@@ -419,7 +448,7 @@ WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A
 PARTICULAR PURPOSE. See the Apache Software License for more details.
 
 You should have received a copy of the Apache Software License along with this
-program. If not, see http://www.apache.org/licenses/LICENSE-2.0.html
+program. If not, see <http://www.apache.org/licenses/LICENSE-2.0.html>
 
 The source code forked from other open source projects will inherit its license.
 

diff --git a/adr/024-assertion-engine.md b/adr/024-assertion-engine.md
@@ -0,0 +1,53 @@
+# 24. Assertion Engine
+
+Date: 2024-10-02
+
+## Decision
+
+1. We decided to use the Rule Engine framework (used for the transformations and validations) to define the assertions for the ReportStream Integration Test.
+2. We decided to refactor the Rule Engine framework to avoid code duplication and add HL7 handling. Before the refactor, the engine was only able to handle FHIR messages.
+3. We decided to create our own simple parser for HL7 that will allow us to access segment fields using an index.
+4. We decided to create our own HL7 expression syntax and validation, inspired by FHIRPath.
+
+## Status
+
+Accepted.
+
+## Context
+
+While creating the RS Integration Test framework, we needed a way to define the assertions that would be evaluated on the output files.
+We decided to use the Rule Engine framework, which is already used for transformations and validations, to define these assertions.
+This will allow us to reuse the existing code and make it easier to maintain.
+
+The reasoning behind the decisions in the previous sections is as follows:
+
+1. The Rule Engine framework is already in place and has been proven to work well for transformations and validations.
+2. Refactoring the Rule Engine framework will allow us to avoid code duplication and make it easier to maintain.
+3. While working with the Hapi library, we found some limitations that made it impossible to access values in the HL7 messages by segment index. The library's typing system doesn't allow to access HL7 fields by index, so we decided to create a very simple parser that would allow us to do that.
+4. In order to create assertions, we needed to define a syntax that would allow us to access HL7 fields and compare them. Since we didn't find an existing specification for it, we decided to create our own HL7 expression syntax and validation, following the same patterns and conventions as FHIRPath so it's easier to understand for those familiar with the FHIRPath specifications.
+
+## Impact
+
+For the Assertion Engine framework, similar impact to the Transformation and Validation Engine ADRs can be assumed.
+
+For the HL7 parser and expression validator, this is the expected impact:
+
+### Positive
+
+- We will be able to access HL7 fields by segment index, which will gives us more flexibility and get HL7 fields programmatically.
+- We will be able to define assertions for HL7 messages using a simple syntax that is easy to understand and maintain.
+
+### Negative
+
+- We will have to maintain the HL7 parser and expression validator, which could add some overhead to the project.
+
+### Risks
+
+- The HL7 parser and expression validator could introduce bugs that could affect the assertion evaluation.
+- The HL7 parser and expression validator may not cover all the cases we need.
+
+## Related ADRs
+
+- [Automated RS Integration Test](025-automated-rs-integration-test.md)
+- [Validation Engine](021-validation-engine.md)
+- [Transformation Engine](022-transformation-engine.md)
diff --git a/adr/025-automated-rs-integration-test.md b/adr/025-automated-rs-integration-test.md
@@ -0,0 +1,73 @@
+# 25. Automated RS Integration Test
+
+Date: 2024-10-02
+
+## Decisions
+
+1. We decided to create a full end-to-end test that covers the interaction of ReportStream and TI, using
+    existing ingestion and delivery mechanisms
+2. We decided to create Github Action workflows to schedule two tasks:
+   - One workflow submits sample HL7 files to ReportStream in staging, with output files expected to be delivered to an Azure blob container
+   - Another workflow later runs the integration tests on the output and input files
+3. We decided to use MSH-10 to match the input and output files, and to filter the receivers in ReportStream when MSH-6.2 not available.
+
+## Status
+
+Accepted.
+
+## Context
+
+### Decision 1
+
+The RS and TI applications each have their own unit and integration tests, but we didn't have any tests
+that cover the interaction between RS and TI, and we also didn't have a way to know when changes in
+RS have unintended consequences that impact our workflows.
+
+Submitting data to RS using their existing REST endpoints and receiving it using their existing delivery
+mechanisms helps make these tests realistic.
+
+### Decision 2
+
+Since we decided to use RS's existing REST endpoints, we needed a way to submit data to them, and a way
+to trigger the data flow and subsequent tests on some kind of schedule. We chose Github Actions for this
+because it's easy to both schedule them based on a CRON expression and to run them manually as needed. Github
+Actions also gave us a lightweight way to send the files to RS without having to add a new service.
+
+We are using two separate actions - the first one sends data to RS, and the second one (currently
+scheduled 2 hours after the first) triggers the tests to run. The length of time it takes a file to
+run through the whole workflow (from RS to TI to RS to final delivery) usually doesn't take long, but we
+built in extra time in case of any issues that cause delays.
+
+### Decision 3
+
+We're using the value in MSH-10 for two purposes: matching input and output files, and some filtering in RS.
+
+We chose MSH-10 to match files on because it's a value that shouldn't change and should be unique to
+a particular message. We're also using it to route these test messages because in some cases, we apply
+transformations that will overwrite HL7 fields used for routing (MSH-5 and MSH-6), so we can't rely on those.
+
+## Impact
+
+### Positive
+
+- We will have a way to test the integration between RS and TI
+- We will be able to catch issues early when changes in RS break our workflows
+
+### Negative
+
+- We will run daily tests in RS' and our staging infrastructure, which will take up resources
+
+### Risks
+
+- If we forget to add additional assertions when new transformations are added, these tests may give us
+  a false sense of confidence
+- Because we rely on MSH-10 for matching files, engineers will have to take care in setting this field
+  when they create additional tests in future
+- If we don't maintain the filtering in RS based on MSH-6.2 and MSH-10, we may not be able to route the test messages
+  correctly
+- Because we're using RS's existing REST endpoints and staging set up, if RS changes their endpoints or
+  the way they handle staging, these tests may break
+
+## Related ADRs
+
+- [Assertion Engine](024-assertion-engine.md)
diff --git a/build.gradle b/build.gradle
@@ -87,6 +87,7 @@ subprojects {
 
 ext.jacoco_excludes = [
         '**/e2e/**',
+        '**/rs-e2e/**',
         '**/javalin/App*',
         '**/jackson/Jackson*',
         '**/slf4j/LocalLogger*',
@@ -105,6 +106,7 @@ tasks.register('allUnitTests') {
     dependsOn 'app:test'
     dependsOn 'shared:test'
     dependsOn 'etor:test'
+    dependsOn 'rs-e2e:test'
 }
 
 tasks.register('allBuilds') {