RS setup scripts updates (#1496)

* Simplified load-etor-org-settings script to only run in local env

* Using preferred gradlew commands instead of prime ones. Also added env variable for RS_HOME

* Moved common functions to /scripts/utils.sh and importing scripts using absolute path

* Extract env variables to a shared .env file, updated docs, renamed some of the envars. Also working of having absolute paths so the scripts could run from anywhere

* Fixed .env loading and some cleanup

* Renamed reset.sh => setup.sh

* Consolidated RS setup scripts into one

* Moved rs/setup.sh to setup-rs.sh and removed rs/ folder

* Updated RS setup instructions in readme to:
- Use new setup-rs.sh script
- Add alternate ways to build and run RS
- Fixed RS docs URL
- Clean up and simplify

* Moved files to reorganize and simplify file structure

* Updated paths

* Reverted moving hurl files

* Moved and renamed function scripts

* Moved and renamed more scripts + added to readme

* Moved function to common.sh

* Removed unused hurl/readme.md

* Fixed .env file reference

* Added to readme and added user message to find instructions

* Added context to instruction

* Added setup script for env vars

* Changed relative path to hurl files for absolute paths

* Fixed typos and cleanup

* Removed setup script not working as intended

* Readme update

* Fixed issues after running shellcheck

* Removed dead code and renamed readme section

* Reverted adding double quotes as it introduces a bug

* Added missing export of env vars that are required to create vault credentials

* Fixed typo

.shellcheckrc

@@ -0,0 +1,2 @@
+
+disable=SC1090,SC1091

README.md

@@ -332,7 +332,7 @@ For database documentation: [/docs/database.md](/docs/database.md)
 
 1. Checkout `main` branch for `CDCgov/trusted-intermediary`
 2. Run `./generate_env.sh` to generate `.env` file with required environment variables
-3. Run TI with `./gradlew clean app:run`
+3. Run TI with `./gradlew clean run`
 
 #### ReportStream Setup
 
@@ -341,17 +341,14 @@ After enabling this option it is recommended that you delete all docker images a
 with this option enabled.
 
 1. Checkout `master` branch for `CDCgov/prime-reportstream`
-2. Create a symbolic link or copy the scripts found at [/scripts/rs](/scripts/rs) to `prime-reportstream/prime-router`
-   - **Note**: follow the instructions in [/scripts/rs/readme.md](/scripts/rs/readme.md) to set up the environment variable
-3. CD to `prime-reportstream/prime-router`
-4. Run the `./cleanslate` script. For more information you can refer to the [ReportStream docs](https://github.com/CDCgov/prime-reportstream/blob/master/prime-router/docs/docs-deprecated/getting-started/getting-started.md#building-the-baseline)
-5. If attempting to access the metadata endpoint in ReportStream add the variable `ETOR_TI_baseurl="http://host.docker.internal:8080"` to `.prime-router/.vault/env/.env.local` file before building the container
-6. Run RS with `docker compose up --build -d`
-7. Run the `reset.sh` script to reset the database
-8. Run the `update_org_yaml.sh` script to update the RS organization settings
-9. Run the `load-etor-org-settings.sh` to apply the ETOR organization settings
-10. Run the `setup-local-vault.sh` script to set up the local vault secrets
-    - You can verify that the script created the secrets successfully by going to `http://localhost:8200/` in your browser, use the token in `prime-router/.vault/env/.env.local` to authenticate, and then go to `Secrets engines` > `secret/` to check the available secrets
+2. Build RS (for more information please refer to the [ReportStream docs](https://github.com/CDCgov/prime-reportstream/blob/master/prime-router/docs/getting-started/README.md)):
+   - If building for the first time, run: `./cleanslate` in `prime-reportstream/prime-router`
+   - Otherwise run: `./gradlew clean package` in `prime-reportstream` root folder
+   - If attempting to access the metadata endpoint in RS add the variable `ETOR_TI_baseurl="http://host.docker.internal:8080"` to `prime-router/.vault/env/.env.local` file before building the container
+3. Run RS with `docker compose up -d`. You may also use `./gradlew quickRun`
+4. Run the RS setup script in this repository: `/scripts/setup/setup-reportstream.sh`
+   - Before running the script, make sure to follow the instructions in [/scripts/README.md](/scripts/README.md)
+   - You can verify the script created vault secrets successfully by going to `http://localhost:8200/` in your browser, use the token in `prime-router/.vault/env/.env.local` to authenticate, and then go to `Secrets engines` > `secret/` to check the available secrets
 
 #### Submit request to ReportStream
 

scripts/.env.template

@@ -0,0 +1,26 @@
+# Core settings
+# Use $HOME or the full path to your home directory instead of ~
+CDCTI_HOME="/path/to/trusted-intermediary"
+RS_HOME="/path/to/prime-reportstream"
+
+# API URLs
+RS_LCL_API_URL="http://localhost:7071"
+RS_STG_API_URL="https://staging.prime.cdc.gov:443"
+RS_PRD_API_URL="https://prime.cdc.gov:443"
+TI_LCL_API_URL="http://localhost:8080"
+TI_STG_API_URL="https://cdcti-stg-api.azurewebsites.net:443"
+TI_PRD_API_URL="https://cdcti-prd-api.azurewebsites.net:443"
+
+# Client keys
+TI_LOCAL_PRIVATE_KEY_PATH="${CDCTI_HOME}/mock_credentials/organization-trusted-intermediary-private-key-local.pem"
+TI_LOCAL_PUBLIC_KEY_PATH="${CDCTI_HOME}/mock_credentials/organization-trusted-intermediary-public-key-local.pem"
+RS_LOCAL_PRIVATE_KEY_PATH="${CDCTI_HOME}/mock_credentials/organization-report-stream-private-key-local.pem"
+
+# Storage settings
+AZURITE_CONNECTION_STRING="DefaultEndpointsProtocol=http;AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==;BlobEndpoint=http://localhost:10000/devstoreaccount1;" # pragma: allowlist secret
+
+# Message snapshot suffixes
+FILE_NAME_SUFFIX_STEP_0="_0_initial_message"
+FILE_NAME_SUFFIX_STEP_1="_1_hl7_translation"
+FILE_NAME_SUFFIX_STEP_2="_2_fhir_transformation"
+FILE_NAME_SUFFIX_STEP_3="_3_hl7_translation_final"

scripts/README.md

@@ -1,3 +1,236 @@
 # Scripts
 
-These scripts rely in a `CDCTI_HOME` environment variable that needs to be set in your environment to your local path to the CDC-TI codebase. You may update and run the `start-here.sh` script to set it. You may also want to add it to your shell's startup file to persist it.
+## Setup
+
+Follow the instructions below to load the environments variables required for these scripts
+
+1. Copy `.env.template` to `.env`
+    ```
+    cp .env.template .env
+    ```
+2. Edit `.env` and make sure to update at least:
+   - `CDCTI_HOME`: local path to the `trusted-intermediary` codebase
+   - `RS_HOME`: local path to the `prime-reportstream` codebase
+   - **Note**: if you don't set `CDCTI_HOME`, none of these scripts will work. Also, please use `$HOME` or the full path to your home directory instead of `~`
+3. Export the environment variables in `.env` by running
+   ```
+   set -a; source .env; set +a
+   ```
+   **Note**: you may also want to add it to your shell's startup file so you don't need to run it for every terminal session.
+4. Run your script
+
+## Available Scripts
+
+### submit.sh
+
+Sends a HL7 message to RS and tracks its status throughout the flow until final delivery. When running locally, it grabs the snapshots of the file in azurite after converting to FHIR, after applying transformations in TI, and after converting back to HL7; and it copies those files to the same folder where the submitted file is. If running in a deployed environment we currently don't have a way to download the files from Azure, but the script will print the relative path for the files in the blob storage container.
+
+#### Requirements
+
+- hurl
+- jq
+- azure-cli
+
+#### Usage
+
+```
+Usage: ./submit.sh -f <message_file.hl7> [-e <environment>]
+
+Options:
+    -f <FILE>                   Message file path (Required)
+    -e <ENVIRONMENT>            Environment: local|staging|production (Default: local)
+    -x <RS_CLIENT_PRIVATE_KEY>  Path to the client private key for authentication with RS API (Required for non-local environments)
+    -z <TI_CLIENT_PRIVATE_KEY>  Path to the client private key for authentication with TI API (Optional for all environments)
+    -h                          Display this help and exit
+```
+
+### rs.sh
+
+Submit requests to RS API endpoints
+
+#### Requirements
+
+- hurl
+- jwt-cli
+
+#### Usage
+
+```
+Usage: ./rs.sh <ENDPOINT_NAME> [OPTIONS]
+
+ENDPOINT_NAME:
+    The name of the endpoint to call (required)
+
+Options:
+    -f <REL_PATH>       Path to the hl7/fhir file to submit (Required for waters API)
+    -r <ROOT_PATH>      Root path to the hl7/fhir files (Default: /Users/bbogado/Code/Flexion/CDC-TI/trusted-intermediary/examples/)
+    -t <CONTENT_TYPE>   Content type for the message (Default: application/hl7-v2)
+    -e <ENVIRONMENT>    Environment: local|staging|production (Default: local)
+    -c <CLIENT_ID>      Client ID (Default: flexion)
+    -s <CLIENT_SENDER>  Client sender (Default: simulated-sender)
+    -k <KEY_PATH>       Path to the client private key (Required for non-local environments)
+    -i <SUBMISSION_ID>  Submission ID for history API (Required for history API)
+    -v                  Verbose mode
+    -h                  Display this help and exit
+```
+
+#### Examples
+
+Sending an order to local environment
+
+```
+./rs.sh waters -f Test/Orders/003_AL_ORM_O01_NBS_Fully_Populated_0_initial_message.hl7
+```
+
+Sending a result to local environment
+
+```
+./rs.sh waters -f Test/Results/002_AL_ORU_R01_NBS_Fully_Populated_0_initial_message.hl7
+```
+
+Sending an order to staging
+
+```
+./rs.sh waters -f Test/Orders/003_AL_ORM_O01_NBS_Fully_Populated_0_initial_message.hl7 -e staging -k /path/to/client/staging/private/key
+```
+
+Checking the history in local environment for a submission id
+
+```
+./rs.sh history -i 100
+```
+
+Checking the history in staging for a submission id
+
+```
+./rs.sh history -i 100 -e staging -k /path/to/client/staging/private/key
+```
+
+### ti.sh
+
+Submit requests to TI API endpoints
+
+#### Requirements
+
+- hurl
+- jwt-cli
+
+#### Usage
+
+```
+Usage: ./ti.sh <ENDPOINT_NAME> [OPTIONS]
+
+ENDPOINT_NAME:
+    The name of the endpoint to call (required)
+
+Options:
+    -f <REL_PATH>       Path to the hl7/fhir file to submit (Required for orders and results APIs)
+    -r <ROOT_PATH>      Root path to the hl7/fhir files (Default: /Users/bbogado/Code/Flexion/CDC-TI/trusted-intermediary/examples/)
+    -e <ENVIRONMENT>    Environment: local|staging (Default: local)
+    -c <CLIENT>         Client ID to create JWT with (Default: report-stream)
+    -k <KEY_PATH>       Path to the client private key (Required for non-local environments)
+    -i <SUBMISSION_ID>  Submission ID for metadata API (Required for orders, results and metadata API)
+    -v                  Verbose mode
+    -h                  Display this help and exit
+```
+
+#### Examples
+
+Submit an order to local environment:
+```
+./ti.sh orders -f Test/Orders/003_AL_ORM_O01_NBS_Fully_Populated_1_hl7_translation.fhir -i 100
+```
+
+Submit an order to staging:
+```
+./ti.sh orders -f Test/Orders/003_AL_ORM_O01_NBS_Fully_Populated_0_initial_message.hl7 -e staging -k /path/to/client/staging/private/key
+
+```
+
+Submit a result to local environment:
+```
+./ti.sh results -f Test/Results/002_AL_ORU_R01_NBS_Fully_Populated_1_hl7_translation.fhir -i 100
+```
+
+Get metadata from local environment:
+```
+./ti.sh metadata -i 100
+```
+
+Authenticate to local environment:
+```
+./ti.sh auth
+```
+
+Get OpenAPI docs from local environment:
+```
+./ti.sh openapi
+```
+
+Get Health info from local environment:
+```
+./ti.sh health
+```
+
+### epic.sh
+
+Submit requests to Epic API endpoints
+
+#### Requirements
+
+- hurl
+
+#### Before running the script
+
+- Add the `client` id to `epic.sh`
+- Update the `secret` variable path
+
+#### Usage
+
+`./epic.sh results`
+
+### setup/update-examples-snapshots.sh
+
+Sends all the HL7 files with `_0_initial_message.hl7` suffix in the `/examples` folder to a locally running RS instance. As the `submit.sh` script, it downloads the snapshots at each stage. This script is helpful to keep all the message snapshots in the examples folder up to date
+
+#### Requirements
+
+- hurl
+- jq
+- azure-cli
+
+#### Usage
+
+```
+./update-examples-snapshots.sh
+```
+
+### setup/setup-reportstream.sh
+
+Setup script for ReportStream
+
+#### Requirements
+
+- yq
+
+#### Usage
+
+```
+./setup-reportstream.sh
+```
+
+### lib/common.sh
+
+Utility functions shared by scripts
+
+### lib/submission-utils.sh
+
+Functions to submit requests to RS, check the submission status throughout the whole flow, and downloading snapshots from azurite
+
+## Resources
+
+- [hurl](https://hurl.dev/)
+- [jq](https://jqlang.github.io/jq/)
+- [yq](https://github.com/mikefarah/yq)
+- [azure-cli](https://learn.microsoft.com/en-us/cli/azure/)
+- [jwt-cli](https://github.com/mike-engel/jwt-cli)

scripts/hurl/epic.sh → scripts/epic.sh

@@ -1,6 +1,7 @@
 #!/bin/bash
 
-source ./utils.sh
+[ -z "${CDCTI_HOME}" ] && echo "Error: Environment variable CDCTI_HOME is not set. Please refer to /scripts/README.md for instructions" && exit 1
+source "$CDCTI_HOME/scripts/lib/common.sh"
 
 client=
 audience=https://epicproxy-np.et0502.epichosted.com/FhirProxy/oauth2/token
@@ -15,5 +16,5 @@ hurl \
     --variable "fpath=$fpath" \
     --file-root "$root" \
     --variable "jwt=$jwt_token" \
-    epic/results.hurl \
-    $@
+    "$CDCTI_HOME"/scripts/epic/results.hurl \
+    "$@"