Merge pull request #1 from datatheorem/feature/blocking-mode

Feature/blocking mode
datatheorem · Jul 15, 2024 · 0d6d7df · 0d6d7df
2 parents 543132e + d660351
commit 0d6d7df
Show file tree

Hide file tree

Showing 6 changed files with 220 additions and 41 deletions.
diff --git a/README.md b/README.md
@@ -6,8 +6,9 @@ for security & privacy issues using static, dynamic, and behavioral analysis for
 More information can be found here:  
 https://www.datatheorem.com/products/mobile-secure
 
-## Example
+## Examples
 
+### Basic Example
 Add the following to your `pipeline.yml`:
 
 ```yml
@@ -21,9 +22,50 @@ steps:
       - datatheorem/data-theorem-mobile-secure:
           UPLOAD_API_KEY: $(buildkite-agent secret get DT_UPLOAD_API_KEY)
           SIGNED_BINARY_PATH: "app-debug.apk" # path to the pre-prod mobile binary built in the previous step
-          SOURCEMAP_PATH: "mapping.txt" # optional
 ```
 
+### Example with optional `SOURCEMAP_PATH`:
+An optional Java mapping.txt file for deobfuscating Android binaries.
+
+```yml
+steps:
+  - label: "Build Mobile App Binary"
+    # replace this step with your own logix to build the pre-prod mobile binary that you want to scan
+    command: "echo 'Example mobile binary build step...'"
+
+  - label: "Upload Mobile App Binary to Data Theorem for scanning"
+    plugins:
+      - datatheorem/data-theorem-mobile-secure:
+          UPLOAD_API_KEY: $(buildkite-agent secret get DT_UPLOAD_API_KEY)
+          SIGNED_BINARY_PATH: "app-debug.apk" # path to the pre-prod mobile binary built in the previous step
+          SOURCEMAP_PATH: "mapping.txt" # path to mapping.txt
+```
+
+### Example with scan result polling
+Optionally, you can configure the plugin to wait for the scan to complete and print out hte number of new security findings.
+To do this, add the extra flag `POLL_SCAN_RESULTS: true`
+This mode will also require to set up a Data Theorem Mobile Results API Key
+It can be retrieved or created at [DevSecOps -> Data Theorem Results API](https://www.securetheorem.com/devsecops/v2/results_api_access)
+And set it as a secret accessible to your BuildKite pipeline.
+
+```yml
+steps:
+  - label: "Build Mobile App Binary"
+    # replace this step with your own logix to build the pre-prod mobile binary that you want to scan
+    command: "echo 'Example mobile binary build step...'"
+
+  - label: "Upload Mobile App Binary to Data Theorem for scanning"
+    plugins:
+      - datatheorem/data-theorem-mobile-secure:
+          UPLOAD_API_KEY: $(buildkite-agent secret get DT_UPLOAD_API_KEY)
+          SIGNED_BINARY_PATH: "app-debug.apk" # path to the pre-prod mobile binary built in the previous step
+          POLL_SCAN_RESULTS: true
+          MOBILE_RESULTS_API_KEY: $(buildkite-agent secret get DT_MOBILE_RESULTS_API_KEY)
+```
+
+The plugin's logs should look like this for a successful scan with no discovered security issues
+![buildkite-data-theorem-mobile-secure-plugin-polling-mode-no-issues.png](images%2Fbuildkite-data-theorem-mobile-secure-plugin-polling-mode-no-issues.png)
+
 ## Configuration
 
 ### `UPLOAD_API_KEY` (Required, string)
@@ -36,7 +78,26 @@ We recommend using [BuildKite Secrets](https://buildkite.com/docs/pipelines/secu
 - In the BuildKite pipeline definition, you can pass the API Key as `UPLOAD_API_KEY: $(buildkite-agent secret get DT_UPLOAD_API_KEY)` in the plugin's inputs
 
 ### `SIGNED_BINARY_PATH` (Required, string)
-Path to the Mobile App binary file that has been built and should be sent for scanning
+Path to the mobile binary (APK, IPA, APPX or XAP) to be scanned.
 
 ### `SOURCEMAP_PATH` (Optional, string)
-Optionally, you can upload a sourcemap file for de-obfuscation
+An optional path to a Java mapping.txt file for deobfuscating Android binaries.
+Note: Once deobfuscation is enabled for PRE_PROD or ENTERPRISE Android app, future uploads of the same app will also require a mapping file.
+See [How To Enable De-obfuscation of Android Scan Results Using A Mapping File](https://datatheorem.atlassian.net/servicedesk/customer/portal/1/article/61669389) for more information.
+
+### `POLL_SCAN_RESULTS` (Optional, boolean)
+When set to `true`, the plugin will poll for the scan's status until completion and print if the scan has found any new issues
+This requires a Data Theorem Mobile Results API Key to be set (see below)
+
+### `MOBILE_RESULTS_API_KEY` (Optional, string)
+API Key you can retrieve in the Data theorem Portal [DevSecOps -> Data Theorem Results API](https://www.securetheorem.com/devsecops/v2/results_api_access)
+This is only required if you want to poll for scan results instead of exiting after starting the scan.
+
+Hard-coding the raw value of the API key is not recommended for security reasons.
+We recommend using [BuildKite Secrets](https://buildkite.com/docs/pipelines/security/secrets/buildkite-secrets)
+
+- On your agent cluster, define a secret named `DT_MOBILE_RESULTS_API_KEY` and set the value to what you have retrieved from the Data Theorem Portal
+- In the BuildKite pipeline definition, you can pass the API Key as `MOBILE_RESULTS_API_KEY: $(buildkite-agent secret get DT_MOBILE_RESULTS_API_KEY)` in the plugin's inputs
+
+It should look like this in your Buildkite agent secret settings
+![buildkite-data-theorem-mobile-secure-plugin-secrets.png](images%2Fbuildkite-data-theorem-mobile-secure-plugin-secrets.png)
diff --git a/hooks/command b/hooks/command
@@ -1,6 +1,24 @@
 #!/bin/bash
 set -euo pipefail
 
+# Function to handle errors
+handle_error() {
+    local http_code=$1
+    local response_body=$2
+    case $http_code in
+        401)
+            echo "Error: Unauthorized (HTTP 401). Please verify that your API key as valid."
+            ;;
+        403)
+            echo "Error: Forbidden (HTTP 403). Please verify that your API key as valid scopes to access Mobile Secure results for this mobile app."
+            ;;
+        *)
+            echo "Failed to call API. HTTP Code: ${http_code}"
+            echo "Response: ${response_body}"
+            ;;
+    esac
+    exit 1
+}
 
 # Check if the environment variable is set
 if [ -z "$DT_UPLOAD_API_KEY" ]; then
@@ -19,7 +37,7 @@ source_map_path="${BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_SOURCEMAP_PATH:-}
 if [ -n "$source_map_path" ]; then
     # Check if the path points to an existing file
     if [ -f "$source_map_path" ]; then
-      :
+        :
     else
         echo "Error: sourcemap file '${BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_SOURCEMAP_PATH}' does not exist"
         exit 1
@@ -37,41 +55,132 @@ else
     exit 1
 fi
 
+if [[ "$BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_POLL_SCAN_RESULTS" == "true" ]]; then
+    echo "POLL_SCAN_RESULTS is true"
+    if [ -z "$DT_MOBILE_RESULTS_API_KEY" ]; then
+        echo "Error: Environment variable RESULTS_API_KEY must be set when POLL_SCAN_RESULTS is set to true."
+        exit 1
+    fi
+fi
+
 maxRetries=3
-for (( retry = 0; retry < maxRetries; retry++ ))
-do
-  # Step 1: get the upload URL
-  echo "Get upload url"
-  step1_response=$(curl -s -w "%{http_code}" -X POST -H "Authorization: APIKey ${DT_UPLOAD_API_KEY}"  --data ""  https://api.securetheorem.com/uploadapi/v1/upload_init)
-  http_code=${step1_response: -3}
-  response_body=${step1_response::-3}
-
-  # For older versions of bash e.g. GNU bash, version 3.2.57(1)-release (x86_64-apple-darwin21)
-  # response_body=${step1_response%???}
-  # http_code=${step1_response#${response_body}}
-
-  # Check that http status code is 200
-  [ ! ${http_code} -eq 200 ] && echo ${response_body} && exit 1
-  upload_url=$(echo ${response_body} | jq -r ".upload_url")
-  echo ${upload_url}
-
-  # Step 2: upload the APK
-  echo "Upload app"
-  # Prepare `curl` command with main file
-  curl_command=(curl -F "file=@${file_path}")
-
-  # Check if source map file exists and add it to `curl` command
-  if [ -n "$source_map_path" ]; then
-      curl_command+=(-F "sourcemap=@${source_map_path}")
-      echo "Including sourcemap (${source_map_path})"
-  fi
-
-  # Add upload URL to `curl` command
-  curl_command+=(${upload_url})
-  step2_response=$("${curl_command[@]}") && echo ${step2_response} && break
+upload_success=false
+timeout_duration=300  # Timeout after 5 minutes (300 seconds)
+
+for (( retry = 0; retry < maxRetries; retry++ )); do
+    # Step 1: get the upload URL
+    echo "Get upload URL"
+    step1_response=$(curl -s -w "%{http_code}" -X POST -H "Authorization: APIKey ${DT_UPLOAD_API_KEY}" --data "" https://api.securetheorem.com/uploadapi/v1/upload_init)
+    http_code=${step1_response: -3}
+    response_body=${step1_response::-3}
+
+    # Check that http status code is 200
+    if [ "$http_code" -ne 200 ]; then
+        handle_error "$http_code" "$response_body"
+    fi
+
+    upload_url=$(echo ${response_body} | jq -r ".upload_url")
+    echo ${upload_url}
+
+    # Step 2: upload the APK
+    echo "Upload app"
+    # Prepare `curl` command with main file
+    curl_command=(curl -F "file=@${file_path}")
+
+    # Check if source map file exists and add it to `curl` command
+    if [ -n "$source_map_path" ]; then
+        curl_command+=(-F "sourcemap=@${source_map_path}")
+        echo "Including sourcemap (${source_map_path})"
+    fi
+
+    # Add upload URL to `curl` command
+    curl_command+=(${upload_url})
+    step2_response=$("${curl_command[@]}" -s -w "%{http_code}")
+    http_code=${step2_response: -3}
+    response_body=${step2_response::-3}
+
+    if [ "$http_code" -ne 200 ]; then
+        handle_error "$http_code" "$response_body"
+    fi
+
+    mobile_app_id=$(echo "$response_body" | jq -r '.mobile_app_id')
+    scan_id=$(echo "$response_body" | jq -r '.scan_id')
+
+    if [[ -n "$mobile_app_id" && -n "$scan_id" ]]; then
+        upload_success=true
+        break
+    fi
+
+    if [ $retry -eq $((maxRetries - 1)) ]; then
+        echo "Max retries reached. Exiting."
+        exit 1
+    fi
+
+    echo "Retrying... ($((retry + 1))/$maxRetries)"
 done
 
-if [ ${retry} -ge ${maxRetries} ]; then
-  echo "Upload failed after ${maxRetries} attempts"
-  exit 1
-fi
+if $upload_success && [[ "$BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_POLL_SCAN_RESULTS" == "true" ]]; then
+    echo "Polling for scan status: scan id=$scan_id"
+
+    start_time=$(date +%s)
+
+    # Polling loop
+    while true; do
+        current_time=$(date +%s)
+        elapsed_time=$((current_time - start_time))
+
+        if [ $elapsed_time -gt $timeout_duration ]; then
+            echo "Timeout: Static scan did not complete within 5 minutes."
+            exit 0
+        fi
+
+        response=$(curl -s -w "%{http_code}" -X GET "https://api.securetheorem.com/apis/mobile_security/results/v2/mobile_apps/${mobile_app_id}/scans/${scan_id}" \
+        -H "Authorization: APIKey ${DT_MOBILE_RESULTS_API_KEY}")
+
+        # Extract the HTTP status code and response body
+        http_code=${response: -3}
+        response_body=${response::-3}
+
+        if [ "$http_code" -eq 200 ]; then
+            static_scan_status=$(echo "$response_body" | jq -r '.static_scan.status')
+            echo "Scan Status: $static_scan_status"
+
+            if [ "$static_scan_status" == "COMPLETED" ]; then
+                echo "Scan completed successfully."
+
+                # Extract the start_date from the response for results_since parameter
+                start_date=$(echo "$response_body" | jq -r '.start_date')
+
+                # Step 3: Get security findings
+                findings_response=$(curl -s -w "%{http_code}" -X GET "https://api.securetheorem.com/apis/mobile_security/results/v2/security_findings" \
+                -H "Authorization: APIKey ${DT_MOBILE_RESULTS_API_KEY}" \
+                -G --data-urlencode "mobile_app_id=${mobile_app_id}" \
+                --data-urlencode "status_group=OPEN" \
+                --data-urlencode "results_since=${start_date}")
+
+                findings_http_code=${findings_response: -3}
+                findings_response_body=${findings_response::-3}
+
+                if [ "$findings_http_code" -ne 200 ]; then
+                    handle_error "$findings_http_code" "$findings_response_body"
+                fi
+
+                total_count=$(echo "$findings_response_body" | jq -r '.pagination_information.total_count')
+
+                if [ "$total_count" -gt 0 ]; then
+                    echo "FAILED: Found $total_count open security findings"
+                    echo "For more information visit https://www.securetheorem.com/mobile-secure/v2/security/"
+                    exit 0 # Log scan result, but don't fail the pipeline
+                else
+                    echo "PASSED: No security findings found."
+                    exit 0 # ok
+                fi
+            else
+                echo "Static scan is still ongoing..."
+                sleep 30  # Wait for 30 seconds before next poll
+            fi
+        else
+            handle_error "$http_code" "$response_body"
+        fi
+    done
+fi
diff --git a/hooks/environment b/hooks/environment
@@ -13,4 +13,5 @@ resolve_secret() {
 }
 
 # Fetch and resolve the API key
-export DT_UPLOAD_API_KEY=$(resolve_secret "$BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_UPLOAD_API_KEY")
+export DT_UPLOAD_API_KEY=$(resolve_secret "$BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_UPLOAD_API_KEY")
+export DT_MOBILE_RESULTS_API_KEY=$(resolve_secret "$BUILDKITE_PLUGIN_DATA_THEOREM_MOBILE_SECURE_MOBILE_RESULTS_API_KEY")
diff --git a/images/buildkite-data-theorem-mobile-secure-plugin-polling-mode-no-issues.png b/images/buildkite-data-theorem-mobile-secure-plugin-polling-mode-no-issues.png
diff --git a/images/buildkite-data-theorem-mobile-secure-plugin-secrets.png b/images/buildkite-data-theorem-mobile-secure-plugin-secrets.png
diff --git a/plugin.yml b/plugin.yml
@@ -16,6 +16,14 @@ configuration:
       type: string
     SOURCEMAP_PATH:
       type: string
+    POLL_SCAN_RESULTS:
+      type: "boolean"
+      default: false
+      description: |
+        If the plugin should wait until the scan is completed and check the scan results
+        When set to `true`, `MOBILE_RESULTS_API_KEY` also needs to be set
+    MOBILE_RESULTS_API_KEY:
+      type: string
   required:
     - UPLOAD_API_KEY
     - SIGNED_BINARY_PATH