The objective of this tool to validate targets in Target Servers & Apigee API Proxy Bundles exported from Apigee. Validation is done by deploying a sample proxy which check if HOST & PORT is open from Apigee.
NOTE: Discovery of Targets in API Proxy & Sharedflows is limited to only parsing URL from
TargetEndpoint&ServiceCalloutPolicy.
NOTE: Dynamic targets are NOT supported, Ex :
https://host.{request.formparam.region}.example.com}
-
Python3.x
-
Java
-
Maven >= 3.9.6
-
If you are pushing the data to gcp metrics, you require
roles/monitoring.editorrole. -
Please install the required Python dependencies
python3 -m pip install -r requirements.txt
- Please build the java callout jar by running the below command
bash callout/build_java_callout.sh
- Please fill in
input.properties
[source]
baseurl=https://x.x.x.x/v1 # Apigee Base URL. e.g http://management-api.apigee-opdk.corp:8080
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID
auth_type=basic # API Auth type basic | oauth
[target]
baseurl=https://apigee.googleapis.com/v1 # Apigee Base URL
org=xxx-xxxx-xxx-xxxxx # Apigee Org ID
auth_type=oauth # API Auth type basic | oauth
[csv]
file=input.csv # Path to input CSV. Note: CSV needs HOST & PORT columns
default_port=443 # default port if port is not provided in CSV
[validation]
check_csv=true # 'true' to validate Targets in input csv
check_proxies=true # 'true' to validate Proxy Targets else 'false'
skip_proxy_list=mock1,stream # Comma separated list of proxies to skip validation;
proxy_export_dir=export # Export directory needed when check_proxies='true'
api_env=dev # Target Environment to deploy Validation API Proxy
api_name=target_server_validator # Target API Name of Validation API Proxy
api_force_redeploy=false # set 'true' to Re-deploy Target API Proxy
api_hostname=example.apigee.com # Target VirtualHost or EnvGroup Domain Name
api_ip=<IP> # IP address corresponding to api_hostname. Use if DNS record doesnt exist
report_format=csv # Report Format. Choose csv or md (defaults to md)
[gcp_metrics]
enable_gcp_metrics=true # set 'true' to push target server's host and status to GCP metrics
project_id=xxx-xxx-xxx # Project id of GCP project where the data will be pushed
metric_name=custom.googleapis.com/<metric_name> # Replace <metric_name> with custom metric name
enable_dashboard=true # set 'true' to create the dashboard with alerting policy
dashboard_title=Apigee Target Server Monitoring Dashboard # Monitoring Dashboard Title
alert_policy_name=Apigee Target Server Validator Policy # Alerting Policy Name
notification_channel_ids=xxxxxxxx # Comma separated list of Notification Channel ids
[target_server_state_file]
state_file=gs://bucket_name/path/to/file/scan_output.json # GCS Bucket path to store --scan output
# state_file=file://scan_output.json # File path to store --scan output (only one can be used either GCS or file)
gcs_project_id=xxx-xxxx-xxx-xxxxx # GCS bucket project id
To get the notification channel id, use the following command
gcloud beta monitoring channels list --project=<project_id>
This command will display all available notification channels within your project. You can select the appropriate one based on your requirements. Locate the notification channel ID under the name field in the format projects/<project_id>/notificationChannels/<notification_channel_id>, and insert it into the input.properties file.
- Sample input CSV with target servers
NOTE: You need to set
check_csv=truein thevalidationsection ofinput.properties
NOTE: You need to set
file=<CSV Name>in thecsvsection ofinput.propertiesIf PORT is omitted from the csv, the value of default_port will be used frominput.properties.
HOST,PORT
httpbin.org
mocktarget.apigee.net,80
smtp.gmail.com,465
- Please run below commands to authenticate,
gcloud auth application-default set-quota-project <project_id>
You can skip the quota-project if you want.
Another way to authenticate is to use the environmnet variables based on the Apigee flavours.
export APIGEE_OPDK_ACCESS_TOKEN=$(echo -n "<user>:<password>" | base64) # Access token for Apigee OPDK
export APIGEE_ACCESS_TOKEN=$(gcloud auth print-access-token) # Access token for Apigee X/Hybrid
- Export Target Server Details
- Export Proxy Bundle
- Parse Each Proxy Bundle for Target
- Run Validate API against each Target (optional)
- Generate csv/md Report or push data to GCP Monitoring Dashboard
The script supports the below arguments
--onboardoption to create validator proxy, custom metric descriptors, alerting policy and dashboard--scanoption to fetch target servers from Environment target servers, api proxies & csv file--monitoroption to check the status of target servers and generate report or push to GCP metrics--offboardoption to delete validator proxy, custom metric descriptors, alerting policy and dashboard--inputPath to input properties file
To onboard, run
python3 main.py --input path/to/input_file --onboard
Make sure you have build the java callout jar before running onboard.
To scan, run
python3 main.py --input path/to/input_file --scan
To monitor, run
python3 main.py --input path/to/input_file --monitor
To offboard, run
python3 main.py --input path/to/input_file --offboard
You can also pass multiple arguments at the same time.
--onboard deploys an API proxy to validate if the target servers are reachable or not. To use the API proxy, make sure your payloads adhere to the following format:
[
{
"host": "example.com",
"port": 443
},
{
"host": "example2.com",
"port": 443
},
// Add up to 8 more host-port combinations as needed
]The response will look like this -
[
{
"host": "example.com",
"port": 443,
"status" : "REACHABLE"
},
{
"host": "example2.com",
"port": 443,
"status" : "UNKNOWN_HOST"
},
// and so on
]Validation Report: report.md OR report.csv can be found in the same directory as the script.
Please check a Sample report
The script can also create a GCP Monitoring Dashboard with an alerting widget like shown below:
This script creates a custom metric with labels as hostname and status. The possible statuses, namely REACHABLE NOT_REACHABLE, and UNKNOWN_HOST, are determined by calling the validator proxy. These statuses are then assigned values of 1, 0.5, and 0, respectively.
Then, an alerting policy is created with a threshold of 0.75. Entries below this threshold trigger alerts sent to designated notification channels. Finally, this policy is added as a widget on the GCP dashboard.
To run the pipeline script (pipeline.sh), follow these steps:
Before running the pipeline script, ensure you have the following prerequisites configured:
-
Environment Variables: Set up the necessary environment variables required by the script. These variables should include:
APIGEE_X_ORG: Your Apigee organization ID.APIGEE_X_ENV: The Apigee environment to deploy to.APIGEE_X_HOSTNAME: The hostname for your Apigee instance.
NOTE: This pipeline will create a test notification channel with type email and email_address as
[email protected]. -
IAM Roles: To set up the monitoring dashboard and alerts, make sure that you have
roles/monitoring.editorrole. -
Input Properties Template: This script requires an
input.propertiesfile for the necessary configuration parameters and will create a correspondinggenerated.propertiesfile by replacing the environment variables with their values. Ensure that the values are set properly in this file before running the script.
To execute the pipeline, use the following command:
./pipeline.sh