Merge remote-tracking branch 'origin' into docker-compose-version-bump

.github/workflows/tyk-demo-tests.yml

@@ -11,7 +11,7 @@ jobs:
       - run: echo "🐧 This job is now running on a ${{ runner.os }} server hosted by GitHub!"
       - run: echo "🔎 The name of your branch is ${{ github.ref }} and your repository is ${{ github.repository }}."
       - name: Check Out Repository Code
-        uses: actions/checkout@v3
+        uses: actions/checkout@v4
       - run: echo "💡 The ${{ github.repository }} repository has been cloned to the runner."
       - name: Create Environment File
         run: |
@@ -22,15 +22,15 @@ jobs:
       - name: Bootstrap and Test All Eligible Deployments
         run: sudo ./scripts/test-all.sh
       - name: Store Logs
-        if: success() || failure()
-        uses: actions/upload-artifact@v3
+        if: always()
+        uses: actions/upload-artifact@v4
         with:
           name: logs
           path: logs/
       - name: Show Bootstrap Log
-        if: success() || failure()
+        if: always()
         run: cat logs/bootstrap.log
       - name: Show Test Log
-        if: success() || failure()
+        if: always()
         run: cat logs/test.log
       - run: echo "🍏 This job's status is ${{ job.status }}."

CONTRIBUTING.md

@@ -137,7 +137,7 @@ These utility scripts are available in the `scripts` directory:
 * `add-gateway.sh`: Creates a new Tyk Gateway container, using the same configuration as the base Tyk deployment Gateway
 * `common.sh`: Contains functions useful for bootstrap scripts
 * `export.sh`: Uses the Dashboard API to export API and Policy definitions, overwriting data used to bootstrap the base Tyk deployment
-* `test.sh`: Uses a Newman container to run the Postman collection tests for only the **Tyk** deployment
+* `test.sh`: Uses a Newman container to run the Postman collection tests for all deployment that are currently bootstrapped
 * `test-all.sh`: As `test.sh`, but runs for all deployments
 * `update-hosts.sh`: Adds the necessary hosts to the `/etc/hosts` file
 
@@ -299,6 +299,14 @@ Once the tests are finished, the `delete` function can be called to remove the k
 tyk.dashboardApi.tools.apiKey.delete(pm);
 ```
 
+## Test Automation
+
+Deployments are included in test scripts (`scripts/test.sh` and `scripts/test-all.sh`) if all of the following criteria are met:
+
+- A Postman collection is found in the deployment
+- The collection does *not* contain a variable `test-runner-ignore` with the value `true`
+- The collection contains tests
+
 ## Testing Responses
 
 The Tyk Demo Postman collection contains many requests, each of which demonstrate a particular piece of functionality. Testing the responses generated by these requests provides validation that the desired result was achieved.

README.md

@@ -2,41 +2,41 @@
 
 # Tyk Demo
 
-This repo provides an example installation of Tyk. It uses Docker Compose to provide a quick, simple deployment, where you can choose what features to include.
+Tyk Demo offers a pre-configured deployment that includes various examples showcasing Tyk's capabilities. Powered by Docker Compose for swift setup, it allows you to pick and choose the features you want to explore.
 
-Tyk Demo was built to enable the sharing of knowledge and combining of effort amongst client-facing technical Tyk folks. But it's also useful to anyone interested in playing with a Tyk installation.
+The primary use case for Tyk Demo is to act as a centralised knowledge sharing resource for client-facing technical Tyk staff. However, anyone curious about Tyk can benefit from Tyk Demo. It's hands-on sandbox environment allows anyone to experiment and learn more about Tyk's features.
 
-The repo contains various Postman collections that make it easy to explore and experience Tyk's features and functionality.
+Postman collections are included for some deployments. These are pre-configured to interact with the features and functionality available in the deployments, making it easy to explore and experience Tyk.
 
 See the [Contributor Guide](CONTRIBUTING.md) for information on how to contribute to and extend this repository.
 
-> :warning: Please note that this repo has been created on Mac OS X with Docker Desktop for Mac. You may experience issues if using a different operating system or approach. 
+> :warning: Please note that this repo has been created on Mac OS X with Docker Desktop for Mac. You may experience issues if using a different operating system or approach.
 
 If you encounter a problem using this repository, please try to fix it yourself and create a pull request so that others can benefit.
 
 # Overview
 
-The concept is that there is a **base deployment** of Tyk, which gives you the usual Tyk components: Gateway, Dashboard, Pump, plus the databases Redis and MongoDB. This standard deployment can be extended by including additional **feature deployments** as needed. The feature deployments cover particular scenarios for Tyk, such as:
+This project leverages a concept of deployments, enabling users to choose what gets deployed.
 
-* Single sign on
-* Analytics export
-* Tracking
-* CI/CD
-* 2nd Tyk Environment
-* Instrumentation
+- **Base deployment**: The mandatory deployment that provides standard Tyk components (Gateway, Dashboard, Pump), databases (Redis and MongoDB), and other supporting software to enhance the demo experience.
+- **Feature deployments**: Extend the base deployment functionality. These deployments cover specific scenarios for Tyk, such as single sign-on, analytics export, etc.
 
-Each feature deployment has its own directory, with the necessary files to deploy the feature and a readme to describe how to use it.
+Each deployment has a dedicated directory containing all necessary deployment files and additional information.
 
-There is a focus on simplicity. Docker Compose is used to provision the containers, and bootstrap scripts are used to initialise the environment so that it is ready to use straight away - applying configuration and populating data.
+This approach focuses on simplicity. Running a single command triggers Docker Compose to create containers, and bootstrap scripts initialise the environment. Everything is handled automatically, from configuration application to data population.
 
 ## Requirements
 
 ### License requirements
 - Get a valid [Tyk Self-Managed license](https://tyk.io/pricing-self-managed/) key (click **"start now"** under **Free trial**). **This is a self-service option!**
-- If you want to run MDCB deployment (distributed set up with control plane and data plans), then you need to [contact Tyk team](https://tyk.io/pricing-self-managed/) to get a license key. Please leave it if it's the first time you are trying out Tyk and you are not in a position to get engaged at this moment.
+- If you want to run the MDCB deployment (distributed set up with control plane and data planes), then you need to [contact the Tyk team](https://tyk.io/pricing-self-managed/) to get an MDCB license key.
 
 ### Software
-docker compose
+The base deployment requires:
+- Docker Desktop, with Docker Compose
+- jq
+
+Note that some feature deployments may have additional requirements. See deployment readmes for more information.
 
 ## Repository Structure
 
@@ -48,7 +48,7 @@ docker compose
 
 ## Deployments
 
-The deployment directories (`deployments/*`) contain the various deployments which can be made with this repo. The **base deployment** is in the `deployments/tyk` directory. The other directories are **feature deployments**, which extend the base deployment functionality and require the base deployment in order to function correctly.
+The deployment directories (`deployments/*`) contain the various deployments which can be made with this repo. The *base deployment* is in the `deployments/tyk` directory. The other directories are *feature deployments*, which extend the base deployment functionality and require the base deployment in order to function correctly.
 
 All of the directories contain `docker-compose.yml`, `bootstrap.sh` and `README.md` files specific to the deployment. They may also contain directories called `data` or `volumes`, which hold the data necessary during bootstrapping or providing as mapped volumes into the container.
 
@@ -64,7 +64,7 @@ All of the directories contain `docker-compose.yml`, `bootstrap.sh` and `README.
 * [Mail server](deployments/mailserver/README.md)
 * [MDCB](deployments/mdcb/README.md)
 * [MQTT](deployments/mqtt/README.md)
-* [OpenTelemetry with Jaeger](deployments/otel/README.md)
+* [OpenTelemetry with Jaeger](deployments/otel-jaeger/README.md)
 * [Portal](deployments/portal/README.md)
 * [SLIs and SLOs with Prometheus and Grafana](deployments/slo-prometheus-grafana/README.md)
 * [SSO](deployments/sso/README.md)

deployments/analytics-datadog/docker-compose.yml

@@ -17,7 +17,7 @@ services:
       - .env
 
   tyk-pump-datadog:
-    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.8.3}
+    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.9.0}
     networks:
       - tyk
     volumes:

deployments/analytics-kibana/bootstrap.sh

@@ -6,6 +6,8 @@ log_start_deployment
 bootstrap_progress
 
 kibana_base_url="http://localhost:5601"
+gateway_base_url="http://$(jq -r '.host_config.override_hostname' deployments/tyk/volumes/tyk-dashboard/tyk_analytics.conf)"
+gateway_api_credentials=$(cat deployments/tyk/volumes/tyk-gateway/tyk.conf | jq -r .secret)
 
 log_message "Waiting for kibana to return desired response"
 wait_for_response "$kibana_base_url/app/kibana" "200"
@@ -45,6 +47,12 @@ fi
 log_ok
 bootstrap_progress
 
+log_message "Waiting for API availability"
+# this api id is for the 'basic open api' called by the next section
+wait_for_api_loaded "727dad853a8a45f64ab981154d1ffdad" "$gateway_base_url" "$gateway_api_credentials"
+log_ok
+bootstrap_progress        
+
 log_message "Sending a test request to provide Kibana with data, as Tyk bootstrap requests will not have been picked up by the Pump from this deployment"
 log_http_result "$(curl -s localhost:8080/basic-open-api/get -o /dev/null -w "%{http_code}" 2>> logs/bootstrap.log)"
 bootstrap_progress

deployments/analytics-kibana/docker-compose.yml

@@ -18,7 +18,7 @@ services:
     volumes:
       - elasticsearch-data:/usr/share/elasticsearch/data
   tyk-pump-elasticsearch:
-    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.8.3}
+    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.9.0}
     networks:
       - tyk
     volumes:

deployments/analytics-splunk/bootstrap.sh

@@ -7,6 +7,8 @@ bootstrap_progress
 
 splunk_base_url="http://localhost:8000"
 splunk_base_mgmt_url="https://localhost:8089"
+gateway_base_url="http://$(jq -r '.host_config.override_hostname' deployments/tyk/volumes/tyk-dashboard/tyk_analytics.conf)"
+gateway_api_credentials=$(cat deployments/tyk/volumes/tyk-gateway/tyk.conf | jq -r .secret)
 
 log_message "Waiting for splunk to return desired response"
 wait_for_response "$splunk_base_url/en-GB/account/login" "200"
@@ -52,6 +54,14 @@ fi
 log_ok
 bootstrap_progress
 
+log_message "Waiting for API availability"
+# this api id is for the 'basic open api' called by the next section
+wait_for_api_loaded "727dad853a8a45f64ab981154d1ffdad" "$gateway_base_url" "$gateway_api_credentials"
+# this api id is for the 'httpbin acme' API called by the deployment tests
+wait_for_api_loaded "93fd5c15961041115974216e7a3e7175" "$gateway_base_url" "$gateway_api_credentials"
+log_ok
+bootstrap_progress        
+
 log_message "Sending a test request to provide Splunk with data"
 # since request sent in base bootstrap process will not have been picked up by elasticsearch-enabled pump
 log_http_result "$(curl -s localhost:8080/basic-open-api/get -o /dev/null -w "%{http_code}" 2>> logs/bootstrap.log)"

deployments/analytics-splunk/docker-compose.yml

@@ -16,7 +16,7 @@ services:
       - splunk-data:/opt/splunk/var
       - splunk-data:/opt/splunk/etc
   tyk-splunk-pump:
-    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.8.3}
+    image: tykio/tyk-pump-docker-pub:${PUMP_VERSION:-v1.9.0}
     networks:
       - tyk
     volumes:

deployments/federation/bootstrap.sh

@@ -1,12 +1,13 @@
 #!/bin/bash
 
 source scripts/common.sh
-deployment="GraphQL Federations"
+deployment="GraphQL Federation"
 log_start_deployment
 
 # Setup variables
 dashboard_base_url="http://tyk-dashboard.localhost:$(jq -r '.listen_port' deployments/tyk/volumes/tyk-dashboard/tyk_analytics.conf)"
 gateway_base_url="http://$(jq -r '.host_config.override_hostname' deployments/tyk/volumes/tyk-dashboard/tyk_analytics.conf)"
+gateway_api_credentials=$(cat deployments/tyk/volumes/tyk-gateway/tyk.conf | jq -r .secret)
 dashboard_admin_api_credentials=$(cat deployments/tyk/volumes/tyk-dashboard/tyk_analytics.conf | jq -r .admin_secret)
 dashboard_user_api_key=$(get_context_data "1" "dashboard-user" "1" "api-key")
 
@@ -23,6 +24,17 @@ bootstrap_progress
 create_api "deployments/federation/data/apis-supergraph.json" "$dashboard_admin_api_credentials" "$dashboard_user_api_key"
 bootstrap_progress
 
+log_message "Waiting for API availability"
+for file in deployments/federation/data/*; do
+  if [[ -f $file ]]; then
+    target_api_id=$(cat $file | jq '.api_definition.api_id' --raw-output)
+    wait_for_api_loaded "$target_api_id" "$gateway_base_url" "$gateway_api_credentials"
+    bootstrap_progress        
+  fi
+done
+bootstrap_progress        
+log_ok
+
 log_end_deployment
 
 echo -e "\033[2K