diff --git a/README.md b/README.md index 7f7aae2..1365827 100644 --- a/README.md +++ b/README.md @@ -1,142 +1,55 @@ -[![Actions Status](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/workflows/build/badge.svg?branch=master)](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/actions?query=workflow%3Abuild+branch%3Amaster) +# Deploying a Solace PubSub+ Software Event Broker using Operator onto an OpenShift 4 Platform +The Solace PubSub+ Event Broker Operator (Operator) is a Kubernetes-native method to install and manage the lifecycle of a PubSub+ Software Event Broker on any Kubernetes platform including OpenShift. -# Deploying a Solace PubSub+ Software Event Broker onto an OpenShift 4 Platform +>Note: We recommend using the PubSub+ Event Broker Operator. An alternative method using Helm is also available from an [earlier version of this repo](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/tree/v3.1.0). -Solace [PubSub+ Platform](https://solace.com/products/platform/) is a complete event streaming and management platform for the real-time enterprise. The [PubSub+ software event broker](https://solace.com/products/event-broker/software/) efficiently streams event-driven information between applications, IoT devices, and user interfaces running in the cloud, on-premises, and in hybrid environments using open APIs and protocols like AMQP, JMS, MQTT, REST and WebSocket. It can be installed into a variety of public and private clouds, PaaS, and on-premises environments. Event brokers in multiple locations can be linked together in an [event mesh](https://solace.com/what-is-an-event-mesh/) to dynamically share events across the distributed enterprise. +This repository extends the [Solace PubSub+ Event Broker Operator on Kubernetes](https://github.com/SolaceDev/pubsubplus-kubernetes-operator) guide, providing additional specific instructions for the OpenShift 4 Platform. -## Overview - -This project is a best practice template intended for development and demo purposes. It has been tested using OpenShift v4.9. The tested and recommended Solace PubSub+ Software Event Broker version is 9.12. - -This document provides a quick getting started guide to install a Solace PubSub+ Software Event Broker in various configurations onto an OpenShift 4 platform. For OpenShift 3.11, refer to the [archived version of this quick start](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/tree/v1.1.1). - -For detailed instructions, see [Deploying a Solace PubSub+ Software Event Broker onto an OpenShift 4 platform](/docs/PubSubPlusOpenShiftDeployment.md). There is also a general quick start for [Solace PubSub+ on Kubernetes](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md) available, which the OpenShift deployment builds upon. - -The PubSub+ deployment does not require any special OpenShift Security Context; the default `restricted` SCC can be used. - -We recommend using the PubSub+ Helm chart for convenience. An alternative method [using OpenShift templates](/docs/PubSubPlusOpenShiftDeployment.md#step-4-option-2-deploy-using-openshift-templates) is also available. - -> Deprecation warning: deploying using OpenShift Templates is being phased out and the templates in this quickstart will be no longer maintained. The recommended deployment method is to use Helm. If Helm cannot be used then refer to the [PubSub+ Kubernetes documentation](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#alternative-deployment-with-generating-templates-for-the-kubernetes-kubectl-tool) to generate deployment manifests. - -## Pre-requisite: Access to OpenShift Platform - -There are [multiple ways](https://www.openshift.com/try ) to get to an OpenShift 4 platform: -- The detailed [Event Broker on OpenShift](/docs/PubSubPlusOpenShiftDeployment.md#step-1-optional--aws-deploy-a-self-managed-openshift-container-platform-onto-aws) documentation describes how to set up production-ready Red Hat OpenShift Container Platform platform on AWS. -- An option for developers is to locally deploy an all-in-one environment using [CodeReady Containers](https://developers.redhat.com/products/codeready-containers/overview). -- An easy way to get an OpenShift cluster up and running is through the [Developer Sandbox](https://developers.redhat.com/developer-sandbox) program. You can sign up for a free 14-day trial. +Contents: +- [Deploying a Solace PubSub+ Software Event Broker using Operator onto an OpenShift 4 Platform](#deploying-a-solace-pubsub-software-event-broker-using-operator-onto-an-openshift-4-platform) + - [Solace PubSub+ Software Event Broker](#solace-pubsub-software-event-broker) + - [Overview](#overview) + - [Step 1: Set Up OpenShift](#step-1-set-up-openshift) + - [Step 2: Install the PubSub+ Event Broker Operator](#step-2-install-the-pubsub-event-broker-operator) + - [Step 3: Deploy the PubSub+ Software Event Broker](#step-3-deploy-the-pubsub-software-event-broker) + - [Contributing](#contributing) + - [Authors](#authors) + - [License](#license) + - [Resources](#resources) -## Deploying PubSub+ Software Event Broker +## Solace PubSub+ Software Event Broker -The event broker can be deployed in either a three-node High-Availability (HA) group, or as a single-node standalone deployment. For simple test environments that need only to validate application functionality, a single instance will suffice. Note that in production, or any environment where message loss cannot be tolerated, an HA deployment is required. +Solace [PubSub+ Platform](https://solace.com/products/platform/) is a complete event streaming and management platform for the real-time enterprise. The [PubSub+ Software Event Broker](https://solace.com/products/event-broker/software/) efficiently streams event-driven information between applications, IoT devices, and user interfaces running in the cloud, on-premises, and in hybrid environments using open APIs and protocols like AMQP, JMS, MQTT, REST and WebSocket. It can be installed into a variety of public and private clouds, PaaS, and on-premises environments. Event brokers in multiple locations can be linked together in an [Event Mesh](https://solace.com/what-is-an-event-mesh/) to dynamically share events across the distributed enterprise. -In this quick start we go through the steps to set up an event broker using [Solace PubSub+ Helm charts](https://artifacthub.io/packages/search?page=1&repo=solace). - -There are three Helm chart variants available with default small-size configurations: -- `solace-pubsubplus-openshift-dev` - deploys a minimum footprint software event broker for developers (standalone) -- `solace-pubsubplus-openshift` - deploys a standalone software event broker that supports 100 connections -- `solace-pubsubplus-openshift-ha` - deploys three software event brokers in an HA group that supports 100 connections - -For other event broker configurations or sizes, refer to the [PubSub+ Software Event Broker Helm Chart](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/pubsubplus/README.md) documentation. - -You can install Helm charts on an OpenShift Container Platform cluster using the following methods: -* The Developer perspective of the OpenShift Web Console; or -* The CLI - -## Option 1: Installing from the OpenShift Web Console, Developer perspective - -This simple method uses the OpenShift Web Console graphical interface: +## Overview -* In a browser open the OpenShift Web Console, Developer perspective. -* Ensure not to use the `default` project, create a new project if required. -* Find and select the required PubSub+ Helm chart variant from the catalog, then click on "Install". -* Provide a unique Release Name. It is recommended to change the name that is offered by default. The maximum length of the Release Name should be 28 characters. -* If required, provide additional chart configurations. For options, consult the README link at the top of the page. Note that currently the "Form view" offers all the possible fields and the "YAML view" shows only those that have a current configuration value. It may be necessary to refresh the browser to display the latest in "YAML view". +This project is a best practice template intended for development and demo purposes. It has been tested using OpenShift v4.12. The tested and recommended PubSub+ Software Event Broker version is 10.2. -Additional information is available from the [OpenShift documentation](https://docs.openshift.com/container-platform/latest/applications/working_with_helm_charts/configuring-custom-helm-chart-repositories.html#odc-installing-helm-charts-using-Developer perspective_configuring-custom-helm-chart-repositories). +This document provides a quick getting started guide to install the broker in various configurations onto an OpenShift 4 platform. -## Option 2: Installing from CLI +For additional documentation, see [/docs/PubSubPlusOpenShiftDeployment.md](/docs/PubSubPlusOpenShiftDeployment.md) in this repo. -### Step 1: Ensure command-line console access to your OpenShift environment +## Step 1: Set Up OpenShift -Assuming you have access to an OpenShift 4 platform, log in as `kubeadmin` using the `oc login -u kubeadmin` command. +There are [multiple ways](https://www.openshift.com/try ) to set up an OpenShift 4 deployment, including the following examples: +- The detailed [Event Broker on OpenShift](/docs/PubSubPlusOpenShiftDeployment.md#deploy-a-production-ready-openshift-container-platform-onto-aws) documentation describes how to set up a production-ready Red Hat OpenShift Container Platform deployment on AWS. +- An option for developers is to locally deploy an all-in-one environment using [CodeReady Containers](https://developers.redhat.com/products/codeready-containers/overview). However, note that this requires sufficient local resources (minimum 2 CPUs and 4GB memory) in addition to the CodeReady resource requirements. -Ensure your OpenShift environment is ready: +## Step 2: Install the PubSub+ Event Broker Operator -```bash -# This command returns the current user -oc whoami -``` +The certified PubSub+ Event Broker Operator is available in OpenShift from the [integrated OperatorHub catalog](https://catalog.redhat.com/software/search?p=1&vendor_name=Solace%20Corporation). Follow [Adding Operators to a cluster](https://docs.openshift.com/container-platform/latest/operators/admin/olm-adding-operators-to-cluster.html) in the OpenShift documentation to locate and install the "PubSub+ Event Broker Operator". -### Step 2: Install and Configure Helm +## Step 3: Deploy the PubSub+ Software Event Broker -Follow the [instructions from Helm](//github.com/helm/helm#install), or if you're using Linux, simply run: -```bash -curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash +Create a new OpenShift project. It is not recommended to use the `default` project. +```sh +oc new-project solace-pubsubplus ``` -Helm is configured properly if the `helm version` command returns no error. - - -### Step 3: Install the Software Event Broker with the Default Configuration - -1. Add the Solace Helm charts to your local Helm repo: - ```bash - helm repo add openshift-helm-charts https://charts.openshift.io/ - ``` - -2. Create a new project or switch to your existing project. **Important:** do not use the `default` project as its loose permissions don't reflect a typical OpenShift environment. - ```bash - oc new-project solace-pubsubplus - ``` - - By default the latest [Red Hat certified image](https://catalog.redhat.com/software/container-stacks/search?q=solace) of PubSub+ Standard Edition available from `registry.connect.redhat.com` is used. To use a different image, add the following values (comma-separated) to the `--set` commands in Step 3 below: - - ```bash - helm install ... --set image.repository=,image.tag= - ``` - - If it is required by the image repository, you can also add the following: - ```bash - --set image.pullSecretName= - ``` - -3. Use one of the following Helm chart variants to create a deployment (for configuration options and deletion instructions, refer to the [PubSub+ Software Event Broker Helm Chart](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/tree/master/pubsubplus#configuration) documentation): - - - Create a Solace PubSub+ minimum deployment for development purposes using `solace-pubsubplus-openshift-dev`. This variant requires a minimum of 1 CPU and 3.4 GiB of memory to be available to the PubSub+ event broker pod. - ```bash - # Deploy PubSub+ Standard edition, minimum footprint developer version - helm install my-release openshift-helm-charts/solace-pubsubplus-openshift-dev - ``` - - - Create a Solace PubSub+ standalone deployment that supports 100 connections using `solace-pubsubplus-openshift`. A minimum of 2 CPUs and 3.4 GiB of memory must be available to the PubSub+ pod. - ```bash - # Deploy PubSub+ Standard edition, standalone - helm install my-release openshift-helm-charts/solace-pubsubplus-openshift - ``` - - - Create a Solace PubSub+ HA deployment that supports 100 connections using `solace-pubsubplus-openshift-ha`. This deployment requires that at least 2 CPUs and 3.4 GiB of memory are available to *each* of the three event broker pods. - ```bash - # Deploy PubSub+ Standard edition, HA - helm install my-release openshift-helm-charts/solace-pubsubplus-openshift-ha - ``` - - All of the Helm options above start the deployment and write related information and notes to the console. - - Broker services are exposed by default through a Load Balancer that is specific to your OpenShift platform. For details, see the `Services access` section of the notes written to the console. - - > Note: the `solace-pubsubplus-openshift` Helm charts differ from the general `pubsubplus` charts in that the `securityContext.enabled` Helm parameter value is `false` by default, which is required for OpenShift. - -4. Wait for the deployment to complete, following any instructions that are written to the console. You can now [validate the deployment and try the management and messaging services](/docs/PubSubPlusOpenShiftDeployment.md#validating-the-deployment). - - > Note: There is no external Load Balancer support with CodeReady Containers. Services are accessed through NodePorts instead. Check the results of the `oc get svc my-release-pubsubplus` command. This command returns the ephemeral NodePort port numbers for each message router service. Use these port numbers together with CodeReady Containers' public IP addresses, which can be obtained by running the `crc ip` command. - -## Troubleshooting - -If you have any problems, refer to the [Troubleshooting](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#troubleshooting) section of the general PubSub+ Kubernetes Documentation for help. Substitute any `kubectl` commands with `oc` commands. - -If you need to start over, follow the steps to [delete the current deployment](/docs/PubSubPlusOpenShiftDeployment.md#deleting-a-deployment). +From here follow the steps in the [Solace PubSub+ Event Broker Operator Quick Start Guide](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart#3-pubsub-software-event-broker-deployment-examples) to deploy a single-node or an HA event broker. +>Note: the Operator recognizes the OpenShift environment and adjusts the default deployment `spec` parameters for the event broker, including the use of certified RedHat images. For more information, refer to the [detailed documentation](docs/PubSubPlusOpenShiftDeployment.md#broker-spec-defaults-in-openshift) in this repo. ## Contributing diff --git a/docs/PubSubPlusOpenShiftDeployment.md b/docs/PubSubPlusOpenShiftDeployment.md index a039b52..5cc08ab 100644 --- a/docs/PubSubPlusOpenShiftDeployment.md +++ b/docs/PubSubPlusOpenShiftDeployment.md @@ -1,105 +1,65 @@ -# Deploying a Solace PubSub+ Software Event Broker onto an OpenShift 4 Platform - -Solace PubSub+ Software Event Broker meets the needs of big data, cloud migration, and Internet-of-Things initiatives, and enables microservices and event-driven architecture. Capabilities include topic-based publish/subscribe, request/reply, message queues/queueing, and data streaming for IoT devices and mobile/web apps. The event broker supports open APIs and standard protocols including AMQP, JMS, MQTT, REST, and WebSocket. As well, it can be deployed in on-premises datacenters, natively within private and public clouds, and across complex hybrid cloud environments. - -This repository provides an example of how to deploy the Solace PubSub+ Software Event Broker onto an OpenShift 4 platform, including the steps to set up a Red Hat OpenShift Container Platform platform on AWS. - - - -## Overview -There are [multiple ways](https://www.openshift.com/try ) to get to an OpenShift platform. This example uses the Red Hat OpenShift Container Platform for deploying an HA group of software event brokers, but the concepts are transferable to other compatible platforms. We also provide tips for how to set up a simple single-node deployment using [CodeReady Containers](https://developers.redhat.com/products/codeready-containers/overview ) (the equivalent of MiniShift for OpenShift 4) for development, testing, or proof of concept purposes. - -The supported Solace PubSub+ Software Event Broker version is 9.10 or later. - -For the Red Hat OpenShift Container Platform, we use a self-managed 60-day evaluation subscription of [RedHat OpenShift cluster in AWS](https://cloud.redhat.com/openshift/install#public-cloud ) in a highly redundant configuration, spanning three zones. - -This repository expands on the [Solace Kubernetes Quickstart](//github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/README.md ) to provide an example of how to deploy Solace PubSub+ in an HA configuration on the OpenShift Container Platform running in AWS. - -The event broker deployment does not require any special OpenShift Security Context; the default ["restricted" SCC](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html ) can be used. - - -### Related Information - -You might also be interested in one of the following: -- For a hands-on quick start using an existing OpenShift platform, refer to the [Quick Start guide](/README.md). -- For considerations about deploying in a general Kubernetes environment, refer to the [Solace PubSub+ on Kubernetes Documentation](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md) -- For the `pubsubplus` Helm chart configuration options, refer to the [PubSub+ Software Event Broker Helm Chart Reference](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/tree/master/pubsubplus#configuration). -- For OpenShift 3.11, refer to the [archived version of this project](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/tree/v1.1.1). - - -## Table of Contents -- [Production Deployment Architecture](#production-deployment-architecture) -- [Deployment Tools](#deployment-tools) - - [Helm Charts](#helm-charts) - - [OpenShift Templates](#openshift-templates) -- [Deploying Solace PubSub+ onto OpenShift / AWS](#deploying-solace-pubsub-onto-openshift-aws) - - [Step 1: (Optional / AWS) Deploy a Self-Managed OpenShift Container Platform onto AWS](#step-1-optional-aws-deploy-a-self-managed-openshift-container-platform-onto-aws) - - [Step 2: (Optional / ECR) Use a Private Image Registry](#step-2-optional-ECR-use-a-private-image-registry) - - [Step 3, Option 1: Deploy Using Helm](#step-3-option-1-deploy-using-helm) - - [Step 3, Option 2: Deploy Using OpenShift Templates](#step-3-option-2-deploy-using-openshift-templates) -- [Validating the Deployment](#validating-the-deployment) - - [Viewing the Bringup logs](#viewing-the-bringup-logs) -- [Gaining Admin and SSH Access to the Event Broker](#gaining-admin-and-ssh-access-to-the-event-broker) -- [Exposing PubSub+ Services](#exposing-pubsub-services) - - [Routes](#routes) - - [HTTP, no TLS](#http-no-tls) - - [HTTPS with TLS terminate at ingress](#https-with-tls-terminate-at-ingress) - - [HTTPS with TLS re-encrypt at ingress](#https-with-tls-re-encrypt-at-ingress) - - [General TCP over TLS with passthrough to broker](#general-tcp-over-tls-with-passthrough-to-broker) -- [Testing PubSub+ Services](#testing-pubsub-services) -- [Testing Data Access to the Event Broker](#testing-data-access-to-the-event-broker) -- [Deleting a Deployment](#deleting-a-deployment) - - [Delete the PubSub+ Deployment](#delete-the-pubsub-deployment) - - [Delete the AWS OpenShift Container Platform Deployment](#deleting-the-aws-openshift-container-platform-deployment) -- [Experimental: Using NFS for Persistent Storage](#experimental-using-nfs-for-persistent-storage) -- [Resources](#resources) - +# Deploying a Solace PubSub+ Software Event Broker Onto an OpenShift 4 Platform Using Operator + +This document provides platform-specific information for deploying the [Solace PubSub+ Software Event Broker](https://solace.com/products/event-broker/software/) on OpenShift, using the Solace PubSub+ Event Broker Operator (Operator). It complements and should be used together with the [Solace PubSub+ Event Broker Operator User Guide](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md), which has instructions for Kubernetes in general. + +Contents: +- [Deploying a Solace PubSub+ Software Event Broker Onto an OpenShift 4 Platform Using Operator](#deploying-a-solace-pubsub-software-event-broker-onto-an-openshift-4-platform-using-operator) + - [Production Deployment Architecture](#production-deployment-architecture) + - [OpenShift Platform Setup Examples](#openshift-platform-setup-examples) + - [Deploying a Production-Ready OpenShift Container Platform onto AWS](#deploying-a-production-ready-openshift-container-platform-onto-aws) + - [Deleting the AWS OpenShift Container Platform Deployment](#deleting-the-aws-openshift-container-platform-deployment) + - [Deploying CodeReady Containers for OpenShift](#deploying-codeready-containers-for-openshift) + - [Using a Private Image Registry for Broker and Prometheus Exporter Images](#using-a-private-image-registry-for-broker-and-prometheus-exporter-images) + - [Using AWS ECR with CodeReady Containers](#using-aws-ecr-with-codeready-containers) + - [Deployment Considerations](#deployment-considerations) + - [Broker Spec Defaults in OpenShift](#broker-spec-defaults-in-openshift) + - [Accessing Broker Services](#accessing-broker-services) + - [Routes](#routes) + - [HTTP With No TLS](#http-with-no-tls) + - [HTTPS With TLS (Terminate at Ingress)](#https-with-tls-terminate-at-ingress) + - [HTTPS with TLS (Re-encrypt at Ingress)](#https-with-tls-re-encrypt-at-ingress) + - [General TCP over TLS with Passthrough to Broker](#general-tcp-over-tls-with-passthrough-to-broker) + - [Security Considerations](#security-considerations) + - [Helm-based Deployment](#helm-based-deployment) + - [Exposing Metrics to Prometheus](#exposing-metrics-to-prometheus) + - [Broker Deployment in OpenShift Using the Operator](#broker-deployment-in-openshift-using-the-operator) + - [Quick Start](#quick-start) +- [Additional Resources](#additional-resources) +- [Appendix: Using NFS for Persistent Storage](#appendix-using-nfs-for-persistent-storage) ## Production Deployment Architecture The following diagram shows an example of an HA group deployment of PubSub+ software event brokers in AWS: + ![alt text](/docs/images/network_diagram.jpg "Network Diagram") -
The key parts to note in the diagram above are: - the three PubSub+ Container instances in OpenShift pods, deployed on OpenShift (worker) nodes -- the cloud load balancer exposing the event router's services and management interface +- the cloud load balancer exposing the event broker's services and management interface - the OpenShift master nodes(s) - the CLI console that hosts the `oc` OpenShift CLI utility client -## Deployment Tools +## OpenShift Platform Setup Examples -There are two options for tooling to use to deploy the Kubernetes cluster: Helm charts and OpenShift templates. +You can skip this section if you already have your own OpenShift environment available. -#### Helm Charts +There are [multiple ways](https://www.openshift.com/try ) to set up an OpenShift platform. This section provides a distributed production-ready example that uses the Red Hat OpenShift Container Platform for deploying an HA group of software event brokers, but the concepts are transferable to other compatible platforms. -The Kubernetes `Helm` tool allows great flexibility, allowing the process of event broker deployment to be automated through a wide range of configuration options including in-service rolling upgrade of the event broker. This example refers to the [Solace Kubernetes QuickStart project](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/tree/master ) for the Helm setting to use to deploy the event broker onto your OpenShift environment. +This section also give tips for how to set up a simple single-node deployment using [CodeReady Containers](https://developers.redhat.com/products/codeready-containers/overview ) (the equivalent of MiniShift for OpenShift 4) for development, testing, or proof of concept purposes. -#### OpenShift Templates +The last sub-section describes how to use a private image registry, such as AWS ECR, together with OpenShift. -> Deprecation warning: deploying using OpenShift Templates is being phased out and the templates in this quickstart will be no longer maintained. The recommended deployment method is to use Helm. If Helm cannot be used then refer to the [PubSub+ Kubernetes documentation](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#alternative-deployment-with-generating-templates-for-the-kubernetes-kubectl-tool) to generate deployment manifests. +### Deploying a Production-Ready OpenShift Container Platform onto AWS -You can directly use the OpenShift templates included in this project, without any additional tools, to deploy the event broker in a limited number of configurations. Follow the instructions for deploying using OpenShift templates in [Step 3, Option 2](#step-3-option-2-deploy-using-openshift-templates), below. - -## Deploying Solace PubSub+ onto OpenShift / AWS - -The following steps describe how to deploy an event broker onto an OpenShift environment. Optional steps are provided for: -- setting up a self-managed Red Hat OpenShift Container Platform on Amazon AWS infrastructure (marked as *Optional / AWS*) -- using AWS Elastic Container Registry to host the Solace PubSub+ Docker image (marked as *Optional / ECR*). - -**Tip:** You can skip Step 1 if you already have your own OpenShift environment available. - -> Note: If you are using CodeReady Containers, follow the [getting started instructions](https://developers.redhat.com/products/codeready-containers/getting-started) to stand up a working CodeReady Containers deployment that supports Linux, MacOS, and Windows. At the `crc start` step it is helpful to: have a local `pullsecret` file created; specify CPU and memory requirements, allowing 2 to 3 CPU and 2.5 to 7 GiB memory for CRC internal purposes (depending on your platform and CRC version); also specify a DNS server, for example: `crc start -p ./pullsecret -c 5 -m 11264 --nameserver 1.1.1.1`. - -### Step 1: (Optional / AWS) Deploy a Self-Managed OpenShift Container Platform onto AWS - -This step requires the following: +This procedure requires the following: - a free Red Hat account. You can create one [here](https://developers.redhat.com/login ), if needed. - a command console on your host platform with Internet access. The examples here are for Linux, but MacOS is also supported. -- a designated working directory for the OpenShift cluster installation. The automated install process creates files here that are required later for deleting the OpenShift cluster. - ``` - mkdir ~/workspace; cd ~/workspace - ``` +- a designated working directory for the OpenShift cluster installation. +>**Important:** The automated install process creates files here that are required later for deleting the OpenShift cluster. Use a dedicated directory and do not delete any temporary files. + +``` +mkdir ~/workspace; cd ~/workspace +``` To deploy the container platform in AWS, do the following: 1. If you haven't already, log in to your RedHat account. @@ -111,11 +71,11 @@ To deploy the container platform in AWS, do the following: tar -xvf openshift-install-linux.tar.gz # Adjust the filename if needed rm openshift-install-linux.tar.gz ``` -5. Run the utility to create an install configuration. Provide the necessary information at the prompts, including the Pull Secret from the RedHat instructions page. This will create the file `install-config.yaml` with the [installation configuration parameters](https://docs.openshift.com/container-platform/latest/installing/installing_aws/installing-aws-customizations.html#installation-aws-config-yaml_installing-aws-customizations), most importantly the configuration for the worker and master nodes. +5. Run the utility to create an install configuration. Provide the necessary information at the prompts, including the pull secret from the RedHat instructions page. The utility creates the `install-config.yaml` file with the [installation configuration parameters](https://docs.openshift.com/container-platform/latest/installing/installing_aws/installing-aws-customizations.html#installation-aws-config-yaml_installing-aws-customizations), most importantly the configuration for the worker and master nodes. ``` ./openshift-install create install-config --dir=. ``` -6. Edit the `install-config.yaml` file to update the worker node AWS machine type to meet the [minimum CPU and Memory requirements](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#cpu-and-memory-requirements) for the targeted PubSub+ Software Event Broker configuration. When you select an [EC2 instance type](https://aws.amazon.com/ec2/instance-types/), allow at least 1 CPU and 1 GiB memory for OpenShift purposes that cannot be used by the broker. The following is an example updated configuration: +6. Edit the `install-config.yaml` file to update the AWS machine type of the worker node to meet the [minimum CPU and Memory requirements](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#cpu-and-memory-requirements) for the targeted PubSub+ Software Event Broker configuration. When you select an [EC2 instance type](https://aws.amazon.com/ec2/instance-types/), allow at least 1 CPU and 1 GiB memory for OpenShift purposes that cannot be used by the broker. The following is an example of an updated configuration: ``` ... compute: @@ -141,31 +101,52 @@ To deploy the container platform in AWS, do the following: INFO Login to the console with user: "kubeadmin", and password: "CKGc9-XUT6J-PDtWp-d4DSQ" ``` 9. [Install](https://docs.openshift.com/container-platform/latest/installing/installing_aws/installing-aws-default.html#cli-installing-cli_installing-aws-default) the `oc` client CLI tool. -10. Verify that your cluster is working correctly by following the hints from step 8, including verifying access to the OpenShift web-console. +10. Verify that your cluster is working correctly by following the hints from Step 8, including verifying access to the OpenShift web-console. +#### Deleting the AWS OpenShift Container Platform Deployment -### Step 2: (Optional / ECR) Use a Private Image Registry +If you need to delete your [AWS OpenShift Container Platform deployment](#deploy-a-production-ready-openshift-container-platform-onto-aws), run the following commands: + +``` +cd ~/workspace +./openshift-install help # Check options +./openshift-install destroy cluster +``` + +These commands remove all resources of the deployment. + +### Deploying CodeReady Containers for OpenShift + +If you are using CodeReady Containers, follow the [getting started instructions](https://developers.redhat.com/products/codeready-containers/getting-started) to stand up a working CodeReady Containers deployment that supports Linux, MacOS, and Windows. + +At the `crc start` step it is helpful to: +* have a local copy of the OpenShift `pullsecret` file created; +* specify CPU and memory requirements, allowing 2 to 3 CPU and 2.5 to 7 GiB memory for CRC internal purposes (depending on your platform and CRC version); +* also specify a DNS server, for example: `crc start -p ./pullsecret -c 5 -m 11264 --nameserver 1.1.1.1`. + +### Using a Private Image Registry for Broker and Prometheus Exporter Images By default, the deployment scripts pull the Solace PubSub+ image from the [Red Hat containerized products catalog](https://catalog.redhat.com/software/container-stacks/search?q=solace). If the OpenShift worker nodes have Internet access, no further configuration is required. However, if you need to use a private image registry, such as AWS ECR, you must supply a pull secret to enable access to the registry. The steps that follow show how to use AWS ECR for the broker image. 1. Download a free trial of the the Solace PubSub+ Enterprise Evaluation Edition by going to the **Docker** section of the [Solace Downloads](https://solace.com/downloads/?fwp_downloads_types=pubsub-enterprise-evaluation) page, or obtain an image from Solace Support. -2. Push the broker image to the private registry. Follow the specific procedures for the registry you are using. For ECR, see [Using Amazon ECR with the AWS CLI](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html). - >Note: If you are advised to run `aws ecr get-login-password` as part of the "Authenticate to your registry" step and it fails, try running `$(aws ecr get-login --region --no-include-email)` instead. - ![alt text](/docs/images/ECR-Registry.png "ECR Registry") +2. Push the broker image to the private registry. Follow the specific procedures for the registry you are using. For ECR, see the diagram below as well as the instructions in [Using Amazon ECR with the AWS CLI](https://docs.aws.amazon.com/AmazonECR/latest/userguide/getting-started-cli.html).

+ ![alt text](/docs/images/ECR-Registry.png "ECR Registry")
+ >Note: If you are advised to run `aws ecr get-login-password` as part of the "Authenticate to your registry" step and it fails, try running `$(aws ecr get-login --region --no-include-email)` instead. + 3. Create a pull secret from the registry information in the Docker configuration. This assumes that the ECR login happened on the same machine: ``` oc create secret generic \ --from-file=.dockerconfigjson=$(readlink -f ~/.docker/config.json) \ --type=kubernetes.io/dockerconfigjson ``` -4. Use the pull secret you just created (``) in the deployment section, Step 3, below. +4. Use the pull secret you just created (``) in the broker deployment manifest. -For additional information, see the [Using private registries](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#using-private-registries) and [Using ImagePullSecrets for signed images](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#using-imagepullsecrets-for-signed-images) sections of the Solace Kubernetes Quickstart documentation. +For additional information, see the [Using private registries](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md#using-a-private-registry) section of the *Solace PubSub+ Event Broker Operator User Guide*. -#### Using CodeReady Containers -If you are using CodeReady Containers, you may need to perform a workaround if the ECR login fails on the console (e.g., on Windows). In this case, do the following: +#### Using AWS ECR with CodeReady Containers +If you are using CodeReady Containers, you might need to perform a workaround if the ECR login fails on the console (e.g., on Windows). In this case, do the following: 1. Log into the OpenShift node: `oc get node` 2. Run the `oc debug node/` command. 3. At the prompt, run the `chroot /host` command. @@ -179,350 +160,75 @@ If you are using CodeReady Containers, you may need to perform a workaround if t 5. Run `podman pull ` to load the image locally on the CRC node. After you exit the node, you can use your ECR image URL and tag for the deployment. There is no need for a pull secret in this case. -### Step 3, Option 1: Deploy using Helm - -Using Helm to deploy your cluster offers more flexibility in terms of event broker deployment options, compared to those offered by OpenShift templates (see [Option 2](#step-3-option-2-deploy-using-openshift-templates)). - -Additional information is provided in the following documents: -- [Solace PubSub+ on Kubernetes Deployment Guide](//github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md) -- [Kubernetes Deployment Quick Start Guide](//github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/README.md) - -This deployment uses PubSub+ Software Event Broker Helm charts for OpenShift. You can customize it by overriding the [default chart parameters](//github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/tree/master/pubsubplus#configuration). - -Consult the [Deployment Considerations](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#pubsub-software-event-broker-deployment-considerations) section of the general Event Broker in Kubernetes Documentation when planning your deployment. - -To use broker TLS ports, you'll need to [configure certificates on the broker](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#enabling-use-of-tls-to-access-broker-services) or add [OpenShift's service CA bundle to the PubSub+ service](https://docs.openshift.com/container-platform/latest/security/certificates/service-serving-certificate.html#add-service-certificate-apiservice_service-serving-certificate). - -PubSub+ Software Event Broker Helm charts for OpenShift differ from the general PubSub+ Helm charts: -* The `securityContext.enabled` parameter is set to `false` by default, indicating not to use the provided pod security context but to let OpenShift set it using SecurityContextConstraints (SCC). By default OpenShift will use the "restricted" SCC. -* By default the latest [Red Hat certified image](https://catalog.redhat.com/software/container-stacks/search?q=solace) of PubSub+ Standard Edition is used from `registry.connect.redhat.com`. Use a different image tag if required or [use an image from a different registry](#step-2-optional--ecr-use-a-private-image-registry). If you're using a different image, add the `image.repository=,image.tag=` values (comma-separated) to the `--set` commands below. Also specify a pull secret if required: `image.pullSecretName=`. Consult the [Red Hat Knowledgebase](https://access.redhat.com/RegistryAuthentication#registry-service-accounts-for-shared-environments-4) if you run into issues with pulling the PubSub+ image. - -The broker can be [vertically scaled](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#deployment-scaling ) using the `solace.size` chart parameter. - -#### Steps: -1. Install Helm. Use the [instructions from Helm](//github.com/helm/helm#install), or if you're using Linux simply run the following command: - ```bash - curl https://raw.githubusercontent.com/helm/helm/master/scripts/get-helm-3 | bash - ``` - Helm is configured properly if the command `helm version` returns no error. -2. Create a new project or switch to your existing project (do not use the `default` project as its loose permissions don't reflect a typical OpenShift environment): - ``` - oc new-project solace-pubsubplus # adjust your project name as needed here and in subsequent commands - ``` -3. Follow one of the examples below to deploy your cluster. - - ##### For an _HA_ Deployment: - ```bash - # One-time action: Add the PubSub+ charts to local Helm - helm repo add openshift-helm-charts https://charts.openshift.io/ - # Initiate the HA deployment - specify an admin password - helm install my-ha-release \ - --set solace.redundancy=true,solace.usernameAdminPassword= \ - openshift-helm-charts/solace-pubsubplus-openshift - # Check the notes printed on screen - # Wait until all pods are running, ready, and the active event broker pod label is "active=true" - oc get pods --show-labels -w - ``` - - ##### For a Single-Node, _Non-HA_ Deployment (Using a _Pull_ _Secret_): - ```bash - # One-time action: Add the PubSub+ charts to local Helm - helm repo add openshift-helm-charts https://charts.openshift.io/ - # Initiate the non-HA deployment - specify an admin password - helm install my-nonha-release \ - --set solace.redundancy=false,solace.usernameAdminPassword= \ - --set image.pullSecretName= \ - openshift-helm-charts/solace-pubsubplus-openshift - # Check the notes printed on screen - # Wait until the event broker pod is running, ready, and the pod label is "active=true" - oc get pods --show-labels -w - ``` - - **Note**: As an alternative to longer `--set` parameters, it is possible to define the same parameter values in a YAML file: - ```yaml - # Create example values file - specify an admin password - echo " - solace: - redundancy: false, - usernameAdminPassword: " > deployment-values.yaml - # Use values file - helm install my-release \ - -f deployment-values.yaml \ - openshift-helm-charts/solace-pubsubplus-openshift - ``` - -### Step 3, Option 2: Deploy Using OpenShift Templates - -This option use an OpenShift template and doesn't require Helm. This option assumes that you have completed [Step 2](#step-2-optional-using-a-private-image-registry) if required. - -#### About the Template: -- You can copy templates files from the GitHub location to your local disk, edit them, and use them from there. -- Before you deploy, ensure that you determine your event broker [disk space requirements](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#disk-storage). The `BROKER_STORAGE_SIZE` parameter in the template has a default value of 30 gigabytes of disk space. You may need to update this value. -- By default, the template provisions a broker supporting 100 connections. You can adjust `export system_scaling_maxconnectioncount` in the template to increase the number of connections, but you must also ensure that adequate resources are available to the pod(s) by adjusting both `cpu` and `memory` requests and limits. For details, refer to the [System Resource Requirements](https://docs.solace.com/Configuring-and-Managing/SW-Broker-Specific-Config/System-Resource-Requirements.htm) in the Solace documentation. -- If using you are using [TLS to access broker services](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#enabling-use-of-tls-to-access-broker-services), you must configure a server key and certificate on the broker(s). Uncomment the related parts of the template file in your local copy and also specify a value for the `BROKER_TLS_CERT_SECRET` parameter. - - -#### Steps: - -1. Define a strong password for the 'admin' user of the event broker, and then base64 encode the value: - ``` - echo -n 'strong@dminPw!' | base64 - ``` - You will use this value as a parameter when you process the event broker OpenShift template. -2. Create a new project or switch to your existing project (do not use the `default` project as its loose permissions don't reflect a typical OpenShift environment): - ``` - oc new-project solace-pubsubplus # adjust your project name as needed here and in subsequent commands - ``` -3. Follow one of the examples below to deploy your cluster. - - - ##### For a Single-Node, _Non-HA_ Deployment: - This example uses all default values. You can omit default parameters. - - ``` - oc process -f https://raw.githubusercontent.com/SolaceProducts/pubsubplus-openshift-quickstart/master/templates/eventbroker_singlenode_template.yaml \ - DEPLOYMENT_NAME=test-singlenode \ - BROKER_ADMIN_PASSWORD= | oc create -f - - # Wait until all pods are running and ready - oc get pods -w --show-labels - ``` - - ##### For an _HA_ Deployment: - In this example, we specify values for all parameters. - - The `BROKER_IMAGE_REGISTRY_URL` and `BROKER_IMAGE_TAG` parameters default to **registry.connect.redhat.com/solace/pubsubplus-standard** and **latest**, respectively. - - ``` - oc process -f https://raw.githubusercontent.com/SolaceProducts/pubsubplus-openshift-quickstart/master/templates/eventbroker_ha_template.yaml \ - DEPLOYMENT_NAME=test-ha \ - BROKER_IMAGE_REGISTRY_URL= \ - BROKER_IMAGE_TAG= \ - BROKER_IMAGE_REGISTRY_PULLSECRET= - BROKER_STORAGE_SIZE=30Gi \ - BROKER_TLS_CERT_SECRET= # See notes above \ - BROKER_ADMIN_PASSWORD= | oc create -f - - # Wait until all pods are running and ready - oc get pods -w --show-labels - ``` - - -## Validating the Deployment - -If you encounter any issues with your deployment, refer to the [Kubernetes Troubleshooting Guide](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#troubleshooting) for help. Substitute any `kubectl` commands with `oc` commands. Before retrying a deployment, ensure to delete PVCs remaining from the unsuccessful deployment. Use the `oc get pvc` command to obtain a listing. - -From the console, validate your deployment by running the following command: -``` -$ oc get statefulset,service,pods,pvc,pv --show-labels -``` -The output should look like the following: -``` -NAME READY AGE LABELS -statefulset.apps/my-release-pubsubplus 3/3 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus,helm.sh/chart=pubsubplus-2.4.0 - -NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE LABELS -service/my-release-pubsubplus LoadBalancer 172.30.129.136 ac4917b2be7734df09a296f5da4dce38-1140440410.eu-central-1.elb.amazonaws.com 2222:31020/TCP,8080:30035/TCP,1943:30695/TCP,55555:30166/TCP,55003:30756/TCP,55443:32303/TCP,55556:31861/TCP,8008:31233/TCP,1443:32104/TCP,9000:30811/TCP,9443:30173/TCP,5672:31234/TCP,5671:31165/TCP,1883:32291/TCP,8883:32292/TCP,8000:32086/TCP,8443:31426/TCP 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus,helm.sh/chart=pubsubplus-2.4.0 -service/my-release-pubsubplus-discovery ClusterIP None 8080/TCP,8741/TCP,8300/TCP,8301/TCP,8302/TCP 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/managed-by=Helm,app.kubernetes.io/name=pubsubplus,helm.sh/chart=pubsubplus-2.4.0 - -NAME READY STATUS RESTARTS AGE LABELS -pod/my-release-pubsubplus-0 1/1 Running 0 23h active=true,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus,controller-revision-hash=my-release-pubsubplus-68d69ffb5,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-0 -pod/my-release-pubsubplus-1 1/1 Running 0 23h active=false,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus,controller-revision-hash=my-release-pubsubplus-68d69ffb5,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-1 -pod/my-release-pubsubplus-2 1/1 Running 0 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus,controller-revision-hash=my-release-pubsubplus-68d69ffb5,statefulset.kubernetes.io/pod-name=my-release-pubsubplus-2 - -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE LABELS -persistentvolumeclaim/data-my-release-pubsubplus-0 Bound pvc-eb2c8a52-85d4-4bc2-a73d-884559a4e463 10Gi RWO gp2 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus -persistentvolumeclaim/data-my-release-pubsubplus-1 Bound pvc-ab428fa6-4786-4419-a814-a801a0860861 10Gi RWO gp2 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus -persistentvolumeclaim/data-my-release-pubsubplus-2 Bound pvc-3d77864d-3f90-42fe-939d-8a9324a62e20 10Gi RWO gp2 23h app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus - -NAME CAPACITY ACCESS MODES RECLAIM POLICY STATUS CLAIM STORAGECLASS REASON AGE LABELS -persistentvolume/pvc-3d77864d-3f90-42fe-939d-8a9324a62e20 10Gi RWO Delete Bound solace-pubsubplus/data-my-release-pubsubplus-2 gp2 23h failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1a -persistentvolume/pvc-ab428fa6-4786-4419-a814-a801a0860861 10Gi RWO Delete Bound solace-pubsubplus/data-my-release-pubsubplus-1 gp2 23h failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1c -persistentvolume/pvc-eb2c8a52-85d4-4bc2-a73d-884559a4e463 10Gi RWO Delete Bound solace-pubsubplus/data-my-release-pubsubplus-0 gp2 23h failure-domain.beta.kubernetes.io/region=eu-central-1,failure-domain.beta.kubernetes.io/zone=eu-central-1b - -[ec2-user@ip-10-0-23-198 ~]$ -[ec2-user@ip-10-0-23-198 ~]$ -[ec2-user@ip-10-0-23-198 ~]$ oc describe svc my-release-pubsubplus -Name: my-release-pubsubplus -Namespace: solace-pubsubplus -Labels: app.kubernetes.io/instance=my-release - app.kubernetes.io/managed-by=Helm - app.kubernetes.io/name=pubsubplus - helm.sh/chart=pubsubplus-2.4.0 -Annotations: meta.helm.sh/release-name: my-release - meta.helm.sh/release-namespace: solace-pubsubplus -Selector: active=true,app.kubernetes.io/instance=my-release,app.kubernetes.io/name=pubsubplus -Type: LoadBalancer -IP: 172.30.129.136 -LoadBalancer Ingress: ac4917b2be7734df09a296f5da4dce38-1140440410.eu-central-1.elb.amazonaws.com -Port: tcp-ssh 2222/TCP -TargetPort: 2222/TCP -NodePort: tcp-ssh 31020/TCP -Endpoints: 10.129.2.14:2222 -Port: tcp-semp 8080/TCP -TargetPort: 8080/TCP -NodePort: tcp-semp 30035/TCP -Endpoints: 10.129.2.14:8080 -Port: tls-semp 1943/TCP -TargetPort: 1943/TCP -NodePort: tls-semp 30695/TCP -Endpoints: 10.129.2.14:1943 -Port: tcp-smf 55555/TCP -TargetPort: 55555/TCP -NodePort: tcp-smf 30166/TCP -Endpoints: 10.129.2.14:55555 -Port: tcp-smfcomp 55003/TCP -TargetPort: 55003/TCP -NodePort: tcp-smfcomp 30756/TCP -Endpoints: 10.129.2.14:55003 -Port: tls-smf 55443/TCP -TargetPort: 55443/TCP -NodePort: tls-smf 32303/TCP -Endpoints: 10.129.2.14:55443 -Port: tcp-smfroute 55556/TCP -TargetPort: 55556/TCP -NodePort: tcp-smfroute 31861/TCP -Endpoints: 10.129.2.14:55556 -Port: tcp-web 8008/TCP -TargetPort: 8008/TCP -NodePort: tcp-web 31233/TCP -Endpoints: 10.129.2.14:8008 -Port: tls-web 1443/TCP -TargetPort: 1443/TCP -NodePort: tls-web 32104/TCP -Endpoints: 10.129.2.14:1443 -Port: tcp-rest 9000/TCP -TargetPort: 9000/TCP -NodePort: tcp-rest 30811/TCP -Endpoints: 10.129.2.14:9000 -Port: tls-rest 9443/TCP -TargetPort: 9443/TCP -NodePort: tls-rest 30173/TCP -Endpoints: 10.129.2.14:9443 -Port: tcp-amqp 5672/TCP -TargetPort: 5672/TCP -NodePort: tcp-amqp 31234/TCP -Endpoints: 10.129.2.14:5672 -Port: tls-amqp 5671/TCP -TargetPort: 5671/TCP -NodePort: tls-amqp 31165/TCP -Endpoints: 10.129.2.14:5671 -Port: tcp-mqtt 1883/TCP -TargetPort: 1883/TCP -NodePort: tcp-mqtt 32291/TCP -Endpoints: 10.129.2.14:1883 -Port: tls-mqtt 8883/TCP -TargetPort: 8883/TCP -NodePort: tls-mqtt 32292/TCP -Endpoints: 10.129.2.14:8883 -Port: tcp-mqttweb 8000/TCP -TargetPort: 8000/TCP -NodePort: tcp-mqttweb 32086/TCP -Endpoints: 10.129.2.14:8000 -Port: tls-mqttweb 8443/TCP -TargetPort: 8443/TCP -NodePort: tls-mqttweb 31426/TCP -Endpoints: 10.129.2.14:8443 -Session Affinity: None -External Traffic Policy: Cluster -Events: -``` - -Find the **'LoadBalancer Ingress'** value listed in the service description above. This is the publicly accessible Solace Connection URI for messaging clients and management. In the example, it is `ac4917b2be7734df09a296f5da4dce38-1140440410.eu-central-1.elb.amazonaws.com`. - -> **Note**: There is no external Load Balancer support with CodeReady Containers. Services are accessed through NodePorts instead. To access the brokers, use the NodePort port numbers together with the CodeReady Containers' public IP addresses, which can be obtained by running the `crc ip` command. - -### Viewing the Bringup Logs - -To see the deployment events, navigate to: - -- **OpenShift UI > (Your Project) > Applications > Stateful Sets > ((name)-pubsubplus) > Events** +## Deployment Considerations -You can access the log stack for individual event broker pods from the OpenShift UI, by navigating to: +Consult the [Deployment Planning](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md#deployment-planning) section of the general *Solace PubSub+ Event Broker Operator User Guide* when planning your deployment. -- **OpenShift UI > (Your Project) > Applications > Stateful Sets > ((name)-pubsubplus) > Pods > ((name)-solace-(N)) > Logs** +The following sections apply only to the OpenShift platform. - Where **(N)** above is the ordinal of the HA role of the PubSub+ broker: - - 0: Primary event broker - - 1: Backup event broker - - 2: Monitor event broker +### Broker Spec Defaults in OpenShift -![alt text](/docs/images/Solace-Pod-Log-Stack.png "Event Broker Pod Log Stack") - - -## Gaining Admin and SSH Access to the Event Broker - -To access the event brokers, use the Solace Connection URI associated with the load balancer generated by the OpenShift template. As described in the introduction, you access the brokers through the load balancer service, which always point to the active event broker. The default port is 2222 for CLI and 8080 for SEMP/[Solace PubSub+ Broker Manager](https://docs.solace.com/Solace-PubSub-Manager/PubSub-Manager-Overview.htm). - -If you deployed OpenShift in AWS, then the Solace OpenShift QuickStart will have created an EC2 Load Balancer to front the event broker / OpenShift service. The Load Balancer public DNS name can be found in the AWS EC2 console in the 'Load Balancers' section. - -To launch Solace CLI or SSH into the individual event broker instances from the OpenShift CLI, use the following commands: - -``` -# CLI access -oc exec -it XXX-XXX-pubsubplus-X -- cli # adjust pod name to your deployment -# shell access -oc exec -it XXX-XXX-pubsubplus-X -- bash # adjust pod name to your deployment -``` +The Operator detects (1) whether the OpenShift platform is used and (2) the name of the OpenShift project (namespace) for the broker deployment. It automatically adjusts the default values for the parameters listed in the table below. You can override the defaults by explicitly specifying the parameters. -You can also gain access to the Solace CLI and container shell for individual event broker instances from the OpenShift UI. A web-based terminal emulator is available from the OpenShift UI. Navigate to an individual event broker Pod using the OpenShift UI: +| OpenShift Project (Namespace) | Broker Spec Parameter | General Kubernetes Defaults (for information only) | OpenShift Defaults | +| --- | --- | --- | --- | +| Any, excluding `default`.

**Note:** We recommend that you do NOT use the `default` project. | `spec.securityContext.runAsUser` | 1000001| Not set (OpenShift sets it according to the OpenShift project settings) | +|| `spec.securityContext.fsGroup` | 1000002 | Not set (OpenShift sets it according to the OpenShift project settings) | +| `default`

**Note:** Not recommended | `spec.securityContext.runAsUser` | 1000001 | 1000001 | +|| `spec.securityContext.fsGroup` | 1000002 | 1000002 | +| All OpenShift projects | `spec.image` | solace/solace-pubsub-standard | registry.connect.redhat.com/solace/pubsubplus-standard | +| | `spec.monitoring.image` | solace/solace-prometheus-exporter | registry.connect.redhat.com/solace/pubsubplus-prometheus-exporter | -- **OpenShift UI > (Your Project) > Applications > Stateful Sets > ((name)-pubsubplus) > Pods > ((name)-pubsubplus-(N)) > Terminal** +Although `runAsUser` cannot be configured using a broker spec parameter, the Operator similarly adjusts the `runAsUser` settings for the Prometheus exporter pod. -Once you have launched the terminal emulator to the event broker pod you can access the Solace CLI by executing the following command: +### Accessing Broker Services -``` -/usr/sw/loads/currentload/bin/cli -A -``` - -![alt text](/docs/images/Solace-Primary-Pod-Terminal-CLI.png "Event Broker CLI via OpenShift UI Terminal emulator") - -See the [Solace Kubernetes Quickstart README](//github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#gaining-admin-access-to-the-event-broker ) for more details, including admin and SSH access to the individual event brokers. - -## Exposing PubSub+ Services - -The principles of exposing services described in the [PubSub+ in Kubernetes documentation](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#exposing-the-pubsub-software-event-broker-services) apply: +The principles for exposing services that are described in the [Solace PubSub+ Event Broker Operator User Guide](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md#accessing-broker-services) also apply here: * LoadBalancer is the default service type and can be used to externally expose all broker services. This is an option for OpenShift as well and will not be further discussed here. * Ingress and its equivalent, OpenShift Routes, can be used to expose specific services. -### Routes +#### Routes - OpenShift has a default production-ready [ingress controller setup based on HAProxy](https://docs.openshift.com/container-platform/latest/networking/understanding-networking.html#nw-ne-openshift-ingress_understanding-networking). Using Routes is the recommended OpenShift-native way to configure Ingress. Refer to the OpenShift documentation for [more information on Ingress and Routes](https://docs.openshift.com/container-platform/latest/networking/understanding-networking.html#nw-ne-openshift-ingress_understanding-networking) and [how to configure Routes](https://docs.openshift.com/container-platform/latest/networking/routes/route-configuration.html). + OpenShift has a default production-ready [ingress controller setup](https://docs.openshift.com/container-platform/latest/networking/understanding-networking.html#nw-ne-openshift-ingress_understanding-networking) based on HAProxy. Using Routes is the recommended OpenShift-native way to configure Ingress. Refer to the OpenShift documentation for more information on [Ingress and Routes](https://docs.openshift.com/container-platform/latest/networking/understanding-networking.html#nw-ne-openshift-ingress_understanding-networking) and [how to configure Routes](https://docs.openshift.com/container-platform/latest/networking/routes/route-configuration.html). - The same [table provided for Ingress in the Kubernetes quickstart](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#using-ingress-to-access-event-broker-services) applies to PubSub+ services vs. route types: HTTP-type broker services can be exposed with TLS edge-terminated or re-encrypt, or without TLS. General TCP services can be exposed using TLS-passthrough to the broker Pods. + The same table provided for Ingress in the [Solace Kubernetes Quickstart](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/master/docs/PubSubPlusK8SDeployment.md#using-ingress-to-access-event-broker-services) applies here. HTTP-type broker services can be exposed with TLS (edge-terminated or re-encrypt) or without TLS. General TCP services can be exposed using TLS-passthrough to the broker Pods. - The controller's external (router default) IP address can be determined from looking up the external-IP of the `router-default` service, by running `oc get svc -n openshift-ingress`. OpenShift can automatically assign DNS-resolvable unique host names and TLS-certificates when using Routes (except for TLS-passthrough). It is also possible to assign to the services user-defined host names, for which the user must ensure they DNS-resolve to the router IP, and related TLS-certificates include those hostnames in the CN and/or SAN fields. Note: if a PubSub+ service client requires hostnames provided in the SAN field then user-defined TLS certificates must be used as OpenShift-generated certificates only use CN. + The controller's external (router default) IP address can be determined from looking up the external-IP of the `router-default` service, by running `oc get svc -n openshift-ingress`. OpenShift can automatically assign DNS-resolvable unique host names and TLS-certificates when using Routes (except for TLS-passthrough). It is also possible to assign user-defined host names to the services, but you must ensure that they DNS-resolve to the router IP, and that any related TLS-certificates include those hostnames in the CN and/or SAN fields. + +>Note: If a PubSub+ service client requires that hostnames are provided in the SAN field, you must use user-defined TLS certificates, because OpenShift-generated certificates only use the CN field. -The followings provide examples for each router type. Replace `` with the name of the service of your deployment. The port name must match the `service.ports` name in the PubSub+ `values.yaml` file. -Additional services can be exposed by additional route for each. +The following sections provide examples for each router type. Replace `` with the name of the service of your deployment. The port name must match the `service.ports` name in the PubSub+ `values.yaml` file. +Additional services can be exposed by an additional route for each. -##### HTTP, no TLS +##### HTTP With No TLS -This will create an HTTP route to the REST service at path `/`: +The following commands create an HTTP route to the REST service at path `/`: ```bash oc expose svc --port tcp-rest \ --name my-broker-rest-service --path / # Query the route to get the generated host for accessing the service oc get route my-broker-rest-service -o template --template='{{.spec.host}}' ``` -External requests shall be targeted to the host at the HTTP port (80) and the specified path. +External requests are targeted to the host at the HTTP port (80) and the specified path. -##### HTTPS with TLS terminate at ingress +##### HTTPS With TLS (Terminate at Ingress) Terminating TLS at the router is called "edge" in OpenShift. The target port is the backend broker's non-TLS service port. ```bash oc create route edge my-broker-rest-service-tls-edge \ --service \ --port tcp-rest \ - --path / # path is optional and shall not be used for SEMP service + --path / # path is optional and must not be used for SEMP service # Query the route to get the generated host for accessing the service oc get route my-broker-rest-service-tls-edge -o template --template='{{.spec.host}}' ``` -External requests shall be targeted to the host at the TLS port (443) and the specified path. +External requests are targeted to the host at the TLS port (443) and the specified path. -> Note: above will use OpenShift's generated TLS certificate which is self-signed by default and includes a wildcard hostname in the CN field. To use user-defined TLS certificates with more control instead, refer to the [OpenShift documentation](https://docs.openshift.com/container-platform/latest/networking/routes/secured-routes.html#nw-ingress-creating-an-edge-route-with-a-custom-certificate_secured-routes) +> Note: The example above uses OpenShift's generated TLS certificate, which is self-signed by default and includes a wildcard hostname in the CN field. To use user-defined TLS certificates with more control instead, refer to the [OpenShift documentation](https://docs.openshift.com/container-platform/latest/networking/routes/secured-routes.html#nw-ingress-creating-an-edge-route-with-a-custom-certificate_secured-routes). -##### HTTPS with TLS re-encrypt at ingress +##### HTTPS with TLS (Re-encrypt at Ingress) -Re-encrypt requires TLS configured at the backend PubSub+ broker. The target port is now the broker's TLS service port. The broker's CA certificate must be provided in the `--dest-ca-cert` parameter, so the router can trust the broker. +Re-encrypt requires that TLS is configured on the backend PubSub+ broker. The target port is now the broker's TLS service port. The broker's CA certificate must be provided in the `--dest-ca-cert` parameter, so that the router can trust the broker. ```bash oc create route reencrypt my-broker-rest-service-tls-reencrypt \ --service \ @@ -534,9 +240,9 @@ oc get route my-broker-rest-service-tls-reencrypt -o template --template='{{.spe ``` The TLS certificate note in the previous section is also applicable here. -##### General TCP over TLS with passthrough to broker +##### General TCP over TLS with Passthrough to Broker -Passthrough requires TLS-certificate configured on the backend PubSub+ broker that validates all virtual host names for the services exposed, in the CN and/or SAN fields. +Passthrough requires a TLS-certificate configured on the backend PubSub+ broker that validates all virtual host names for the services exposed, in the CN and/or SAN fields. ```bash oc create route passthrough my-broker-smf-service-tls-passthrough \ @@ -544,73 +250,53 @@ oc create route passthrough my-broker-smf-service-tls-passthrough \ --port tls-smf \ --hostname smf.mybroker.com ``` -Here the example PubSub+ SMF messaging service can be accessed at `tcps://smf.mybroker.com:443`. Also, `smf.mybroker.com` must resolve to the router's external IP as discussed above and the broker certificate shall include `*.mybroker.com` in the CN and/or SAN fields. +Here the example PubSub+ SMF messaging service can be accessed at `tcps://smf.mybroker.com:443`. Also, `smf.mybroker.com` must resolve to the router's external IP as discussed above and the broker certificate must include `*.mybroker.com` in the CN and/or SAN fields. The API client must support and use the SNI extension of the TLS handshake to provide the hostname to the OpenShift router for routing the request to the right backend broker. -## Testing PubSub+ Services - -A simple option for testing data traffic though the newly created event broker instance is the [SDKPerf tool](https://docs.solace.com/SDKPerf/SDKPerf.htm). Another option to quickly check messaging is [Try Me!](https://docs.solace.com/Solace-PubSub-Manager/PubSub-Manager-Overview.htm#Test-Messages), which is integrated into the [Solace PubSub+ Broker Manager](https://docs.solace.com/Solace-PubSub-Manager/PubSub-Manager-Overview.htm). - -To try building a client, visit the Solace Developer Portal and select your preferred programming language to [send and receive messages](http://dev.solace.com/get-started/send-receive-messages/ ). For each language there are samples that will help you get started. +### Security Considerations ->**Note**: The Host to be used is the Solace Connection URI. +The event broker deployment does not require any special OpenShift Security Context; the default most restrictive ["restricted-v2" SCC](https://docs.openshift.com/container-platform/latest/authentication/managing-security-context-constraints.html) can be used. -## Deleting a Deployment -You can delete just the PubSub+ deployment, or tear down your entire AWS OpenShift Container Platform. +### Helm-based Deployment -### Delete the PubSub+ Deployment +We recommend using the PubSub+ Event Broker Operator. An alternative method using Helm is described and available from an [earlier version of this repo](https://github.com/SolaceProducts/pubsubplus-openshift-quickstart/tree/v3.2.0). -To delete the deployment or to start over from Step 3 in a clean state, do the following: +## Exposing Metrics to Prometheus -- If you used [Step 3, Option 1 (Helm)](#step-3-option-1-deploy-using-helm) to deploy, execute the following commands: +OpenShift ships with an integrated customized Prometheus deployment, with the following restrictions: +* Monitoring must be enabled for user-defined projects. Only default platform monitoring is enabled by default. +* The Grafana UI has been removed in OpenShift 4.11. Only built-in Dashboards are available. - ``` - helm list # lists the releases (deployments) - helm delete XXX-XXX # deletes instances related to your deployment - "my-release" in the example above - ``` +Monitoring must be enabled for user-defined projects by [creating a `user-workload-monitoring-config` ConfigMap object](https://docs.openshift.com/container-platform/latest/monitoring/enabling-monitoring-for-user-defined-projects.html) in the `openshift-user-workload-monitoring` project. -- If you used [Step 3, Option 2 (OpenShift templates)](#step-3-option-2-deploy-using-openshift-templates) to deploy, run the following: +After this, the only step required to [connect the broker metrics with Prometheus](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md#connecting-with-prometheus) is to [create a ServiceMonitor object](https://github.com/SolaceProducts/pubsubplus-kubernetes-quickstart/blob/main/docs/EventBrokerOperatorUserGuide.md#creating-a-servicemonitor-object) in the project where the broker has been deployed. - ``` - oc process -f DEPLOYMENT_NAME= | oc delete -f - - ``` +Check the OpenShift admin console in "Administrator" view to verify that the monitoring endpoint for the event broker deployment has been connected to Prometheus: -> **Note:** The commands above do not delete the dynamic Persistent Volumes (PVs) and related Persistent Volume Claims (PVCs). If you recreate the deployment with the same name and keep the original PVCs, the original volumes will be mounted with the existing configuration. +![alt text](/docs/images/PrometheusTargets.png "Prometheus targets") -To delete the PVCs (which also deletes the PVs), run the following commands: +To enable custom Dashboards in the Grafana UI, you must install the community Grafana Operator from OpenShift's OperatorHub and then connect it to OpenShift Prometheus via a GrafanaDataSource. -``` -# List PVCs -oc get pvc -# Delete unneeded PVCs -oc delete pvc -``` +## Broker Deployment in OpenShift Using the Operator -To remove the project or to start over in a clean state, delete the project using the OpenShift console or the command line: -``` -oc delete project solace-pubsubplus # adjust your project name as needed -``` -For more details, refer to the [OpenShift Projects](https://docs.openshift.com/container-platform/latest/welcome/index.html) documentation. +### Quick Start -### Deleting the AWS OpenShift Container Platform Deployment +Refer to the [Quick Start Guide](/README.md) in the root of this repo. It provides information about [installing the Operator](/README.md#step-2-install-the-pubsub-event-broker-operator) and [deploying the PubSub+ Event Broker](/README.md#step-3-deploy-the-solace-pubsub-software-event-broker). -To delete your OpenShift Container Platform deployment that was set up at [Step 1](#step-1-optional--aws-deploy-a-self-managed-openshift-container-platform-onto-aws), run the following commands: +# Additional Resources -``` -cd ~/workspace -./openshift-install help # Check options -./openshift-install destroy cluster -``` - -This will remove all resources of the deployment. +For more information about Solace technology in general please visit these resources: +- The Solace Developer Portal website at: http://dev.solace.com +- Understanding [Solace technology.](http://dev.solace.com/tech/) +- Ask the [Solace community](http://dev.solace.com/community/). -## Experimental: Using NFS for Persistent Storage +# Appendix: Using NFS for Persistent Storage -> **Important:** This is only provided for information only as NFS is currently not supported for PubSub+ production deployment. +> **Important:** This section is provided for information only—NFS is currently not supported for PubSub+ production deployments. -The NFS server shall be configured with "root_squash" option. +The NFS server must be configured with the "root_squash" option. For an example deployment, specify the storage class from your NFS deployment ("nfs" in this example) in the `storage.useStorageClass` parameter and ensure `storage.slow` is set to `true`. @@ -621,7 +307,7 @@ The Helm (NFS Server Provisioner)[https://github.com/helm/charts/tree/master/sta ``` sudo oc apply -f https://raw.githubusercontent.com/kubernetes-incubator/external-storage/master/nfs/deploy/kubernetes/scc.yaml ``` -2. Install the NFS helm chart, which will create all dependencies: +2. Install the NFS helm chart, which creates all dependencies: ``` helm install stable/nfs-server-provisioner nfs-test --set persistence.enabled=true,persistence.size=100Gi ``` @@ -646,11 +332,3 @@ If you're using a template to deploy, locate the volume mount for `softAdb` in t # mountPath: /usr/sw/internalSpool/softAdb # subPath: softAdb ``` - -## Resources - -For more information about Solace technology in general please visit these resources: - -- The Solace Developer Portal website at: http://dev.solace.com -- Understanding [Solace technology.](http://dev.solace.com/tech/) -- Ask the [Solace community](http://dev.solace.com/community/). \ No newline at end of file diff --git a/docs/images/ECR-Registry.png b/docs/images/ECR-Registry.png index a60b5a0..6148970 100644 Binary files a/docs/images/ECR-Registry.png and b/docs/images/ECR-Registry.png differ diff --git a/docs/images/GetOpenShiftURL.png b/docs/images/GetOpenShiftURL.png index 77c7980..29121e8 100644 Binary files a/docs/images/GetOpenShiftURL.png and b/docs/images/GetOpenShiftURL.png differ diff --git a/docs/images/PrometheusTargets.png b/docs/images/PrometheusTargets.png new file mode 100644 index 0000000..eb192b9 Binary files /dev/null and b/docs/images/PrometheusTargets.png differ diff --git a/docs/images/network_diagram.jpg b/docs/images/network_diagram.jpg index c50ea88..1c8a755 100644 Binary files a/docs/images/network_diagram.jpg and b/docs/images/network_diagram.jpg differ diff --git a/templates/eventbroker_ha_template.yaml b/templates/eventbroker_ha_template.yaml deleted file mode 100644 index 9627ec8..0000000 --- a/templates/eventbroker_ha_template.yaml +++ /dev/null @@ -1,856 +0,0 @@ ---- -apiVersion: v1 -kind: Template -metadata: - name: pubsubplus-eventbroker-ha-template - annotations: - description: Deploys PubSub+ Event Broker in an HA configuration using persistent storage -objects: - -- kind: Secret - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-secrets" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus - type: Opaque - data: - username_admin_password: "${BROKER_ADMIN_PASSWORD}" - -- kind: ConfigMap - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus - data: - init.sh: |- - export username_admin_passwordfilepath="/mnt/disks/secrets/username_admin_password" - export username_admin_globalaccesslevel=admin - export service_ssh_port='2222' - export service_webtransport_port='8008' - export service_webtransport_tlsport='1443' - export service_semp_tlsport='1943' - export logging_debug_output=all - export system_scaling_maxconnectioncount="100" - # Uncomment if using TLS configuration - # cat /mnt/disks/certs/server/tls.key /mnt/disks/certs/server/tls.crt > /dev/shm/server.cert - # export tls_servercertificate_filepath="/dev/shm/server.cert" - IFS='-' read -ra host_array <<< $(hostname) - node_ordinal=${host_array[-1]} - if [[ ! -z `echo $STATEFULSET_NAMESPACE` ]]; then - namespace=`echo $STATEFULSET_NAMESPACE` - else - namespace=default - fi - service="${DEPLOYMENT_NAME}-pubsubplus" - # Deal with the fact we cannot accept "-" in routre names - service_name=$(echo ${service} | sed 's/-//g') - export routername=$(echo $(hostname) | sed 's/-//g') - export redundancy_enable=yes - export configsync_enable=yes - export redundancy_authentication_presharedkey_key=`cat /mnt/disks/secrets/username_admin_password | awk '{x=$0;for(i=length;i<51;i++)x=x "0";}END{print x}' | base64` # Right-pad with 0s to 50 length - export service_redundancy_firstlistenport='8300' - export redundancy_group_node_${service_name}0_nodetype=message_routing - export redundancy_group_node_${service_name}0_connectvia=${service}-0.${service}-discovery.${namespace}.svc:${service_redundancy_firstlistenport} - export redundancy_group_node_${service_name}1_nodetype=message_routing - export redundancy_group_node_${service_name}1_connectvia=${service}-1.${service}-discovery.${namespace}.svc:${service_redundancy_firstlistenport} - export redundancy_group_node_${service_name}2_nodetype=monitoring - export redundancy_group_node_${service_name}2_connectvia=${service}-2.${service}-discovery.${namespace}.svc:${service_redundancy_firstlistenport} - - case ${node_ordinal} in - 0) - export nodetype=message_routing - export redundancy_matelink_connectvia=${service}-1.${service}-discovery.${namespace}.svc - export redundancy_activestandbyrole=primary - ;; - 1) - export nodetype=message_routing - export redundancy_matelink_connectvia=${service}-0.${service}-discovery.${namespace}.svc - export redundancy_activestandbyrole=backup - ;; - 2) - export nodetype=monitoring - ;; - esac - - startup-broker.sh: |- - #!/bin/bash - APP=`basename "$0"` - IFS='-' read -ra host_array <<< $(hostname) - node_ordinal=${host_array[-1]} - echo "`date` INFO: ${APP}-Node ordinal: ${node_ordinal}" - echo "`date` INFO: ${APP}-Waiting for management API to become available" - password=`cat /mnt/disks/secrets/username_admin_password` - loop_guard=60 - pause=10 - count=0 - while [ ${count} -lt ${loop_guard} ]; do - if /mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 -t ; then - break - fi - run_time=$((${count} * ${pause})) - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Management API not yet accessible" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Solace Management API never came up" >&2 - exit 1 - fi - # Uncomment if using TLS configuration - # rm /dev/shm/server.cert # remove as soon as possible - # cert_results=$(curl --write-out '%{http_code}' --silent --output /dev/null -k -X PATCH -u admin:${password} https://localhost:1943/SEMP/v2/config/ \ - # -H "content-type: application/json" \ - # -d "{\"tlsServerCertContent\":\"$(cat /mnt/disks/certs/server/tls.key /mnt/disks/certs/server/tls.crt | awk '{printf "%s\\n", $0}')\"}") - # if [ "${cert_results}" != "200" ]; then - # echo "`date` ERROR: ${APP}-Unable to set the server certificate, exiting" >&2 - # exit 1 - # fi - # echo "`date` INFO: ${APP}-Server certificate has been configured" - # for non-monitor nodes setup redundancy and config-sync - if [ "${node_ordinal}" != "2" ]; then - resync_step="" - role="" - count=0 - while [ ${count} -lt ${loop_guard} ]; do - role_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/redundancy/active-standby-role[text()]"` - run_time=$((${count} * ${pause})) - case "`echo ${role_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -`" in - "Primary") - role="primary" - break - ;; - "Backup") - role="backup" - break - ;; - esac - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, got ${role_results} for this node's active-standby role" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Could not determine this node's active-standby role" >&2 - exit 1 - fi - # Determine local activity - count=0 - echo "`date` INFO: ${APP}-Management API is up, determined that this node's active-standby role is: ${role}" - while [ ${count} -lt ${loop_guard} ]; do - online_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/redundancy/virtual-routers/${role}/status/activity[text()]"` - local_activity=`echo ${online_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - run_time=$((${count} * ${pause})) - case "${local_activity}" in - "Local Active") - echo "`date` INFO: ${APP}-Node activity status is Local Active, after ${run_time} seconds" - # We should only be here on new cluster create, if not likely a bug - # Need to issue assert master to get back into sync" - resync_step="assert-master" - break - ;; - "Mate Active") - echo "`date` INFO: ${APP}-Node activity status is Mate Active, after ${run_time} seconds" - # This is normal state if we are backup or recreated later on - # will issue a resync master to get back into sync - resync_step="resync-master" - break - ;; - esac - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Local activity state is: ${local_activity}" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Local activity state never become Local Active or Mate Active" >&2 - exit 1 - fi - # If we need to assert master, then we need to wait for mate to reconcile - if [ "${resync_step}" = "assert-master" ]; then - count=0 - echo "`date` INFO: ${APP}-Waiting for mate activity state to be 'Standby'" - while [ ${count} -lt ${loop_guard} ]; do - online_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/redundancy/virtual-routers/${role}/status/detail/priority-reported-by-mate/summary[text()]"` - mate_activity=`echo ${online_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - run_time=$((${count} * ${pause})) - case "${mate_activity}" in - "Standby") - echo "`date` INFO: ${APP}-Activity state reported by mate is Standby, after ${run_time} seconds" - break - ;; - esac - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Mate activity state is: ${mate_activity}, not yet in sync" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Mate not in sync, never reached Standby" >&2 - exit 1 - fi - fi # if assert-master - # Ensure Config-sync connection state is Connected before proceeding - count=0 - echo "`date` INFO: ${APP}-Waiting for config-sync connected" - while [ ${count} -lt ${loop_guard} ]; do - online_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/config-sync/status/client/connection-state"` - connection_state=`echo ${online_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - run_time=$((${count} * ${pause})) - case "${connection_state}" in - "Connected") - echo "`date` INFO: ${APP}-Config-sync connection state is Connected, after ${run_time} seconds" - break - ;; - esac - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Config-sync connection state is: ${connection_state}, not yet in Connected" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Config-sync connection state never reached Connected" >&2 - exit 1 - fi - # Now can issue {resync_step} command - echo "`date` INFO: ${APP}-Initiating ${resync_step}" - /mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "<${resync_step}>" - /mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "<${resync_step}>*" - # Wait for config-sync results - count=0 - echo "`date` INFO: ${APP}-Waiting for config-sync connected" - while [ ${count} -lt ${loop_guard} ]; do - online_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/config-sync/status/oper-status"` - confsyncstatus_results=`echo ${online_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - run_time=$((${count} * ${pause})) - case "${confsyncstatus_results}" in - "Up") - echo "`date` INFO: ${APP}-Config-sync is Up, after ${run_time} seconds" - break - ;; - esac - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Config-sync is: ${confsyncstatus_results}, not yet Up" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Config-sync never reached state \"Up\"" >&2 - exit 1 - fi - fi # if not monitor - echo "`date` INFO: ${APP}-PubSub+ Event Broker bringup is complete for this node." - exit 0 - - - readiness_check.sh: |- - #!/bin/bash - APP=`basename "$0"` - LOG_FILE=/usr/sw/var/k8s_readiness_check.log # STDOUT/STDERR goes to k8s event logs but gets cleaned out eventually. This will also persist it. - tail -n 1000 ${LOG_FILE} > ${LOG_FILE}.tmp; mv -f ${LOG_FILE}.tmp ${LOG_FILE} || : # Limit logs size - exec > >(tee -a ${LOG_FILE}) 2>&1 # Setup logging - FINAL_ACTIVITY_LOGGED_TRACKING_FILE=/tmp/final_activity_state_logged - - # Function to read Kubernetes metadata labels - get_label () { - # Params: $1 label name - echo $(cat /etc/podinfo/labels | awk -F= '$1=="'${1}'"{print $2}' | xargs); - } - - # Function to set Kubernetes metadata labels - set_label () { - # Params: $1 label name, $2 label set value - #Prevent overdriving Kubernetes infra, don't set activity state to same as previous state - previous_state=$(get_label "active") - if [ "${2}" = "${previous_state}" ]; then - #echo "`date` INFO: ${APP}-Current and Previous state match (${2}), not updating pod label" - : - else - echo "`date` INFO: ${APP}-Updating pod label using K8s API from ${previous_state} to ${2}" - echo "[{\"op\": \"add\", \"path\": \"/metadata/labels/${1}\", \"value\": \"${2}\" }]" > /tmp/patch_label.json - K8S=https://kubernetes.default.svc.cluster.local:$KUBERNETES_SERVICE_PORT - KUBE_TOKEN=$(&2 - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - fi - fi - } - - # Main logic: note that there are no re-tries here, if check fails then return not ready. - # HA config - IFS='-' read -ra host_array <<< $(hostname) - node_ordinal=${host_array[-1]} - password=`cat /mnt/disks/secrets/username_admin_password` - - # For update (includes SolOS upgrade) purposes, additional checks are required for readiness state when the pod has been started - # This is an update if the LASTVERSION_FILE with K8s controller-revision-hash exists and contents differ from current value - LASTVERSION_FILE=/var/lib/solace/var/lastConfigRevisionBeforeReboot - if [ -f ${LASTVERSION_FILE} ] && [[ $(cat ${LASTVERSION_FILE}) != $(get_label "controller-revision-hash") ]] ; then - echo "`date` INFO: ${APP}-Upgrade detected, running additional checks..." - # Check redundancy - results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/redundancy/redundancy-status"` - redundancystatus_results=`echo ${results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - if [ "${redundancystatus_results}" != "Up" ]; then - echo "`date` INFO: ${APP}-Redundancy state is not yet up." - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - # Additionally check config-sync status for non-monitoring nodes - if [ "${node_ordinal}" != "2" ]; then - results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/config-sync/status/oper-status"` - confsyncstatus_results=`echo ${results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - if [ "${confsyncstatus_results}" != "Up" ]; then - echo "`date` INFO: ${APP}-Config-sync state is not yet up." - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - fi - fi - # Record current version in LASTVERSION_FILE - echo $(get_label "controller-revision-hash") > ${LASTVERSION_FILE} - # For monitor node just check for 3 online nodes in group; active label will never be set - if [ "${node_ordinal}" = "2" ]; then - role_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -c "/rpc-reply/rpc/show/redundancy/group-node/status[text() = \"Online\"]"` - if [[ ${role_results} != *""* ]]; then - errorinfo=`echo ${results} | xmllint -xpath "string(returnInfo/errorInfo)" - 2>/dev/null` || errorinfo= - echo "`date` INFO: ${APP}-Waiting for valid server status response, got ${errorinfo}" - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - nodes_online=`echo ${role_results} | xmllint -xpath "string(returnInfo/countSearchResult)" -` - if [ "$nodes_online" -eq "3" ]; then - if [ ! -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} ]; then - echo "`date` INFO: ${APP}-All nodes online, monitor node is redundancy ready" - touch ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} - fi - exit 0 - else - echo "`date` INFO: ${APP}-Monitor node is not redundancy ready, ${nodes_online} of 3 nodes online" - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - fi # End Monitor Node - # For Primary or Backup nodes set both service readiness (active label) and k8s readiness (exit return value) - health_result=`curl -s -o /dev/null -w "%{http_code}" http://localhost:5550/health-check/guaranteed-active` - case "${health_result}" in - "200") - if [ ! -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} ]; then - echo "`date` INFO: ${APP}-HA Event Broker health check reported 200, message spool is up" - touch ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} - fi - set_label "active" "true" - exit 0 - ;; - "503") - if [[ $(get_label "active") = "true" ]]; then echo "`date` INFO: ${APP}-HA Event Broker health check reported 503"; fi - set_label "active" "false" - # Further check is required to determine readiness - ;; - *) - echo "`date` WARN: ${APP}-HA Event Broker health check reported unexpected ${health_result}" - set_label "active" "false" - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - esac - # At this point analyzing readiness after health check returned 503 - checking if Event Broker is Standby - case "${node_ordinal}" in - "0") - config_role="primary" - ;; - "1") - config_role="backup" - ;; - esac - online_results=`/mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 \ - -q "" \ - -v "/rpc-reply/rpc/show/redundancy/virtual-routers/${config_role}/status/activity[text()]"` - local_activity=`echo ${online_results} | xmllint -xpath "string(returnInfo/valueSearchResult)" -` - case "${local_activity}" in - "Mate Active") - # Pass readiness check - if [ ! -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} ]; then - echo "`date` INFO: ${APP}-Redundancy is up and node is mate Active" - touch ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} - fi - exit 0 - ;; - *) - echo "`date` WARN: ${APP}-Health check returned 503 and local activity state is: ${local_activity}, failing readiness check." - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - ;; - esac - semp_query.sh: |- - #!/bin/bash - APP=`basename "$0"` - OPTIND=1 # Reset in case getopts has been used previously in the shell. - # Initialize our own variables: - count_search="" - name="" - password="" - query="" - url="" - value_search="" - test_connection_only=false - script_name=$0 - verbose=0 - while getopts "c:n:p:q:u:v:t" opt; do - case "$opt" in - c) count_search=$OPTARG - ;; - n) username=$OPTARG - ;; - p) password=$OPTARG - ;; - q) query=$OPTARG - ;; - u) url=$OPTARG - ;; - v) value_search=$OPTARG - ;; - t) test_connection_only=true - ;; - esac - done - shift $((OPTIND-1)) - [ "$1" = "--" ] && shift - verbose=1 - #echo "`date` INFO: ${APP}-${script_name}: count_search=${count_search} ,username=${username} ,password=xxx query=${query} \ - # ,url=${url} ,value_search=${value_search} ,Leftovers: $@" >&2 - if [[ ${url} = "" || ${username} = "" || ${password} = "" ]]; then - echo "`date` ERROR: ${APP}-${script_name}: url, username, password are madatory fields" >&2 - echo 'missing parameter' - exit 1 - fi - if [ "`curl --write-out '%{http_code}' --silent --output /dev/null -u ${username}:${password} ${url}/SEMP`" != "200" ] ; then - echo "management host is not responding" - exit 1 - fi - if [ "$test_connection_only" = true ] ; then - exit 0 # done here, connection is up - fi - query_response=`curl -sS -u ${username}:${password} ${url}/SEMP -d "${query}"` - # Validate first char of response is "<", otherwise no hope of being valid xml - if [[ ${query_response:0:1} != "<" ]] ; then - echo "no valid xml returned" - exit 1 - fi - query_response_code=`echo $query_response | xmllint -xpath 'string(/rpc-reply/execute-result/@code)' -` - - if [[ -z ${query_response_code} && ${query_response_code} != "ok" ]]; then - echo "query failed -${query_response_code}-" - exit 1 - fi - #echo "`date` INFO: ${APP}-${script_name}: query passed ${query_response_code}" >&2 - if [[ ! -z $value_search ]]; then - value_result=`echo $query_response | xmllint -xpath "string($value_search)" -` - echo "${value_result}" - exit 0 - fi - if [[ ! -z $count_search ]]; then - count_line=`echo $query_response | xmllint -xpath "$count_search" -` - count_string=`echo $count_search | cut -d '"' -f 2` - count_result=`echo ${count_line} | tr "><" "\n" | grep -c ${count_string}` - echo "${count_result}" - exit 0 - fi - -# Uncomment if using provider-specific StorageClass -# Also uncomment further down at volumeClaimTemplates -# This is an example for aws -#- kind: StorageClass -# apiVersion: storage.k8s.io/v1 -# metadata: -# name: "${DEPLOYMENT_NAME}-standard" -# provisioner: kubernetes.io/aws-ebs -# parameters: -# type: gp2 - -- kind: ServiceAccount - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-sa" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus #end gcp - -- kind: Role - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-podtagupdater" - rules: - - apiGroups: [""] # "" indicates the core API group - resources: ["pods"] - verbs: ["patch"] - -- kind: RoleBinding - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-serviceaccounts-to-podtagupdater" - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: "${DEPLOYMENT_NAME}-pubsubplus-podtagupdater" - subjects: - - kind: ServiceAccount - name: "${DEPLOYMENT_NAME}-pubsubplus-sa" - -- kind: Service - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-discovery" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus - annotations: - service.alpha.kubernetes.io/tolerate-unready-endpoints: "true" - spec: - ports: - - port: 8080 - name: tcp-semp - - port: 8741 - name: tcp-ha-mate-link - - port: 8300 - name: tcp-ha-conf-sync0 - - port: 8301 - name: tcp-ha-conf-sync1 - - port: 8302 - name: tcp-ha-conf-sync2 - clusterIP: None - selector: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - publishNotReadyAddresses: true -- kind: Service - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus #end gcp - spec: - type: LoadBalancer - ports: - - port: 2222 - targetPort: 2222 - protocol: TCP - name: tcp-ssh - - port: 8080 - targetPort: 8080 - protocol: TCP - name: tcp-semp - - port: 1943 - targetPort: 1943 - protocol: TCP - name: tls-semp - - port: 55555 - targetPort: 55555 - protocol: TCP - name: tcp-smf - - port: 55003 - targetPort: 55003 - protocol: TCP - name: tcp-smfcomp - - port: 55443 - targetPort: 55443 - protocol: TCP - name: tls-smf - - port: 55556 - targetPort: 55556 - protocol: TCP - name: tcp-smfroute - - port: 8008 - targetPort: 8008 - protocol: TCP - name: tcp-web - - port: 1443 - targetPort: 1443 - protocol: TCP - name: tls-web - - port: 9000 - targetPort: 9000 - protocol: TCP - name: tcp-rest - - port: 9443 - targetPort: 9443 - protocol: TCP - name: tls-rest - - port: 5672 - targetPort: 5672 - protocol: TCP - name: tcp-amqp - - port: 5671 - targetPort: 5671 - protocol: TCP - name: tls-amqp - - port: 1883 - targetPort: 1883 - protocol: TCP - name: tcp-mqtt - - port: 8883 - targetPort: 8883 - protocol: TCP - name: tls-mqtt - - port: 8000 - targetPort: 8000 - protocol: TCP - name: tcp-mqttweb - - port: 8443 - targetPort: 8443 - protocol: TCP - name: tls-mqttweb - selector: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - active: "true" - -- kind: StatefulSet - apiVersion: apps/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - spec: - selector: - matchLabels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - serviceName: "${DEPLOYMENT_NAME}-pubsubplus-discovery" - replicas: 3 - podManagementPolicy: Parallel - updateStrategy: - type: RollingUpdate - template: - metadata: - labels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - spec: - imagePullSecrets: - - name: ${BROKER_IMAGE_REGISTRY_PULLSECRET} - serviceAccountName: "${DEPLOYMENT_NAME}-pubsubplus-sa" - terminationGracePeriodSeconds: 1200 - containers: - - name: pubsubplus - image: "${BROKER_IMAGE_REGISTRY_URL}:${BROKER_IMAGE_TAG}" - imagePullPolicy: IfNotPresent - resources: - requests: - cpu: "2" - memory: 4025Mi - limits: - cpu: "2" - memory: 4025Mi - livenessProbe: - tcpSocket: - port: 8080 - initialDelaySeconds: 300 - timeoutSeconds: 5 - readinessProbe: - initialDelaySeconds: 30 - periodSeconds: 5 - exec: - command: - - /mnt/disks/solace/readiness_check.sh - securityContext: - privileged: false - env: - - name: STATEFULSET_NAME - value: "${DEPLOYMENT_NAME}-pubsubplus" - - name: STATEFULSET_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: TZ - value: :/usr/share/zoneinfo/UTC - - name: UMASK - value: "0022" - command: - - bash - - "-ec" - - | - source /mnt/disks/solace/init.sh - # not using postinstall hooks because of order dependencies - # launch config check then PubSub+ so VCMR can provide return code - nohup /mnt/disks/solace/startup-broker.sh & - /usr/sbin/boot.sh - - lifecycle: - preStop: - exec: - command: - - bash - - "-ec" - - | - while ! pgrep solacedaemon ; do sleep 1; done - killall solacedaemon; - while [ ! -d /usr/sw/var/db.upgrade ]; do sleep 1; done; - volumeMounts: - - name: podinfo - mountPath: /etc/podinfo - - name: config-map - mountPath: /mnt/disks/solace - - name: secrets - mountPath: /mnt/disks/secrets - readOnly: true - # Uncomment if using TLS configuration - # - name: server-certs - # mountPath: /mnt/disks/certs/server - # readOnly: true - - name: dshm - mountPath: /dev/shm - - name: data - mountPath: /usr/sw/jail - subPath: jail - - name: data - mountPath: /usr/sw/var - subPath: var - - name: data - mountPath: /usr/sw/internalSpool - subPath: internalSpool - - name: data - mountPath: /usr/sw/adb - subPath: adb - - name: data - mountPath: /var/lib/solace/diags - subPath: diags - # only mount softAdb when not using NFS, comment it out otherwise - - name: data - mountPath: /usr/sw/internalSpool/softAdb - subPath: softAdb - # use this instead if using NFS: - #- name: soft-adb-ephemeral - # mountPath: /usr/sw/internalSpool/softAdb - ports: - - containerPort: 2222 - protocol: TCP - - containerPort: 8080 - protocol: TCP - - containerPort: 1943 - protocol: TCP - - containerPort: 55555 - protocol: TCP - - containerPort: 55003 - protocol: TCP - - containerPort: 55443 - protocol: TCP - - containerPort: 55556 - protocol: TCP - - containerPort: 8008 - protocol: TCP - - containerPort: 1443 - protocol: TCP - - containerPort: 9000 - protocol: TCP - - containerPort: 9443 - protocol: TCP - - containerPort: 5672 - protocol: TCP - - containerPort: 5671 - protocol: TCP - - containerPort: 1883 - protocol: TCP - - containerPort: 8883 - protocol: TCP - - containerPort: 8000 - protocol: TCP - - containerPort: 8443 - protocol: TCP - volumes: - - name: podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - name: config-map - configMap: - name: "${DEPLOYMENT_NAME}-pubsubplus" - defaultMode: 0755 - - name: secrets - secret: - secretName: "${DEPLOYMENT_NAME}-pubsubplus-secrets" - defaultMode: 0400 - # Uncomment if using TLS configuration - # - name: server-certs - # secret: - # secretName: "${BROKER_TLS_CERT_SECRET}" - # defaultMode: 0400 - - name: dshm - emptyDir: - medium: Memory - # add this if using NFS (together with montPath changes for softADB - # - name: soft-adb-ephemeral - # emptyDir: {} - volumeClaimTemplates: - - metadata: - name: data - spec: - # Uncomment if using provider-specific StorageClass - # Also uncomment StorageClass definition above - # storageClassName: "${DEPLOYMENT_NAME}-standard" - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: "${BROKER_STORAGE_SIZE}" - -parameters: - - name: DEPLOYMENT_NAME - displayName: PubSub+ Event Broker Deployment Name - description: The prefix to use for object names - generate: expression - from: '[A-Z0-9]{8}' - value: example - required: true - - name: BROKER_IMAGE_REGISTRY_URL - displayName: Image Registry URL - description: The image registry URL for the registry containing the PubSub+ Event Broker docker image - value: registry.connect.redhat.com/solace/pubsubplus-standard - required: true - - name: BROKER_IMAGE_REGISTRY_PULLSECRET - displayName: Image Registry Pull Secret - description: The pull secret to be used to obtain the PubSub+ Event Broker docker image - value: - required: false - - name: BROKER_IMAGE_TAG - displayName: PubSub+ Event Broker Docker Image Tag - description: The Docker image tag for the PubSub+ Event Broker docker image from your Docker registry - value: latest - required: true - - name: BROKER_ADMIN_PASSWORD - displayName: Base64 encoded password for PubSub+ username 'admin' - description: The Event Broker 'admin' user's password (base64 encoded). This PubSub+ OpenShift template will create an administrative user with username 'admin' with specified password. - required: true - - name: BROKER_STORAGE_SIZE - displayName: PubSub+ Event Broker Persistent Storage Disk Size - description: The size in gigabytes for a Event Broker Pod's persistent volume (with suffix 'Gi'), example 30Gi for 30 gigabytes - value: 30Gi - required: true - # Uncomment if using TLS configuration - # - name: BROKER_TLS_CERT_SECRET - # displayName: Certificate Secret for Broker TLS Config - # description: The server key and certificate for the event broker packaged in a Kubernetes secret - # value: - # required: true diff --git a/templates/eventbroker_singlenode_template.yaml b/templates/eventbroker_singlenode_template.yaml deleted file mode 100644 index 6951087..0000000 --- a/templates/eventbroker_singlenode_template.yaml +++ /dev/null @@ -1,571 +0,0 @@ ---- -apiVersion: v1 -kind: Template -metadata: - name: pubsubplus-eventbroker-singlenode-template - annotations: - description: Deploys PubSub+ Event Broker in a Single Node configuration -objects: - -- kind: Secret - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-secrets" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus - type: Opaque - data: - username_admin_password: "${BROKER_ADMIN_PASSWORD}" - -- kind: ConfigMap - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus - data: - init.sh: |- - export username_admin_passwordfilepath="/mnt/disks/secrets/username_admin_password" - export username_admin_globalaccesslevel=admin - export service_ssh_port='2222' - export service_webtransport_port='8008' - export service_webtransport_tlsport='1443' - export service_semp_tlsport='1943' - export logging_debug_output=all - export system_scaling_maxconnectioncount="100" - # Uncomment if using TLS configuration - # cat /mnt/disks/certs/server/tls.key /mnt/disks/certs/server/tls.crt > /dev/shm/server.cert - # export tls_servercertificate_filepath="/dev/shm/server.cert" - startup-broker.sh: |- - #!/bin/bash - APP=`basename "$0"` - IFS='-' read -ra host_array <<< $(hostname) - node_ordinal=${host_array[-1]} - echo "`date` INFO: ${APP}-Node ordinal: ${node_ordinal}" - echo "`date` INFO: ${APP}-Waiting for management API to become available" - password=`cat /mnt/disks/secrets/username_admin_password` - loop_guard=60 - pause=10 - count=0 - while [ ${count} -lt ${loop_guard} ]; do - if /mnt/disks/solace/semp_query.sh -n admin -p ${password} -u http://localhost:8080 -t ; then - break - fi - run_time=$((${count} * ${pause})) - ((count++)) - echo "`date` INFO: ${APP}-Waited ${run_time} seconds, Management API not yet accessible" - sleep ${pause} - done - if [ ${count} -eq ${loop_guard} ]; then - echo "`date` ERROR: ${APP}-Solace Management API never came up" >&2 - exit 1 - fi - # Uncomment if using TLS configuration - # rm /dev/shm/server.cert # remove as soon as possible - # cert_results=$(curl --write-out '%{http_code}' --silent --output /dev/null -k -X PATCH -u admin:${password} https://localhost:1943/SEMP/v2/config/ \ - # -H "content-type: application/json" \ - # -d "{\"tlsServerCertContent\":\"$(cat /mnt/disks/certs/server/tls.key /mnt/disks/certs/server/tls.crt | awk '{printf "%s\\n", $0}')\"}") - # if [ "${cert_results}" != "200" ]; then - # echo "`date` ERROR: ${APP}-Unable to set the server certificate, exiting" >&2 - # exit 1 - # fi - # echo "`date` INFO: ${APP}-Server certificate has been configured" - echo "`date` INFO: ${APP}-PubSub+ Event Broker bringup is complete for this node." - exit 0 - - readiness_check.sh: |- - #!/bin/bash - APP=`basename "$0"` - LOG_FILE=/usr/sw/var/k8s_readiness_check.log # STDOUT/STDERR goes to k8s event logs but gets cleaned out eventually. This will also persist it. - tail -n 1000 ${LOG_FILE} > ${LOG_FILE}.tmp; mv -f ${LOG_FILE}.tmp ${LOG_FILE} || : # Limit logs size - exec > >(tee -a ${LOG_FILE}) 2>&1 # Setup logging - FINAL_ACTIVITY_LOGGED_TRACKING_FILE=/tmp/final_activity_state_logged - - # Function to read Kubernetes metadata labels - get_label () { - # Params: $1 label name - echo $(cat /etc/podinfo/labels | awk -F= '$1=="'${1}'"{print $2}' | xargs); - } - - # Function to set Kubernetes metadata labels - set_label () { - # Params: $1 label name, $2 label set value - #Prevent overdriving Kubernetes infra, don't set activity state to same as previous state - previous_state=$(get_label "active") - if [ "${2}" = "${previous_state}" ]; then - #echo "`date` INFO: ${APP}-Current and Previous state match (${2}), not updating pod label" - : - else - echo "`date` INFO: ${APP}-Updating pod label using K8s API from ${previous_state} to ${2}" - echo "[{\"op\": \"add\", \"path\": \"/metadata/labels/${1}\", \"value\": \"${2}\" }]" > /tmp/patch_label.json - K8S=https://kubernetes.default.svc.cluster.local:$KUBERNETES_SERVICE_PORT - KUBE_TOKEN=$(&2 - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - fi - fi - fi - } - - # Main logic: note that there are no re-tries here, if check fails then return not ready. - # nonHA config - health_result=`curl -s -o /dev/null -w "%{http_code}" http://localhost:5550/health-check/guaranteed-active` - case "${health_result}" in - "200") - if [ ! -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} ]; then - echo "`date` INFO: ${APP}-nonHA Event Broker health check reported 200, message spool is up" - touch ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE} - fi - set_label "active" "true" - exit 0 - ;; - "503") - if [[ $(get_label "active") = "true" ]]; then echo "`date` INFO: ${APP}-nonHA Event Broker health check reported 503, message spool is down"; fi - set_label "active" "false" - # Fail readiness check - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - ;; - *) - echo "`date` WARN: ${APP}-nonHA Event Broker health check reported ${health_result}" - set_label "active" "false" - # Fail readiness check - rm -f ${FINAL_ACTIVITY_LOGGED_TRACKING_FILE}; exit 1 - esac - semp_query.sh: |- - #!/bin/bash - APP=`basename "$0"` - OPTIND=1 # Reset in case getopts has been used previously in the shell. - # Initialize our own variables: - count_search="" - name="" - password="" - query="" - url="" - value_search="" - test_connection_only=false - script_name=$0 - verbose=0 - while getopts "c:n:p:q:u:v:t" opt; do - case "$opt" in - c) count_search=$OPTARG - ;; - n) username=$OPTARG - ;; - p) password=$OPTARG - ;; - q) query=$OPTARG - ;; - u) url=$OPTARG - ;; - v) value_search=$OPTARG - ;; - t) test_connection_only=true - ;; - esac - done - shift $((OPTIND-1)) - [ "$1" = "--" ] && shift - verbose=1 - #echo "`date` INFO: ${APP}-${script_name}: count_search=${count_search} ,username=${username} ,password=xxx query=${query} \ - # ,url=${url} ,value_search=${value_search} ,Leftovers: $@" >&2 - if [[ ${url} = "" || ${username} = "" || ${password} = "" ]]; then - echo "`date` ERROR: ${APP}-${script_name}: url, username, password are madatory fields" >&2 - echo 'missing parameter' - exit 1 - fi - if [ "`curl --write-out '%{http_code}' --silent --output /dev/null -u ${username}:${password} ${url}/SEMP`" != "200" ] ; then - echo "management host is not responding" - exit 1 - fi - if [ "$test_connection_only" = true ] ; then - exit 0 # done here, connection is up - fi - query_response=`curl -sS -u ${username}:${password} ${url}/SEMP -d "${query}"` - # Validate first char of response is "<", otherwise no hope of being valid xml - if [[ ${query_response:0:1} != "<" ]] ; then - echo "no valid xml returned" - exit 1 - fi - query_response_code=`echo $query_response | xmllint -xpath 'string(/rpc-reply/execute-result/@code)' -` - - if [[ -z ${query_response_code} && ${query_response_code} != "ok" ]]; then - echo "query failed -${query_response_code}-" - exit 1 - fi - #echo "`date` INFO: ${APP}-${script_name}: query passed ${query_response_code}" >&2 - if [[ ! -z $value_search ]]; then - value_result=`echo $query_response | xmllint -xpath "string($value_search)" -` - echo "${value_result}" - exit 0 - fi - if [[ ! -z $count_search ]]; then - count_line=`echo $query_response | xmllint -xpath "$count_search" -` - count_string=`echo $count_search | cut -d '"' -f 2` - count_result=`echo ${count_line} | tr "><" "\n" | grep -c ${count_string}` - echo "${count_result}" - exit 0 - fi - - -# Uncomment if using provider-specific StorageClass -# Also uncomment further down at volumeClaimTemplates -# This is an example for aws -#- kind: StorageClass -# apiVersion: storage.k8s.io/v1 -# metadata: -# name: "${DEPLOYMENT_NAME}-standard" -# provisioner: kubernetes.io/aws-ebs -# parameters: -# type: gp2 - -- kind: ServiceAccount - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-sa" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus #end gcp - -- kind: Role - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-podtagupdater" - rules: - - apiGroups: [""] # "" indicates the core API group - resources: ["pods"] - verbs: ["patch"] - -- kind: RoleBinding - apiVersion: rbac.authorization.k8s.io/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus-serviceaccounts-to-podtagupdater" - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: Role - name: "${DEPLOYMENT_NAME}-pubsubplus-podtagupdater" - subjects: - - kind: ServiceAccount - name: "${DEPLOYMENT_NAME}-pubsubplus-sa" -- kind: Service - apiVersion: v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - app.kubernetes.io/name: pubsubplus #end gcp - spec: - type: LoadBalancer - ports: - - port: 2222 - targetPort: 2222 - protocol: TCP - name: tcp-ssh - - port: 8080 - targetPort: 8080 - protocol: TCP - name: tcp-semp - - port: 1943 - targetPort: 1943 - protocol: TCP - name: tls-semp - - port: 55555 - targetPort: 55555 - protocol: TCP - name: tcp-smf - - port: 55003 - targetPort: 55003 - protocol: TCP - name: tcp-smfcomp - - port: 55443 - targetPort: 55443 - protocol: TCP - name: tls-smf - - port: 55556 - targetPort: 55556 - protocol: TCP - name: tcp-smfroute - - port: 8008 - targetPort: 8008 - protocol: TCP - name: tcp-web - - port: 1443 - targetPort: 1443 - protocol: TCP - name: tls-web - - port: 9000 - targetPort: 9000 - protocol: TCP - name: tcp-rest - - port: 9443 - targetPort: 9443 - protocol: TCP - name: tls-rest - - port: 5672 - targetPort: 5672 - protocol: TCP - name: tcp-amqp - - port: 5671 - targetPort: 5671 - protocol: TCP - name: tls-amqp - - port: 1883 - targetPort: 1883 - protocol: TCP - name: tcp-mqtt - - port: 8883 - targetPort: 8883 - protocol: TCP - name: tls-mqtt - - port: 8000 - targetPort: 8000 - protocol: TCP - name: tcp-mqttweb - - port: 8443 - targetPort: 8443 - protocol: TCP - name: tls-mqttweb - selector: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - active: "true" - -- kind: StatefulSet - apiVersion: apps/v1 - metadata: - name: "${DEPLOYMENT_NAME}-pubsubplus" - labels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - spec: - selector: - matchLabels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - serviceName: "${DEPLOYMENT_NAME}-pubsubplus-discovery" - replicas: 1 - podManagementPolicy: Parallel - updateStrategy: - type: RollingUpdate - template: - metadata: - labels: - app.kubernetes.io/name: pubsubplus - app.kubernetes.io/instance: "${DEPLOYMENT_NAME}" - spec: - imagePullSecrets: - - name: ${BROKER_IMAGE_REGISTRY_PULLSECRET} - serviceAccountName: "${DEPLOYMENT_NAME}-pubsubplus-sa" - terminationGracePeriodSeconds: 1200 - containers: - - name: pubsubplus - image: "${BROKER_IMAGE_REGISTRY_URL}:${BROKER_IMAGE_TAG}" - imagePullPolicy: IfNotPresent - resources: - requests: - cpu: "2" - memory: 4025Mi - limits: - cpu: "2" - memory: 4025Mi - livenessProbe: - tcpSocket: - port: 8080 - initialDelaySeconds: 300 - timeoutSeconds: 5 - readinessProbe: - initialDelaySeconds: 30 - periodSeconds: 5 - exec: - command: - - /mnt/disks/solace/readiness_check.sh - securityContext: - privileged: false - env: - - name: STATEFULSET_NAME - value: "${DEPLOYMENT_NAME}-pubsubplus" - - name: STATEFULSET_NAMESPACE - valueFrom: - fieldRef: - fieldPath: metadata.namespace - - name: TZ - value: :/usr/share/zoneinfo/UTC - - name: UMASK - value: "0022" - command: - - bash - - "-ec" - - | - source /mnt/disks/solace/init.sh - # not using postinstall hooks because of order dependencies - # launch config check - readiness check script will be launched by readinessProbe - nohup /mnt/disks/solace/startup-broker.sh & - /usr/sbin/boot.sh - lifecycle: - preStop: - exec: - command: - - bash - - "-ec" - - | - while ! pgrep solacedaemon ; do sleep 1; done - killall solacedaemon; - while [ ! -d /usr/sw/var/db.upgrade ]; do sleep 1; done; - volumeMounts: - - name: podinfo - mountPath: /etc/podinfo - - name: config-map - mountPath: /mnt/disks/solace - - name: secrets - mountPath: /mnt/disks/secrets - readOnly: true - # Uncomment if using TLS configuration - # - name: server-certs - # mountPath: /mnt/disks/certs/server - # readOnly: true - - name: dshm - mountPath: /dev/shm - - name: data - mountPath: /usr/sw/jail - subPath: jail - - name: data - mountPath: /usr/sw/var - subPath: var - - name: data - mountPath: /usr/sw/internalSpool - subPath: internalSpool - - name: data - mountPath: /usr/sw/adb - subPath: adb - - name: data - mountPath: /var/lib/solace/diags - subPath: diags - # only mount softAdb when not using NFS, comment it out otherwise - - name: data - mountPath: /usr/sw/internalSpool/softAdb - subPath: softAdb - # use this instead if using NFS: - #- name: soft-adb-ephemeral - # mountPath: /usr/sw/internalSpool/softAdb - ports: - - containerPort: 2222 - protocol: TCP - - containerPort: 8080 - protocol: TCP - - containerPort: 1943 - protocol: TCP - - containerPort: 55555 - protocol: TCP - - containerPort: 55003 - protocol: TCP - - containerPort: 55443 - protocol: TCP - - containerPort: 55556 - protocol: TCP - - containerPort: 8008 - protocol: TCP - - containerPort: 1443 - protocol: TCP - - containerPort: 9000 - protocol: TCP - - containerPort: 9443 - protocol: TCP - - containerPort: 5672 - protocol: TCP - - containerPort: 5671 - protocol: TCP - - containerPort: 1883 - protocol: TCP - - containerPort: 8883 - protocol: TCP - - containerPort: 8000 - protocol: TCP - - containerPort: 8443 - protocol: TCP - volumes: - - name: podinfo - downwardAPI: - items: - - path: "labels" - fieldRef: - fieldPath: metadata.labels - - name: config-map - configMap: - name: "${DEPLOYMENT_NAME}-pubsubplus" - defaultMode: 0755 - - name: secrets - secret: - secretName: "${DEPLOYMENT_NAME}-pubsubplus-secrets" - defaultMode: 0400 - # Uncomment if using TLS configuration - # - name: server-certs - # secret: - # secretName: "${BROKER_TLS_CERT_SECRET}" - # defaultMode: 0400 - - name: dshm - emptyDir: - medium: Memory - # add this if using NFS (together with montPath changes for softADB - # - name: soft-adb-ephemeral - # emptyDir: {} - volumeClaimTemplates: - - metadata: - name: data - spec: - # Uncomment if using provider-specific StorageClass - # Also uncomment StorageClass definition above - # storageClassName: "${DEPLOYMENT_NAME}-standard" - accessModes: [ "ReadWriteOnce" ] - resources: - requests: - storage: "${BROKER_STORAGE_SIZE}" - -parameters: - - name: DEPLOYMENT_NAME - displayName: PubSub+ Event Broker Deployment Name - description: The prefix to use for object names - generate: expression - from: '[A-Z0-9]{8}' - value: example - required: true - - name: BROKER_IMAGE_REGISTRY_URL - displayName: Image Registry URL - description: The image registry URL for the registry containing the PubSub+ Event Broker docker image - value: registry.connect.redhat.com/solace/pubsubplus-standard - required: true - - name: BROKER_IMAGE_REGISTRY_PULLSECRET - displayName: Image Registry Pull Secret - description: The pull secret to be used to obtain the PubSub+ Event Broker docker image - value: - required: false - - name: BROKER_IMAGE_TAG - displayName: PubSub+ Event Broker Docker Image Tag - description: The Docker image tag for the PubSub+ Event Broker docker image from your Docker registry - value: latest - required: true - - name: BROKER_ADMIN_PASSWORD - displayName: Base64 encoded password for PubSub+ username 'admin' - description: The Event Broker 'admin' user's password (base64 encoded). This PubSub+ OpenShift template will create an administrative user with username 'admin' with specified password. - required: true - - name: BROKER_STORAGE_SIZE - displayName: PubSub+ Event Broker Persistent Storage Disk Size - description: The size in gigabytes for a Event Broker Pod's persistent volume (with suffix 'Gi'), example 30Gi for 30 gigabytes - value: 30Gi - required: true - # Uncomment if using TLS configuration - # - name: BROKER_TLS_CERT_SECRET - # displayName: Certificate Secret for Broker TLS Config - # description: The server key and certificate for the event broker packaged in a Kubernetes secret - # value: - # required: true \ No newline at end of file