Document and support workaround for cross-region DMR performance tuni…

…ng (#25)

* Document and support workaround for cross-region DMR performance tuning

README.md

@@ -96,7 +96,9 @@ chmod 755 copy_solace_image_to_gkr.sh
 
 ### Step 4: Use Google Cloud SDK or Cloud Shell to create the three node GKE cluster
 
-* Download and execute the cluster creation script. It would be alright to accept the default values for all the script's arguments if you were setting up and running a single message broker; however, some need to be changed to support the 3 node HA cluster. If you want to run the HA cluster in a single GCP zone, specify `-n = 3` as the number of nodes per zone and a single `-z <zone>`. If you want the HA cluster spread across 3 zones within a region - which is the configuration recommended for production situations - specify the 3 zones as per the example below, but leave the number of nodes per zone at the default value of 1.
+* Download and execute the cluster creation script. Accept the default values for all the script's arguments if you were setting up and running a single message broker; however, some need to be changed to support the 3 node HA cluster. If you want to run the HA cluster in a single GCP zone, specify `-n = 3` as the number of nodes per zone and a single `-z <zone>`. If you want the HA cluster spread across 3 zones within a region - which is the configuration recommended for production situations - specify the 3 zones as per the example below, but leave the number of nodes per zone at the default value of 1.
+
+**Important:** if connecting Solace brokers across GCP regions, there is a known issue affecting TCP throughput with the default node OS image type Ubuntu and default settings. In this case additionally specify the node image as Container-Optimized OS (cos) and a flag to apply performance tuning: `-i cos -p`. 
 
 ```sh
 wget https://raw.githubusercontent.com/SolaceProducts/solace-gke-quickstart/master/scripts/create_cluster.sh
@@ -108,11 +110,13 @@ This will create a GKE cluster of 3 nodes spread across 3 zones:
 
 ![alt text](/images/Nodes_across_zones.png "Google Container Engine nodes")
 
-Here are two more GKE `create_cluster.sh` arguments you may need to consider changing for your deployment:
+Here are more GKE `create_cluster.sh` arguments you may need to consider changing for your deployment:
+
+* The default cluster name is `solace-cluster` which can be changed by specifying the `-c <cluster name>` command line argument.
 
-* solace-message-broker-cluster: The default cluster name, which can be changed by specifying the `-c <cluster name>` command line argument.
+* The default machine type is "n1-standard-4". To use a different [Google machine type](https://cloud.google.com/compute/docs/machine-types ), specify `-m <machine-type>`. Note that the minimum CPU and memory requirements must be satisfied for the targeted message broker size, see the next step.
 
-* n1-standard-4: The default machine type. To use a different [Google machine type](https://cloud.google.com/compute/docs/machine-types ), specify `-m <machine type>`. Note that the minimum CPU and memory requirements must be satisfied for the targeted message broker size, see the next step.
+* The default node OS image type is Ubuntu. Specify [other node image type](https://cloud.google.com/kubernetes-engine/docs/concepts/node-images ) using `-i <image-type>`
 
 <br>
 

scripts/create_cluster.sh

@@ -27,9 +27,10 @@ machine_type="n1-standard-4"
 image_type="UBUNTU"
 number_of_nodes="1"
 zones="us-central1-b"
+bridge_perf_tune=false
 verbose=0
 
-while getopts "c:i:m:n:z:" opt; do
+while getopts "c:i:m:n:z:p" opt; do
     case "$opt" in
     c)  cluster_name=$OPTARG
         ;;
@@ -41,14 +42,35 @@ while getopts "c:i:m:n:z:" opt; do
         ;;
     z)  zones=$OPTARG
         ;;
+    p)  perf_tune=true
+        ;;
     esac
 done
 
 shift $((OPTIND-1))
 [ "$1" = "--" ] && shift
 
 verbose=1
-echo "`date` INFO: cluster_name=${cluster_name}, machine_type=${machine_type}, image_type=${image_type}, number_of_nodes=${number_of_nodes}, zones=${zones} ,Leftovers: $@"
+echo "`date` INFO: cluster_name=${cluster_name}, machine_type=${machine_type}, image_type=${image_type}, number_of_nodes=${number_of_nodes}, zones=${zones}, perf_tune=${perf_tune} ,Leftovers: $@"
+
+# multi-region bridge performance tuning
+node_performance_tune () {
+  command="echo '
+  net.core.rmem_max = 134217728
+  net.core.wmem_max = 134217728
+  net.ipv4.tcp_rmem = 4096 25165824 67108864
+  net.ipv4.tcp_wmem = 4096 25165824 67108864
+  net.ipv4.tcp_mtu_probing=1' | sudo tee /etc/sysctl.d/98-solace-sysctl.conf ; sudo sysctl -p /etc/sysctl.d/98-solace-sysctl.conf"
+
+  list=`gcloud compute instances list | grep gke-$1-`
+  echo 'Applying multi-region bridge performance tuning to nodes'
+  while read -r a b c ; do
+    echo $a
+    gcloud compute ssh --ssh-flag="-T -o StrictHostKeyChecking=no" --zone $b $a -- "$command" &>/dev/null &
+  done <<< "$list"
+  wait
+}
+
 
 echo "`date` INFO: INITIALIZE GCLOUD"
 echo "#############################################################"
@@ -58,4 +80,7 @@ gcloud config set compute/zone ${zone_array[0]}
 echo "`date` INFO: CREATE CLUSTER"
 echo "#############################################################"
 gcloud container clusters create ${cluster_name} --machine-type=${machine_type} --image-type=${image_type} --node-locations=${zones} --num-nodes=${number_of_nodes}
+if $perf_tune ; then
+  node_performance_tune ${cluster_name}
+fi
 gcloud container clusters get-credentials ${cluster_name}