Terraform driven eks deploy alternative providers (#123)

* new: provision standalone eks cluster using terraform only

Create custom 'brokerpak-eks-terraform-provision' module using eks-brokerpak as source; specify custom parameters to provision with

* update: terraform and modules

* new: add bind resource for eks-terraform cluster

Create user-provided service based on binding to pass to solrcloud broker

* fix/refactor: bind depends on provision

* update/fix: child modules can't specify their own provider blocks

Reference: GSA-TTS/datagov-brokerpak-eks#79

* cleanup: remove unreferenced variable

* new: add eks_terraform params for staging/prod

* fix: ensure bind actually waits for cluster_functional

Explicit output from module to module

* new: install aws/kubectl/helm tools for terraform apply

* fix: don't mess up ubuntu's path

* test: attempt to fix github action path

Trying to determine where home is and where the real path actually is

* test: ensure that the aws-iam-authenticator was installed

Also tests whether the command is available later in the same job

* test: the command actually wasn't added to the path yet..

Reference: https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#adding-a-system-path

* fix: high hopesgit add .

aws-iam-authenticator needs AWS Credentials in the env

* new: successfully configure providers for eks as a submodule

* revert: no need to set AWS creds

The S3 Backend is set up properly alongside terraform AWS credentials, so this broke that

* update: supply the module's required_provider using the correct alias

* fix: provide AWS creds for the module to use with kubernetes provider

The module needs AWS creds so the kubernetes provider can use aws-iam-authenticator when it connects.

* workaround: temporarily comment out the resources that aren't working

We have yet to resolve the issue with the kubernetes_provider in the bind module not working, due to it having a dependency on another module.

* refactor: move declaration of IaaS-only resources to the managed-boundary

Technically this is where they belong, in terms of our SSP and diagram terminology.

Note also that we are no longer invoking the submodules separately, but rather directly invoking the parent module at `datagov-brokerpak-eks/terraform`

* lint: fix terraform fmt

* fix: uncomment CF-side resources

* fix: name "credentials" is implicit

* update: eks-brokerpak no longer needs a branch

Dual support for brokerpak mode or terraform module mode is enabled :)

* fix: terraform string literals don't like newlines

Reference: hashicorp/terraform-provider-vault#307

* fix: use data sources for space/org lookup

* fix: set variables based on service name

update VCAP_SERVICE selection to use the correct service based on service name, not service type

* new: install tools as TERRAFORM_PRE_RUN

Since dflook action is a separate container, the tools need to be pulled in properly.  The terraform working directory becomes part of the path for that step, so installing the tools there works :)

* lint: ran terraform fmt

* new: add tools to known path

After lots of tries, this seems to be the most reliable method

* temp: work off of branch until it gets merged

Too many changes to keep track off.. the branch is fully functional for terraform only stuff, but brokerpak stuff is a bit broken :/

Co-authored-by: Bret Mogilefsky <[email protected]>

.github/workflows/commit.yml

@@ -40,6 +40,12 @@ jobs:
       TF_VAR_cf_password: ${{ secrets.TF_VAR_cf_password }}
       TF_VAR_aws_access_key_id: ${{ secrets.TF_VAR_aws_access_key_id }}
       TF_VAR_aws_secret_access_key: ${{ secrets.TF_VAR_aws_secret_access_key }}
+      TERRAFORM_PRE_RUN: |
+          ./install-tools.sh
+          cp helm /usr/local/bin/
+          cp kubectl /usr/local/bin/
+          cp aws-iam-authenticator /usr/local/bin/
+          aws-iam-authenticator help
 
     steps:
       - name: checkout

.gitignore

@@ -45,3 +45,7 @@ app-solrcloud
 app-ssb-*.zip
 .backend.secrets
 .env.*.secrets
+
+linux-amd64/
+bin/
+kubeconfig*

Dockerfile

@@ -0,0 +1,14 @@
+FROM hashicorp/terraform:1.1.5 as upstream
+
+FROM alpine/k8s:1.20.7
+
+COPY --from=upstream /bin/terraform /bin/terraform
+
+RUN apk update
+RUN apk upgrade
+# Install git so we can use it to grab Terraform modules
+RUN apk add --update git
+
+WORKDIR /bin
+ENTRYPOINT ["/bin/terraform"]
+CMD ["help"]

app-setup-solrcloud.sh

@@ -16,17 +16,18 @@ mkdir -p $APP_NAME/bin
 
 # Generate a .profile to be run at startup for mapping VCAP_SERVICES to needed
 # environment variables
+# export SERVICE_NAME=ssb-solrcloud-k8s
 cat > $APP_NAME/.profile << 'EOF'
 # Locate additional binaries needed by the deployed brokerpaks
 export PATH="$PATH:${PWD}/bin"
 
 # Export credentials for the k8s cluster and namespace where the Solr brokerpak
 # should manage instances of SolrCloud. We get these from the binding directly.
-export SOLR_SERVER=$(echo $VCAP_SERVICES | jq -r '.["aws-eks-service"][] | .credentials.server')
-export SOLR_CLUSTER_CA_CERTIFICATE=$(echo $VCAP_SERVICES | jq -r '.["aws-eks-service"][] | .credentials.certificate_authority_data')
-export SOLR_TOKEN=$(echo $VCAP_SERVICES | jq -r '.["aws-eks-service"][] | .credentials.token')
-export SOLR_NAMESPACE=$(echo $VCAP_SERVICES | jq -r '.["aws-eks-service"][] | .credentials.namespace')
-export SOLR_DOMAIN_NAME=$(echo $VCAP_SERVICES | jq -r '.["aws-eks-service"][] | .credentials.domain_name')
+export SOLR_SERVER=$(echo $VCAP_SERVICES | jq -r '.[][]| select(.name=="ssb-solrcloud-k8s") | .credentials.server')
+export SOLR_CLUSTER_CA_CERTIFICATE=$(echo $VCAP_SERVICES | jq -r '.[][]| select(.name=="ssb-solrcloud-k8s") | .credentials.certificate_authority_data')
+export SOLR_TOKEN=$(echo $VCAP_SERVICES | jq -r '.[][]| select(.name=="ssb-solrcloud-k8s") | .credentials.token')
+export SOLR_NAMESPACE=$(echo $VCAP_SERVICES | jq -r '.[][]| select(.name=="ssb-solrcloud-k8s") | .credentials.namespace')
+export SOLR_DOMAIN_NAME=$(echo $VCAP_SERVICES | jq -r '.[][]| select(.name=="ssb-solrcloud-k8s") | .credentials.domain_name')
 EOF
 chmod +x $APP_NAME/.profile
 

application-boundary.tf

@@ -43,25 +43,48 @@ resource "cloudfoundry_service_instance" "k8s_cluster" {
     delete = "40m"
   }
   depends_on = [
-    module.broker_eks,
-    cloudfoundry_service_instance.solrcloud_broker_k8s_cluster
+    module.broker_eks
   ]
 }
 
-resource "cloudfoundry_service_instance" "solrcloud_broker_k8s_cluster" {
-  name         = "ssb-solrcloud-k8s"
-  space        = data.cloudfoundry_space.broker_space.id
-  service_plan = module.broker_eks.plans["aws-eks-service/raw"]
-  tags         = ["k8s"]
-  json_params  = "{\"mng_min_capacity\": 8, \"mng_max_capacity\": 12, \"mng_desired_capacity\": 10}"
-  timeouts {
-    create = "60m"
-    update = "90m" # in case of an EKS destroy/create
-    delete = "40m"
-  }
-  depends_on = [
-    module.broker_eks
-  ]
+# resource "cloudfoundry_service_instance" "solrcloud_broker_k8s_cluster" {
+#   name         = "ssb-solrcloud-k8s"
+#   space        = data.cloudfoundry_space.broker_space.id
+#   service_plan = module.broker_eks.plans["aws-eks-service/raw"]
+#   tags         = ["k8s"]
+#   json_params  = "{\"mng_min_capacity\": 8, \"mng_max_capacity\": 12, \"mng_desired_capacity\": 10}"
+#   timeouts {
+#     create = "60m"
+#     update = "90m" # in case of an EKS destroy/create
+#     delete = "40m"
+#   }
+#   depends_on = [
+#     module.broker_eks
+#   ]
+# }
+
+data "cloudfoundry_org" "gsa" {
+  name = var.broker_space.org
+}
+
+data "cloudfoundry_space" "dev-ssb" {
+  name = var.broker_space.space
+  org  = data.cloudfoundry_org.gsa.id
+}
+
+resource "cloudfoundry_user_provided_service" "ssb-solrcloud-k8s" {
+  name             = "ssb-solrcloud-k8s"
+  space            = data.cloudfoundry_space.dev-ssb.id
+  credentials_json = <<-JSON
+    {
+      "certificate_authority_data": "${module.brokerpak-eks-terraform.certificate_authority_data}",
+      "domain_name": "${module.brokerpak-eks-terraform.domain_name}",
+      "kubeconfig": "${replace(module.brokerpak-eks-terraform.kubeconfig, "\n", "\\n")}",
+      "namespace": "${module.brokerpak-eks-terraform.namespace}",
+      "server": "${module.brokerpak-eks-terraform.server}",
+      "token": "${module.brokerpak-eks-terraform.token}"
+    }
+  JSON
 }
 
 module "broker_solrcloud" {
@@ -72,7 +95,8 @@ module "broker_solrcloud" {
   broker_space  = var.broker_space
   client_spaces = var.client_spaces
   enable_ssh    = var.enable_ssh
-  services      = [cloudfoundry_service_instance.solrcloud_broker_k8s_cluster.id]
+  # services      = [cloudfoundry_service_instance.solrcloud_broker_k8s_cluster.id]
+  services = [cloudfoundry_user_provided_service.ssb-solrcloud-k8s.id]
 }
 
 module "broker_solr" {

docker-compose.yaml

@@ -2,7 +2,9 @@ version: '3.7'
 
 services:
   terraform:
-    image: hashicorp/terraform:1.0.8
+    image: datagov-ssb/terraform:latest
+    build:
+      context: .
     volumes:
       - .:/code
     working_dir: /code

install-tools.sh

@@ -0,0 +1,25 @@
+#!/bin/bash
+set -ex
+
+# TODO: Check sha256 sums
+HELM_VERSION="3.7.1"
+KUBECTL_VERSION="1.22.3"
+
+AWS_IAM_AUTH_VERSION_URL="https://amazon-eks.s3.us-west-2.amazonaws.com/1.21.2/2021-07-05/bin/linux/amd64/aws-iam-authenticator"
+
+BASE_URL="https://get.helm.sh"
+TAR_FILE="helm-v${HELM_VERSION}-linux-amd64.tar.gz"
+
+# Install the Helm binary
+curl -f -L ${BASE_URL}/${TAR_FILE} |tar xvz && \
+    mv linux-amd64/helm helm && \
+    chmod +x helm && \
+    rm -rf linux-amd64
+
+# Install kubectl
+curl -f -LO https://storage.googleapis.com/kubernetes-release/release/v${KUBECTL_VERSION}/bin/linux/amd64/kubectl && \
+    chmod +x kubectl
+
+# Install aws-iam-authenticator
+curl -f -LO ${AWS_IAM_AUTH_VERSION_URL} && \
+    chmod +x aws-iam-authenticator