Learn How to Operate in the Brokerpak/Broker/SSB World

[nickumia-reisys]

User Story

In order to make contributions to the SSB, the Data.gov Core Team wants to follow these steps to become familiar with the development, deployment, maintenance and management of SSB components.

Acceptance Criteria

• GIVEN this ticket is complete
  WHEN the assignee performs a demonstration
  THEN foundational learning of the SSB is complete

Background

Nick is leaving 😢

Security Considerations (required)

Knowledge Transfer is important and needs to happen to maintain awareness of the project.

Sketch

Theoretical Learning

• Read information about Cloud Service Broker
  • Understand the CSB CLI
    • NOTE: We are running v0.10.0 of the CSB, the documentation is for the newest version v0.19.9. This will require additional work to upgrade our CSB to the latest version. This tutorial assumes that we are sticking with v0.10.0 for the short term.
• Read information about developing a Brokerpak
• Read information about Brokerpak Specifications
  • NOTE: Understand the complex relationship between
    • Terraform provision inputs/outputs
    • Terraform bind inputs/outputs
    • Broker provision inputs/outputs
    • Broker bind inputs/outputs
• Read information about datagov-brokerpak-eks AND datagov-brokerpak-solr
  • Understand KinD
  • Investigate K8S Changelog
• Read Information about SSB
• Read information about Terraform (aws tutorial, An Introduction to Terraform)
  • Understand where to get information from...
    • Terraform Registry (providers + modules)
      • Example: helm_release from helm provider
    • ArtifactHub (helm charts)
    • AWS EKS Documentation
    • Module specific references (e.g. ALB Controller)
• Read through some historical issues/prs
  • Switch back to EKS Brokerpak paradigm from statically-provisioned EKS instance #3758
  • datagov-ssb -- 'The request is invalid' #3718
  • Standing terraform driven EKS deployments for each SSB space #3696
  • SSB never finishes provisioning SMTP service when a domain is provided #3570
  • Solr 8 instance credentials are not bound on cloud.gov #3523
  • SSB environment promotion workflow #2868

Practical Learning (the Solrcloud example here "should" be very similar to the environment in which airflow would exist.)

• datagov-brokerpak-solr
  • WORKFLOW 1 - Running Terraform
    • Spin up KinD cluster by doing make kind-up
    • Pass KinD config/credentials to Terraform code directly and launch SolrCloud in KinD on local machine
      • Do terraform apply in provision directory to create SolrCloud
      • Do terraform apply in bind directory to generate credentials to talk to SolrCloud
      • Play around with SolrCloud
      • Do terraform destroy in bind directory to delete credentials
      • See that the credentials no longer work
      • Do terraform destroy in the provision directory to delete SolrCloud (notice, the KinD cluster is still running)
    • Destroy KinD cluster by doing make kind-down
  • WORKFLOW 2 - Running the CSB in Docker on Local Machine
    • Spin up KinD cluster by doing make kind-up
    • Build broker by doing make build up
    • Send a request to create solrcloud by doing something like docker exec csb-service-solr /bin/cloud-service-broker client provision --serviceid 'b9013a91-9ce8-4c18-8035-a135a8cd6ff9' --planid 'e35e9675-413f-4f42-83de-ad5003357e77' --instanceid "demo-user-solrcloud" --params '{ \"replicas\": 2 }'
    • Send a request to create credentials for solrcloud by doing something like docker exec csb-service-solr /bin/cloud-service-broker client bind --serviceid 'b9013a91-9ce8-4c18-8035-a135a8cd6ff9' --planid 'e35e9675-413f-4f42-83de-ad5003357e77' --instanceid "demo-user-solrcloud" --bindingid "testid"
    • NOTE: The Terraform state is managed in the docker container running the broker (IN THIS CASE), since the resources are all local in KinD, it is okay to completely destroy the container without deprovisioning. HOWEVER, if this was creating resources on AWS, those resources would need to be manually cleaned up in the case this container was destroyed before deprovisioning all instances.
    • Send a request to unbind
    • Send a request to deprovision
    • Stop/Remove the broker container and tear down KinD
• datagov-brokerpak-eks
  • WORKFLOW 1 - Running Terraform
    • Setup AWS Credentials for ssb-development account
    • Perform README in the terraform/modules/provision-aws directory.
    • See the k8s cluster in action by doing kubectl commands
    • Do terraform apply in the terraform/modules/bind directory.
    • See the difference between an "admin" kubeconfig and a "user" kubeconfig
    • Do terraform destroy in the terraform/modules/bind directory.
    • Do terraform destroy in the terraform/modules/provision-aws directory.
  • WORKFLOW 2 - Running the CSB in Docker on Local Machine
    • Very similar to the steps above, but KEEP IN MIND the AWS Resource Orphaning Case mentioned above.
  • Extra Credit!
    • Provision a EKS cluster with the datagov-brokerpak-eks repo (either terraform or broker workflow).
    • Pass the kubeconfig into the datagov-brokerpak-solr repo (either terraform or broker workflow).
    • Launch SolrCloud in EKS.
• datagov-ssb
  • Make a release in some brokerpak (EKS or Solr)
  • Update the app-setup-<service>.sh script to reference the new version
  • Push PR to develop branch.
  • Check the deployment in github actions.
  • If successful, try to create service in development-ssb space
  • If not successful, contact @nickumia or @mogul for additional explanations
  • If everything is functional from the development-ssb test, create PR to main
  • Review terraform plan comment in PR (generated from github actions)
  • If plan looks acceptable, merge and see change in production!
  • If plan does not look acceptable, then evaluate changes and correct as necessary.
    • An "acceptable" plan varies by situation, however, most updates will just change the broker app (i.e. a specific one, whether it's solr or eks). However, if you are adding additional resources, of course new things will be created. If a resource is changing that should NOT be affected by the changes in the PR, then that definitely constitutes as a BAD plan.