Spin up Airflow POC using local KinD cluster

[btylerburton]

User Story

In order to determine the best practice for running Airflow at scale, datagovteam would like to create an Airflow POC using Kubernetes in Docker (KinD).

Acceptance Criteria

[ACs should be clearly demoable/verifiable whenever possible. Try specifying them using BDD.]

• GIVEN [a contextual precondition]
  [AND optionally another precondition]
  WHEN [a triggering event] happens
  THEN [a verifiable outcome]
  [AND optionally another verifiable outcome]

Background

[Any helpful contextual notes or links to artifacts/evidence, if needed]

Security Considerations (required)

[Any security concerns that might be implicated in the change. "None" is OK, just be explicit here!]

Sketch

• Install KinD on your local machine
• Install Airflow Helm Chart
• Deploy airflow helm release
• Understand what comes packaged with the helm release
• Configure ingress-controller for webserver interaction

[nickumia-reisys]

This is the airflow-equivalent work for the datagov-brokerpak-solr section under the "Practical Learning (the Solrcloud example here "should" be very similar to the environment in which airflow would exist.)" chapter in

• Learn How to Operate in the Brokerpak/Broker/SSB World #4483

This ticket is more-so doing everything by-hand. Follow on work would being converting the "by-hand" work into "terraform" work and then finally into "broker" work.

• "By-Hand": invoking the kubectl and helm commands
• "Terraform": representing everything in terraform code and defining the resources based on the kubernetes and helm providers based on the same logic used in the "by-hand" method.
• "Broker": packaging the terraform code into the CSB and creating resources by using the CSB CLI.