Add support for Bind Module Reference

[nickumia-reisys]

Getting error 'Module module.brokerpak-eks-terraform-bind contains provider configuration' in ssb broker

relates to GSA/data.gov#3696

[mogul]

Where we're at: As of the refactoring commits tonight in this branch and in the corresponding datagov-ssb repo branch, the modules no longer have any providers in them, the root module over them both creates appropriate providers for each module and injects them. Terraform validate approves! However, when I run terraform apply it's getting stuck because the deployment of csi-ebs driver and ingress-nginx helm-release are timing out.

This is almost certainly because those pods have daemonsets and cannot run on Fargate, and the MNG group is not available to schedule. We'll need to review this series of commits tomorrow to see what I b0rked up:

[mogul]

Where we're at:

• Commenting out module.provision.module.eks' fargate_profiles configuration got everything working again.
  • We'll leave that commented out for now, though in future we may or may not want it available, depending on whether we get everything we need from auto-scaling our managed-node groups.
• Destroys are not clean.
  • For some reason during destroy the helm provider makes use of the module.provision.kubernetes_service_account.admin service account we create in admin_account.tf, despite the helm provider explicitly not using that account when it's configured in provision-providers.tf.
    • That service account loses the cluster-admin role binding almost immediately, so Helm fails because the account doesn't have access to secrets.
    • We could make the four helm_release resources depend on that binding to get around this, but that's not solving the real problem: Why isn't the helm provider using the configured AWS creds and generated tokens!?
  • If you work around that, destroy of the helm_release ingress_nginx is sometimes getting stuck in state uninstalling. 🙄 I discovered that the helm provider has a debug=true argument that will hopefully help us figure this out once and for all.
  • And if you get around that, of course, the NLB and corresponding target groups never get deleted by alb_controller, which means the certificate doesn't get destroyed, which means the VPC can't be deleted.

[mogul]

• Destroys are not clean.

  • For some reason during destroy the helm provider makes use of the module.provision.kubernetes_service_account.admin service account we create in admin_account.tf, despite the helm provider explicitly not using that account when it's configured in provision-providers.tf.
    • That service account loses the cluster-admin role binding almost immediately, so Helm fails because the account doesn't have access to secrets.
    • We could make the four helm_release resources depend on that binding to get around this, but that's not solving the real problem: Why isn't the helm provider using the configured AWS creds and generated tokens!?

We figured this problem out.

[mogul]

Lots of teensy bugs fixed along the way to this point:

$ make demo-run
Testing aws-eks-service:raw:instance-bmogilefsky3:binding
To work directly with the instance:
export KUBECONFIG=/tmp/tmp.pAidiG73VS
export DOMAIN_NAME=instance-bmogilefsky3.ssb-dev.data.gov
Running tests...
Deploying the test fixture...
deployment.apps/deployment-2048 created
service/service-2048 created
ingress.networking.k8s.io/ingress-2048 created
Waiting 3 minutes for the workload to start and the DNS entry to be created...
Testing that the ingress is resolvable via SSL, and that it's properly pointing at the 2048 app...curl: (52) Empty reply from server
make: *** [Makefile:109: demo-run] Error 1

That's the next thing to look into.

[mogul]

Oh hey, IT PASSED! Maybe there was just some cruft left in place from my last local invocation of test with an instance of the same name that prevented it from passing locally.

[nickumia-reisys]

This is an awesome feature add! I won't hark on it right now, but we should probably update the README or add a file in the docs dir to record the dual functionality of brokerpak or terraform module.

AMAZING WORK THOUGH!! 🍹

P.S. Alas I can't merge this PR because technically it's mine 😕