🧟 Revive the Dead #112

nickumia-reisys · 2023-10-05T21:37:11Z

Related to

Checklist:

Fix double terraform apply requirement
- Increase reliability of EKS LB deprovisioning GSA/data.gov#3617 provides some background on a "clean terraform apply"
Review this PR as a team and make sure everyone is aware of all changes
Reach out to @nickumia if there are any open questions about design choices
Are the changes breaking?
- If yes, check with active members/maintainers/users in #cg-contributors
- If no, great job! 🥳
Are new tests required (and added) for changed resources?
If fargate is working, do we really want to delete managed node groups?? If we ever need to switch back again, we'll have to revive work in the same way that this PR did to revive fargate 😖
- Oooo I think the best way to "consolidate" this is to offer TWO PLANS!! We never really had a use-case for that before this 😲

Background:
In an attempt to leverage k8s for airflow development, the datagov-brokerpak-eks repo needs to be on a supported version of k8s. At the time of writing, there are 130 resources managed by this terraform code. Most of it is documented in our cleanup documentation. These 130 resources are managed through 57 terraform resources and 4 terraform modules (eks, aws_load_balancer_controller, vpc and iam_assumable_role_karpenter). Within these resources, Helm charts are used to create many more intangible resources inside of k8s. Referencing all of the documentation and understanding how to make updates to this is important and requires time and effort.

Many resources were only included because we knew it was dependency, either through direct documentation or trial and error. I'm trying to cover everything before 10/12; however, I suspect that there will be additional work to do. See notes below on the status of current efforts.

Notes:

If there is an EKS cluster provisioned already AND you change tags on the module.eks resource, you MUST delete the current cluster and reprovision from scratch, since the tags interfere with talking to the kubernetes control plane.
How to locate the ALB Controller Helm Release:

Misc Notes:

Link to deprecation warning for eks module

References:

Main EKS Module
EKS Module -fargate_profiles
Important historical comment on autoscalers
autoscaler-provisioner to cluster-provisioner docs
- Helm
- Helm value configs
- AWS-specific config

@Jin-Sun-tts

- We only have the GSA ISE AMI for K8S v1.21... since AWS doesn't allow us to create anything older than v1.23, we need to request new AMIs from GSA ISE, related issues: - GSA/data.gov#3812 - GSA/data.gov#3808 - GSA/data.gov#3761 - Fargate is a lot easier to use for @Jin-Sun-tts's proof of concept :)

- Reference for k8s docker image: https://hub.docker.com/r/alpine/k8s/tags?page=1&name=1.23

we're going to try to use just fargate and then maybe do a combo support of fargate + managed node groups with variable inputs

apparent, this changed?? aws/aws-cli#6920 (comment)

might have breaking changes to consider, primarily aws provider

- Description: The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts - Description: The name of the EKS cluster

This was maybe defaulting to true in an older version of eks module.. I think this configuration is safe because we are specifying a subset of public cidrs to allow access

This works with the newer version of kubectl, aws cli, eks and k8s - https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/eks_cluster#arn - aws cli install: https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html - kubectl version: aws/aws-cli#6920 (comment) - - gravitational/teleport#23032 - https://docs.aws.amazon.com/eks/latest/userguide/create-kubeconfig.html

- k8s v1.28 utilized newer version of the networking api, which forced a newer version of the alb controller (and by extension, the helm chart that deploys that controller) - ANSWER: kubernetes-sigs/aws-load-balancer-controller#2495 (comment) - This is a temp fix on a PR until the full initial revamp is complete

- ERROR: │ Error: unable to build kubernetes objects from release manifest: resource mapping not found for name: 'ingress-nginx-controller' namespace: 'kube-system' from '': no matches for kind 'HorizontalPodAutoscaler' in version 'autoscaling/v2beta2' │ ensure CRDs are installed first - Solution: took a chance and just updated the chart version

… k8s version

- Revive Fargate configuration for karpenter + EFS from old commits - Update calico ... v3.26.2 adding missing files

The data resource should not need to depend on anything.. maybe the issue is changes within the eks module... the resources aren't properly linked anymore.. although... the better fix might be using the module.eks variable and see if there have been improvements that make the cluster referable directly from there??

This seems like the better answer to the last commit concern

Reference: https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest?tab=outputs

The data is base64encoded already, so just decode it and it is the PEM data itself

This looks to be working. It is an upgrade because the autoscaler-provisioner is deprecated. It might not be working because of fargate configuration that still needs some fine-tuning (in terms of spinning up new nodes). However, the helm chart deploys. I haven't hit a blocker that is definitive that we can't use fargate. This requires more investigation. References: - https://github.com/kubernetes/autoscaler/blob/master/cluster-autoscaler/cloudprovider/aws/README.md#auto-discovery-setup - https://github.com/kubernetes/autoscaler/blob/master/charts/cluster-autoscaler/Chart.yaml - https://artifacthub.io/packages/helm/cluster-autoscaler/cluster-autoscaler

- Calico and Starboard don't work, so just comment them out so that terraform can apply mostly cleanly. There is still an issue about running terrraform apply. I think terraform appy needs to be run multiple times because coredns cannot provision effectively because fargate isn't ready?? It should be some depends_on to be added someplace, but mentioning since it's important for brokerpak operations - Lots of design choices to be made. The hardest thing is iteration speed and having the time, energy and resources to do the work. However, I'll actively make comments on things to try if people get stuck working on this.

Moment of truth with this push haha..

nickumia-reisys added 14 commits October 4, 2023 18:21

update: k8s version for dockerfile

8cfa646

- Reference for k8s docker image: https://hub.docker.com/r/alpine/k8s/tags?page=1&name=1.23

fix: comment out more managed node config

ba9ec7b

fix: comment out more managed node groups

9389025

we're going to try to use just fargate and then maybe do a combo support of fargate + managed node groups with variable inputs

update: kubectl apiversion

197044f

apparent, this changed?? aws/aws-cli#6920 (comment)

lint: terraform fmt

3b31493

refactor: move aws_launch_template

c3a38d2

update: all provider versions

2eea708

might have breaking changes to consider, primarily aws provider

update: eks/vpc modules

ce1375c

new: update to LATEST k8s version

3fefe58

fix: cluster_name/id per module update

dbbcc3a

- Description: The ID of the EKS cluster. Note: currently a value is returned only for local EKS clusters created on Outposts - Description: The name of the EKS cluster

fix: one more reference to cluster_id

977ed57

fix?: cluster_endpoint_public_access

5aa1e7c

This was maybe defaulting to true in an older version of eks module.. I think this configuration is safe because we are specifying a subset of public cidrs to allow access

nickumia-reisys mentioned this pull request Oct 5, 2023

Update ALB Controller v2.6.1 qvest-digital/terraform-kubernetes-alb-ingress-controller#23

Closed

nickumia-reisys added 4 commits October 5, 2023 19:38

refactor: move ami to managed node group file

80d112b

new: update kubernetes_secret admin token to be compatible with newer…

7f18117

… k8s version

This was referenced Oct 7, 2023

Update ALB Controller v2.6.1 GSA/terraform-kubernetes-aws-load-balancer-controller#32

Draft

Run CIS Scans on EKS AMIs GSA/data.gov#3760

Open

nickumia-reisys added 2 commits October 10, 2023 18:07

new: karpenter/efs/calico

cdc072a

- Revive Fargate configuration for karpenter + EFS from old commits - Update calico ... v3.26.2 adding missing files

nickumia-reisys linked an issue Oct 11, 2023 that may be closed by this pull request

Learn How to Operate in the Brokerpak/Broker/SSB World GSA/data.gov#4483

Open

69 tasks

nickumia-reisys added 6 commits October 11, 2023 13:51

refactor: aws_eks_cluster to module.eks

7a925a8

This seems like the better answer to the last commit concern

fix: cluster_id doesn't exist, use cluster_name instead

f38a018

Reference: https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/latest?tab=outputs

fix: 'cluster_ca_certificate' is not a valid PEM encoded certificate

733c72d

The data is base64encoded already, so just decode it and it is the PEM data itself

lint: terraform fmt

0bab69c

Moment of truth with this push haha..

update: brokerpak packaged versions

b11f99f

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🧟 Revive the Dead #112

🧟 Revive the Dead #112

nickumia-reisys commented Oct 5, 2023 •

edited

Loading

🧟 Revive the Dead #112

Are you sure you want to change the base?

🧟 Revive the Dead #112

Conversation

nickumia-reisys commented Oct 5, 2023 • edited Loading

nickumia-reisys commented Oct 5, 2023 •

edited

Loading