Skip to content

Latest commit

 

History

History
182 lines (150 loc) · 11.2 KB

README.md

File metadata and controls

182 lines (150 loc) · 11.2 KB

Control plane EKS cluster

  • Create a management EKS cluster in a single zone
  • Use SPOT instances
  • Use Amazon Linux AMI (Currently Bottlerocket doesn't work and having issues with AL2023)
  • Install and configure Karpenter
  • Install and configure Flux
  • Write a secret that contains the cluster's specific variables that will be used with Flux. (please refer to variables substitutions)
  • Deploy Cilium

How to apply this?

  1. Edit the file backend.tf and put your own S3 bucket name.
  2. Create a file that contains your own variables. Here's an example:

variables.tfvars

env          = "dev"
cluster_name = "mycluster-0"

flux_sync_repository_url = "https://github.com/Smana/cloud-native-ref.git"

tags = {
  GithubRepo = "cloud-native-ref"
  GithubOrg  = "Smana"
}


karpenter_limits = {
  "default" = {
    cpu    = "20"
    memory = "64Gi"
  }
  "io" = {
    cpu    = "20"
    memory = "64Gi"
  }
}

# Optional if an external OIDC provider should be used to authenticate users
cluster_identity_providers = {
  zitadel = {
    client_id      = "702vqsrjicklgb7c5b7b50i1gc"
    issuer_url     = "https://auth.cloud.ogenki.io"
    username_claim = "email"
    groups_claim   = "groups"
  }
}
  1. Apply with :information_source: Git branch or tag in the format refs/heads/main or refs/tags/v1.0.0 tofu apply -var-file variables.tfvars --var=git_ref=<flux_git_ref>

Cleaning things up

In order to really clean everything you should follow these steps:

  1. Suspend Flux reconciliations

    flux suspend kustomization --all

  2. Delete Gateways (These create AWS loadbalancers)

    kubectl delete gateways --all-namespaces --all

  3. Wait 3/4 minutest and delete all IRSA and EPI

    kubectl delete irsa,epi --all-namespaces --all

  4. tofu destroy --var-file variables.tfvars

One step:

flux suspend kustomization --all && \
kubectl delete gateways --all-namespaces --all && sleep 60 && \
kubectl delete irsa,epi --all-namespaces --all && sleep 30 && \
tofu destroy --var-file variables.tfvars

Requirements

Name Version
terraform ~> 1.4
aws ~> 5.0
flux 1.4.0
github ~> 6.0
helm >= 2.7
http >= 3.4
kubectl >= 2.0.0
kubernetes >= 2.20
random ~> 3.5
tls ~> 4.0

Providers

Name Version
aws ~> 5.0
aws.virginia ~> 5.0
helm >= 2.7
http >= 3.4
kubectl >= 2.0.0
kubernetes >= 2.20

Modules

Name Source Version
eks terraform-aws-modules/eks/aws ~> 20
irsa_crossplane terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks 5.52.2
irsa_ebs_csi_driver terraform-aws-modules/iam/aws//modules/iam-role-for-service-accounts-eks 5.52.2
karpenter terraform-aws-modules/eks/aws//modules/karpenter ~> 20.0

Resources

Name Type
aws_eks_pod_identity_association.karpenter resource
aws_iam_policy.crossplane_ec2 resource
aws_iam_policy.crossplane_eks resource
aws_iam_policy.crossplane_iam resource
aws_iam_policy.crossplane_kms resource
aws_iam_policy.crossplane_s3 resource
helm_release.aws_ebs_csi_driver resource
helm_release.cilium resource
helm_release.flux-operator resource
helm_release.karpenter resource
kubectl_manifest.flux resource
kubectl_manifest.gateway_api_crds resource
kubectl_manifest.karpenter resource
kubernetes_annotations.gp2 resource
kubernetes_secret.flux_system resource
kubernetes_storage_class_v1.gp3 resource
aws_caller_identity.this data source
aws_ecrpublic_authorization_token.token data source
aws_eks_cluster_auth.cluster_auth data source
aws_secretsmanager_secret_version.github_pat data source
aws_security_group.tailscale data source
aws_subnets.intra data source
aws_subnets.private data source
aws_vpc.selected data source
http_http.gateway_api_crds data source
kubectl_filename_list.flux data source
kubectl_filename_list.karpenter_default data source
kubectl_filename_list.karpenter_io data source

Inputs

Name Description Type Default Required
cilium_version Cilium cluster version string "1.16.5" no
cluster_identity_providers Map of cluster identity provider configurations to enable for the cluster. any {} no
cluster_name Name of the EKS cluster to be created string n/a yes
cluster_version k8s cluster version string "1.31" no
ebs_csi_driver_chart_version EBS CSI Driver Helm chart version string "2.25.0" no
enable_flux_image_update_automation Enable Flux image update automation bool false no
enable_ssm If true, allow to connect to the instances using AWS Systems Manager bool false no
env The environment of the EKS cluster string n/a yes
flux_git_ref Git branch or tag in the format refs/heads/main or refs/tags/v1.0.0 string n/a yes
flux_operator_version Flux Operator version string "0.12.0" no
flux_sync_repository_url The repository URL to sync with Flux string n/a yes
gateway_api_version Gateway API CRDs version string "v1.2.0" no
github_token_secretsmanager_id SecretsManager id from where to retrieve the Github Personal Access Token. (The key must be 'github-token') string "github/flux-github-pat" no
iam_role_additional_policies Additional policies to be added to the IAM role map(string) {} no
karpenter_limits Define limits for Karpenter per node pool. 
map(object(
    {
      cpu    = optional(number, 50),
      memory = optional(string, "50Gi")
    }
    )
  )
 n/a yes
karpenter_version Karpenter version string "1.1.1" no
region AWS Region string "eu-west-3" no
tags A map of tags to add to all resources map(string) {} no

Outputs

No outputs.