Skip to content

Latest commit

 

History

History
127 lines (99 loc) · 3.83 KB

README.md

File metadata and controls

127 lines (99 loc) · 3.83 KB

Prerequisites

Tools Installation

Install the required tools:

  • Terraform
  • kubectl
  • AWS CLI

For macOS:

brew install terraform kubectl awscli

Set Up AWS Credentials

Edit the ~/.aws/credentials file:

[awsadrian]
aws_access_key_id=
aws_secret_access_key=

Save and source your profile:

source ~/.zshrc
export AWS_PROFILE="awsadrian"

Initialize the Terraform Backend

If no tfstate is initiated or available, create a backend in AWS to store the tfstate:

This will add an S3 bucket for storing the tfstate.

do not apply if already applied as you will not be able to write over without the tfstate

cd eks-setup/backend/
tf init
tf plan

How to apply the code

cd eks-setup/
tf init
tf plan -var-file=dev.tfvars
tf apply -var-file=dev.tfvars

Terraform Structure

Base: VPC and Networking

  • VPC with its dependencies
  • Security groups
  • Public and private subnets with a NAT gateway

Kubernetes

  • EKS Cluster
  • EKS CNI and auth
  • EKS Node Group: Added only to one subnet for simplicity and cost-efficiency
  • IAM Policies:
    • Roles
    • Groups
    • Policies
    • Service Accounts (SA)
  • Policies folder for better visibility
  • Cluster Autoscaler
  • Storage setup
  • Minimum Applications:
    • Cluster Autoscaler
    • EBS-CSI
    • ArgoCD

ArgoCD Setup

  • ArgoCD is applied and configured with:
    • Values for individual setups
    • Config folder containing:
      • Applications: YAML files for deployed apps
      • Projects: Definitions for app separation and deployment locations

Application Layer

All deployments are managed via ArgoCD. Applications include:

  • monitor_grafana.yaml
  • monitor_prometheus.yaml
  • system_alb: AWS Application Load Balancer
  • system_metric-server

Commands for Access

ArgoCD Server

kubectl port-forward deployment/argo-cd-argocd-server 8080:8080 -n argo-system

Grafana

kubectl port-forward service/grafana 3000:80 -n monitoring-system

Access Grafana at http://localhost:3000. The password is stored in the configuration and should be secured.

Monitoring Dashboards

Final Ideas

  • This is not an ideal format as normally I would separate the VPC, the EKS, and the next layer as an ArgoCD deployment, but to keep it simpler:

    • We are using the default workspace. With the above separation, we should have multiple workspaces and multiple tfstates. This would provide better scalability and options to change/remove things easier and safer.
    • There is no ideal way to apply this, and sometimes we need to wait for or use other tricks to make this setup work.

  • When ArgoCD is applied, we encounter a chicken-and-egg situation where we cannot apply everything in one go. There are three options to make this work:

    1. Target for the first time: Preferable, but this requires an extra step and a separate tfstate (decided against).
    2. Apply as a shell instead of a Terraform resource: After this is applied, it needs to be applied manually or targeted and forced (chosen option).
    3. Helm install manually outside of Terraform: Similar to option 2 but more manual.
    4. Use Terragrunt with hooks: Perhaps the ideal way but involves using Terragrunt, which introduces other challenges.

  • Access is open to the public, and permissions are assigned to only two users.

  • I’ve used Cluster Autoscaler instead of Karpenter. For non-critical environments like monitoring or non-production, Karpenter with spot instances would be cheaper.