Home Cluster

My homelab Kubernetes cluster in declarative state. Flux automatically deploys any changes to the /cluster directory.

Overview

• Architecture
• Installation
• Thanks

Architecture

The cluster runs on a single k3s node running in a k3os VM on a single Proxmox host. I'm currently running the entire cluster on my previous desktop machine.

Virtual Hosts

• DNS server (LXC)
• k3s (VM)
• NFS fileserver (VM)

Hardware

• 4x12TB HDD
• 32GB DDR4 2400MHz RAM
• Intel i7 6700k
• Nvidia 1070
• HUSBZB-1 Zigbee + Z-Wave Adapter
• Arduino Uno + WS2812B LED strip

Networking

1. This cluster uses a custom Traefik ingress controller to configure hostnames using a subdomain of my registered tld.
2. Traefik uses cert-manager with LetsEncrypt and Cloudflare ACME DNS solver to provide certificates for all my services without exposing any ports on my router to the public internet.
3. The coreDNS server provides DNS for my home network. Using the k8s_gateway plugin it can query the cluster's ingresses.

Repository Structure

The Git repository contains the following directories under cluster and are ordered below by how Flux will apply them.

• base directory is the entrypoint to Flux
• crds directory contains custom resource definitions (CRDs) that need to exist globally in your cluster before anything else exists
• core directory (depends on crds) are important infrastructure applications (grouped by namespace) that should never be pruned by Flux
• apps directory (depends on core) is where your common applications (grouped by namespace) could be placed, Flux will prune resources here if they are not tracked by Git anymore

cluster
├── apps
│   ├── default
│   └── networking
├── base
│   └── flux-system
├── core
│   ├── cert-manager
│   ├── metallb-system
│   └── namespaces
└── crds
    └── cert-manager

As well as the following directories under server which is used to provision the infrastructure on the Proxmox host.

• packer directory contains Packer configs for used to build the custom Proxmox template images
• terraform directory contains Terraform templates used to deploy the infrastructure to Proxmox
• ansible directory contains Ansible playbooks for configuring the machines once deployed

server
├── ansible
│   ├── inventory
│   ├── playbooks
│   └── requirements.yaml
├── packer
│   ├── images
│   └── templates
└── terraform
    └── proxmox

Storage

I provisioned Proxmox using the ZFS filesystem which provides many benefits including redundancy and snapshots. My current RAID configuration (mirror vdevs, roughly RAID 10) offers 1/2 of my total storage capacity.

Storage for the k3s cluster is provided by the virtualized NFS server. The NFS Subdirectory External Provisioner Helm chart is used to provision persistent volume claims automatically.

Installation

The below steps will provision the k3s cluster as well as a fileserver and a DNS server.

Tools

These tools should be installed on the machine you'll be managing the cluster from.

Tool	Description
Terraform	Uses the Proxmox provisioner to create VMs and LXC containers
Packer	Creates custom Proxmox template images
Ansible	Performs configuration on machines once the OS is installed
kubectl	Runs commands against Kubernetes clusters
flux	Operator that manages cluster resources based on Git repositories
SOPS	Encrypts Kubernetes secrets with GnuPG
GnuPG	Encrypts and signs data
pre-commit	Runs checks before git commit
helm	Manages Kubernetes applications
prettier	Formats code
go-task	A task runner comparable to GNU make

Prerequisites

• You must have a server with Proxmox VE installed

  • Must use ZFS for storage
  • Must have sufficient storage space (I use RAID10 with 4x12tb drives)

• You must have a public/private ssh key-pair generated and added to ssh agent

  # Generate key
  ssh-keygen -t ed25519 -C "[email protected]"
  # Add to ssh agent (you may need to enable the agent separately)
  ssh-add ~/.ssh/id_ed25519

• You must have the SOPS GPG private key imported

• Install the pre-commit hooks to ensure linting runs on every commit as well as to ensure unencrypted secrets are not committed

  task pre-commit:init

• Install Ansible dependencies

  task ansible:install

Provisioning cluster

1. Double check the packer config, terraform config, and cluster settings, then add your secrets to the secrets files

   echo "proxmox_password = \"TERRAFORM ADMIN USER PASSWORD\"" >> server/terraform/proxmox/secrets.auto.tfvars
   echo "proxmox_password = \"TERRAFORM ADMIN USER PASSWORD\"" >> server/packer/images/secrets.auto.pkrvars.hcl
   echo "domain = \"your-domain.com\"" >> server/packer/images/secrets.auto.pkrvars.hcl

2. Build the template images

   task packer:all

3. Check the terraform plan and apply it

   task terraform:init
   task terraform:plan
   task terraform:apply

4. Set your DHCP server to use the new DNS server

Thanks

This cluster has been heavily inspired by the k8s@home community. Thank you to everyone that contributes there as well as to the authors of the open source technologies which operate this cluster.