tofu/terraform output command does not produce kubeconfig

[rafaelbeckel]

Hi, all!

I may be missing something obvious, but I followed the steps in the README, and all resources are created correctly on Hetzner.

However, when I try to run terraform output kubeconfig, I get no output at all, so I can't connect kubectl to my cluster:

> terraform output kubeconfig     

│ Warning: No outputs found
│ 
│ The state file either has no outputs defined, or all the defined outputs are empty. Please define an output in your configuration with the `output` keyword and run `tofu refresh` for it to become
│ available. If you are using interpolation, please verify the interpolated value is not empty. You can use the `tofu console` command to assist.

EDIT:

This is caused because the terraform apply process never finishes. I get stuck in an infinite loop of "Waiting for MicroOS to become available..." until the script is killed, so the output is never generated. Inside Hetzner's Web UI, I can see the servers are created and are ready.

Error: local-exec provisioner error
│ 
│   with module.kube-hetzner.module.control_planes["1-0-control-plane-nbg1"].hcloud_server.server,
│   on .terraform/modules/kube-hetzner/modules/host/main.tf line 64, in resource "hcloud_server" "server":
│   64:   provisioner "local-exec" {
│ 
│ Error running command 'timeout 600 bash <<EOF
│   until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAuthentication=yes -i /tmp/rtor37yb8bculxcxba2v -o ConnectTimeout=2 -p 2220 [email protected] true 2> /dev/null
│   do
│     echo "Waiting for MicroOS to become available..."
│     sleep 3
│   done
│ EOF
│ ': signal: killed.

Config File

# kube.tf

locals {
  hcloud_token = var.hcloud_token
}

module "kube-hetzner" {
  cluster_name = "my-cluster-name"
  source = "kube-hetzner/kube-hetzner/hcloud"

  providers = {
    hcloud = hcloud
  }
  hcloud_token = var.hcloud_token

  ssh_public_key  = file("~/.ssh/id_ed25519.pub")
  ssh_private_key = file("~/.ssh/id_ed25519")

  network_region = "eu-central"

  control_plane_nodepools = [
    {
      name        = "control-plane-fsn1"
      server_type = "cax11"
      location    = "fsn1"
      labels      = []
      taints      = []
      count       = 1
    },
    {
      name        = "control-plane-nbg1"
      server_type = "cax11"
      location    = "nbg1"
      labels      = []
      taints      = []
      count       = 1
    },
    {
      name        = "control-plane-hel1"
      server_type = "cax11"
      location    = "hel1"
      labels      = []
      taints      = []
      count       = 1
    }
  ]

  agent_nodepools = [
    {
      name        = "agent-arm-small"
      server_type = "cax11"
      location    = "fsn1"
      labels      = []
      taints      = []
      count       = 1
    }
  ]

  autoscaler_nodepools = [
    {
      name        = "autoscaled-arm"
      server_type = "cax11"
      location    = "fsn1"
      min_nodes   = 0
      max_nodes   = 10
      labels      = {
        "autoscaler.kubernetes.io/enabled" = "true"
      }
      taints      = []
    }
  ]

  create_kubeconfig = false

  ingress_controller       = "nginx"
  load_balancer_type       = "lb11"
  load_balancer_location   = "fsn1"

  automatically_upgrade_k3s = true
  automatically_upgrade_os  = true
  system_upgrade_use_drain  = true

  enable_metrics_server     = true

  initial_k3s_channel              = "stable"
  cluster_autoscaler_version       = "v1.30.0"
  cluster_autoscaler_log_level     = 4
  cluster_autoscaler_server_creation_timeout = 20

  enable_wireguard = true
}

provider "hcloud" {
  token = var.hcloud_token
}

terraform {
  required_version = ">= 1.5.0"
  required_providers {
    hcloud = {
      source  = "hetznercloud/hcloud"
      version = ">= 1.43.0"
    }
  }
}

output "kubeconfig" {
  value     = module.kube-hetzner.kubeconfig
  sensitive = true
}

variable "hcloud_token" {
  sensitive = true
  default   = ""
}

When I set create_kubeconfig to true, it doesn't create one either.

[mysticaltech]

@rafaelbeckel If you are on mac make sure you have this installed brew install coreutils

[zeidlos]

@rafaelbeckel are you sure this is your full kube.tf?
The log indicates that it wants to ssh into port 2220

until ssh -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no -o IdentitiesOnly=yes -o PubkeyAuthentication=yes -i /tmp/rtor37yb8bculxcxba2v -o ConnectTimeout=2 -p 2220 [email protected] true 2> /dev/nul

So I guess you used ssh_port = 2220?
If so, check your clusters firewall if that SSH port actually is not blocked.

I've had the same issue and filed this bug and a PR to fix it:
#1624

[rafaelbeckel]

Previously, I ran into the issue that the default file creates more servers than it's allowed by new accounts on Hetzner (they allow only 5). So I removed the redundant control planes, and now my resources are under the limits. However, I'm still stuck in the infinite loop.

I tried with SSH ports 2220 and the default 22.

I also installed coreutils and am still getting the same issue.