helm charts failing deployment via terraform, working via helm cli

[anthosz]

Terraform, Provider, Kubernetes and Helm Versions

Terraform version: v1.10.3
Provider version: ~2.17
Kubernetes version: 1.31.4 via talos v1.9.1

Affected Resource(s)

• helm_release: cilium 1.16.5
• helm_repository: https://helm.cilium.io/

Terraform Configuration Files

resource "helm_release" "cilium" {
  name             = "cilium"
  repository       = "https://helm.cilium.io/"
  chart            = "cilium"
  namespace        = "cilium"
  create_namespace = true
  timeout          = 300
  version          = "1.19.5
  wait             = true
  atomic           = true
  set {
    name  = "ipam.mode"
    value = "kubernetes"
  }
  set {
    name  = "kubeProxyReplacement"
    value = true
  }
  set {
    name  = "securityContext.capabilities.ciliumAgent"
    value = "{CHOWN,KILL,NET_ADMIN,NET_RAW,IPC_LOCK,SYS_ADMIN,SYS_RESOURCE,DAC_OVERRIDE,FOWNER,SETGID,SETUID}"
  }
  set {
    name  = "securityContext.capabilities.cleanCiliumState"
    value = "{NET_ADMIN,SYS_ADMIN,SYS_RESOURCE}"
  }
  set {
    name  = "cgroup.autoMount.enabled"
    value = false
  }
  set {
    name  = "cgroup.hostRoot"
    value = "/sys/fs/cgroup"
  }
  set {
    name  = "k8sServiceHost"
    value = "localhost"
  }
  set {
    name  = "k8sServicePort"
    value = 7445
  }
  set {
    name  = "ingressController.enabled"
    value = true
  }
  depends_on = [talos_machine_bootstrap.talos]
}

Steps to Reproduce

1. Check that ingressController.enabled is well set to true (only way to reproduce)
2. terraform apply

Expected Behavior

cilium release deployed

Actual Behavior

cilium release stuck to "pending-upgrade" or "pending-install"

$ helm -n cilium status cilium
NAME: cilium
LAST DEPLOYED: Fri Jan 17 20:24:00 2025
NAMESPACE: cilium
STATUS: pending-upgrade
REVISION: 2
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.16.5.

For any further help, visit https://docs.cilium.io/en/v1.16/gettinghelp

Important Factoids

Running on talos v1.9.1
Procedure adapted from https://www.talos.dev/v1.9/kubernetes-guides/network/deploying-cilium/

Each time I want to set ingressController.enabled value to true from terraform, it block to pending (even if I upgrade or install).

What is frustrating is that I cannot found what can explain that. According to the chart code, there is job or something like that triggered when we enable it but only some rbac/roles creations and cannot find any logs that can explain these logs (and cannot check the logs because I can only reproduce it via tf module).

If I try to patch it or install it from helm, all good.

$ helm -n cilium upgrade cilium cilium/cilium --reuse-values --set ingressController.enabled=true
Release "cilium" has been upgraded. Happy Helming!
NAME: cilium
LAST DEPLOYED: Fri Jan 17 20:40:49 2025
NAMESPACE: cilium
STATUS: deployed
REVISION: 4
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.16.5.

For any further help, visit https://docs.cilium.io/en/v1.16/gettinghelp

References

#467

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

[vaughnw128]

Hi @anthosz! Just wanted to ditto this issue. I'm actually encountering the exact same issue with my deployment of Cilium on Talos using both helm provider 3.0.0-pre1, and 2.17.0.

I bypass this currently by supplying wait = false to the resource block.

resource "helm_release" "cilium" {
  depends_on = [kubectl_manifest.gateway_crds, talos_cluster_kubeconfig.kubeconfig]
  name       = "cilium"
  chart      = "./charts/cilium"
  namespace  = "kube-system"

  wait = false
}

I'm curious to see what comes back as the reason, but thought you may like this fix in the meantime.

[anthosz]

Hi,

Finaly the issue is close. The fact is that cilium ingress waited an IP attribution.

So it's not really tf the issue but the fact that helm is not so verbose :/

Thank you!