Deployment rollout issue when upgrading provider from 2.0.4 to 2.1.2

[senilio]

After attempting to upgrade the kubectl provider from 2.0.4 to 2.1.2, I've had issues with Deployment resources specifically. The apply hangs until timeout.

Reproduction steps:

1. Apply a simple k8s Deployment using provider version 2.0.4:

resource "kubectl_manifest" "deployment" {
  yaml_body       = <<-EOT
apiVersion: apps/v1
kind: Deployment
metadata:
  name: httpbin
spec:
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: httpbin-service
  template:
    metadata:
      labels:
        app.kubernetes.io/name: httpbin-service
    spec:
      containers:
        - image: kennethreitz/httpbin
          name: httpbin
          ports:
            - containerPort: 80
              protocol: TCP
EOT
}

2. Upgrade provider version to 2.1.2
3. Run terraform apply again, which will result in the following:

Terraform will perform the following actions:

  # kubectl_manifest.deployment will be updated in-place
  ~ resource "kubectl_manifest" "deployment" {
        id                      = "/apis/apps/v1/namespaces/default/deployments/httpbin"
        name                    = "httpbin"
      ~ yaml_incluster          = (sensitive value)
        # (15 unchanged attributes hidden)
    }

This diff seems to come from the addition of the live_manifest_incluster field in the state.
The apply now hangs indefinitely.

Notes:
Adding wait_for_rollout = false prior to applying with 2.1.2 will fix the issue.

Do I need to disable wait_for_rollout for my existing resources before upgrading provider version, or is this a bug? I don't believe any changes are made to the actual manifest by upgrading the provider, so waiting for Rollout shouldn't be needed.

TF debug error after timeout is:

[ERROR] updating manifest failed: httpbin failed to rollout Deployment

I noticed this issue with Deployment, but I assume it goes for DaemonSet and StatefulSet as well.

Thanks!

[alekc]

I have actually experienced this on 2.0.4 as well. Cni was half broken which caused delay in rollout, and this is the output

Initializing the backend...
Initializing provider plugins...
- Finding alekc/kubectl versions matching "2.0.4"...
- Installing alekc/kubectl v2.0.4...
- Installed alekc/kubectl v2.0.4 (self-signed, key ID 772FB27A86DAFCE7)
Partner and community providers are signed by their developers.
If you'd like to know more about provider signing, you can read about it here:
https://www.terraform.io/docs/cli/plugins/signing.html
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above. Include this file in your version control repository
so that Terraform can guarantee to make the same selections by default when
you run "terraform init" in the future.

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

11:50:55 in terraform-provider-kubectl/test on  master [?] via 🐹 v1.22.6 took 2.3s 
➜ terraform apply -auto-approve

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # kubectl_manifest.deployment will be created
  + resource "kubectl_manifest" "deployment" {
      + api_version             = "apps/v1"
      + apply_only              = false
      + field_manager           = "kubectl"
      + force_conflicts         = false
      + force_new               = false
      + id                      = (known after apply)
      + kind                    = "Deployment"
      + live_manifest_incluster = (sensitive value)
      + live_uid                = (known after apply)
      + name                    = "httpbin"
      + namespace               = (known after apply)
      + server_side_apply       = false
      + uid                     = (known after apply)
      + validate_schema         = true
      + wait_for_rollout        = true
      + yaml_body               = (sensitive value)
      + yaml_body_parsed        = <<-EOT
            apiVersion: apps/v1
            kind: Deployment
            metadata:
              name: httpbin
            spec:
              replicas: 1
              selector:
                matchLabels:
                  app.kubernetes.io/name: httpbin-service
              template:
                metadata:
                  labels:
                    app.kubernetes.io/name: httpbin-service
                spec:
                  containers:
                  - image: kennethreitz/httpbin
                    name: httpbin
                    ports:
                    - containerPort: 80
                      protocol: TCP
        EOT
      + yaml_incluster          = (sensitive value)
    }

Plan: 1 to add, 0 to change, 0 to destroy.
kubectl_manifest.deployment: Creating...
kubectl_manifest.deployment: Still creating... [10s elapsed]
kubectl_manifest.deployment: Still creating... [20s elapsed]
kubectl_manifest.deployment: Still creating... [30s elapsed]
kubectl_manifest.deployment: Still creating... [40s elapsed]
kubectl_manifest.deployment: Still creating... [50s elapsed]
kubectl_manifest.deployment: Still creating... [1m0s elapsed]
kubectl_manifest.deployment: Still creating... [1m10s elapsed]
kubectl_manifest.deployment: Still creating... [1m20s elapsed]
kubectl_manifest.deployment: Still creating... [1m30s elapsed]
kubectl_manifest.deployment: Still creating... [1m40s elapsed]
kubectl_manifest.deployment: Still creating... [1m50s elapsed]
kubectl_manifest.deployment: Still creating... [2m0s elapsed]
kubectl_manifest.deployment: Still creating... [2m10s elapsed]
kubectl_manifest.deployment: Still creating... [2m20s elapsed]
kubectl_manifest.deployment: Still creating... [2m30s elapsed]
kubectl_manifest.deployment: Still creating... [2m40s elapsed]

Upon checking in the docs, it looks like its working as expected

Closing this one off

[yongzhang]

I have the same issue when upgrading from 2.0.4 to 2.1.3, why closing this issue? The problem is that, I didn't see kubernetes roll out deployment changes.

@alekc Your example seems for create, could you reproduce a case for update in place?

[yongzhang]

By checking the commit history, I guess this is caused by #112.

Because there're actually no kubernetes yaml changes, so deployments won't have rolling updates, then it get stuck.