Terraform replan timed out while waiting for condition

[eloo-abi]

Describe the bug

Doing a terraform replan i get the whole time a timeout.
timed out waiting for the condition

When i check the terraform k8s object i see the replan is triggered but i never get the result in my cli.

Environment

• Weave-Gitops Version
• Flux Version
• Kubernetes version

To Reproduce
Steps to reproduce the behavior:

1. gitops replan terraform tf-resource
2. Get this error Error: timed out waiting for the condition

Expected behavior

I would expect to get the plan output in the console instead of an error.

Actual Behavior

$ gitops replan terraform jowe-tf-controller-test  
 Replan requested for flux-system/jowe-tf-controller-test
Error: timed out waiting for the condition

Additional Context (screenshots, logs, etc)

Here we can see that the replan is triggered

jowe-tf-controller-test   True      No drift: main@sha1:d77ffe193f67dfe74f93205d4aea5187b00c89ce   4h46m
jowe-tf-controller-test   False     Replan requested                                               4h46m
jowe-tf-controller-test   False     Replan requested                                               4h46m
jowe-tf-controller-test   Unknown   Reconciliation in progress                                     4h46m
jowe-tf-controller-test   Unknown   Initializing                                                   4h47m
jowe-tf-controller-test   Unknown   Terraform Planning                                             4h47m
jowe-tf-controller-test   True      Plan no changes: main@sha1:d77ffe193f67dfe74f93205d4aea5187b00c89ce   4h47m

[bigkevmcd]

@eloo-abi Do you have any logs from the process, how long did it take? did it fail?

[eloo-abi]

according to the runtime of the pods i would say its just a few seconds

jowe-tf-controller-test-tf-runner               1/1     Running             0          5s
jowe-tf-controller-test-tf-runner               1/1     Running             0          11s
jowe-tf-controller-test-tf-runner               1/1     Terminating         0          21s
jowe-tf-controller-test-tf-runner               1/1     Terminating         0          21s
jowe-tf-controller-test-tf-runner               0/1     Terminating         0          23s
jowe-tf-controller-test-tf-runner               0/1     Terminating         0          23s
jowe-tf-controller-test-tf-runner               0/1     Terminating         0          23s
jowe-tf-controller-test-tf-runner               0/1     Pending             0          0s
jowe-tf-controller-test-tf-runner               0/1     Pending             0          0s
jowe-tf-controller-test-tf-runner               0/1     ContainerCreating   0          0s
jowe-tf-controller-test-tf-runner               1/1     Running             0          5s
jowe-tf-controller-test-tf-runner               1/1     Running             0          10s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Pending             0          0s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Pending             0          0s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     ContainerCreating   0          0s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Running             0          4s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Running             0          4s
jowe-tf-controller-test-tf-runner                  1/1     Terminating         0          20s
jowe-tf-controller-test-tf-runner                  1/1     Terminating         0          20s
jowe-tf-controller-test-tf-runner                  0/1     Terminating         0          22s
jowe-tf-controller-test-tf-runner                  0/1     Terminating         0          22s
jowe-tf-controller-test-tf-runner                  0/1     Terminating         0          22s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Terminating         0          26s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Terminating         0          26s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Terminating         0          28s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Terminating         0          28s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Terminating         0          28s
jowe-tf-controller-test-tf-runner                  0/1     Pending             0          0s
jowe-tf-controller-test-tf-runner                  0/1     Pending             0          0s
jowe-tf-controller-test-tf-runner                  0/1     ContainerCreating   0          0s
jowe-tf-controller-test-tf-runner                  1/1     Running             0          3s
jowe-tf-controller-test-tf-runner                  1/1     Running             0          9s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Pending             0          0s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     Pending             0          0s
jowe-tf-controller-test-workspace-test-tf-runner   0/1     ContainerCreating   0          0s
jowe-tf-controller-test-tf-runner                  1/1     Terminating         0          20s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Running             0          4s
jowe-tf-controller-test-workspace-test-tf-runner   1/1     Running             0          4s
jowe-tf-controller-test-tf-runner                  1/1     Terminating         0          20s

[bigkevmcd]

Hi @eloo-abi thanks for this, I've had a look, and it should wait for up to 30 seconds before timing out.

But, I don't see any cases in your output that indicate that any are reaching that long.

Could a replan take over 30s?

If so, we could allow you to configure the timeout value, but clearly, it can (and will) timeout at some point.

[eloo-abi]

as the plan is just creating a file (some example) i guess its not related to long planning..

maybe its more about the "waiting for condition" part..

i'm not sure what the implementation there is.. maybe its waiting to have the terraform resource in a specific state?
and they do stuff?

because i have seen similar implementations in the past and mostly the problem was that the condition/state was changed to fast for the cli to remember..

e.g. we assume the cli is awaiting the state 2 and polls every two seconds

and the resources states is changing like this

t0: state 1
t1 : state 2
t2: state 3
t3: state 3
t4: state 4

then the cli would only see

state 1
state 3

and as it is waiting for "state 2" its going to timeout

could this maybe the case?

[bigkevmcd]

@eloo-abi I'm reading the code to see if that could be the case, but I don't think so, it's basically waiting for the plan pending state to be populated.

https://github.com/weaveworks/tf-controller/blob/main/tfctl/replan.go#L39-L54

I'm trying to reproduce this.

[chanwit]

We have recently fixed a related issue in the Terraform Controller:

flux-iac/tofu-controller#825

The fix is available for you to try from the main branch image.

[chanwit]

Cc @yiannistri