Terraform keeps saying there's a change to be applied to a heroku_config resource, even after applying.

[oboxodo]

Terraform Version

$ terraform -v
Terraform v1.3.4
on linux_amd64
+ provider registry.terraform.io/davidji99/herokux v1.1.0
+ provider registry.terraform.io/heroku/heroku v5.1.5

Affected Resource(s)

• heroku_config

Terraform Configuration Files

This is a simplified and anonymized version but should represent faithfully the core of my config.

resource "heroku_space" "my-app-space" {
  name         = "my-app-space"
  organization = "myorg"
  region       = "virginia"
}

resource "heroku_app" "my-app" {
  name   = "my-app"
  region = "virginia"
  stack  = "heroku-20"
  acm    = false
  space  = heroku_space.my-app-space.id
  organization {
    name   = "myorg"
    locked = true
  }
}

variable "SOME_SENSITIVE_VAR_1" {
  type      = string
  sensitive = true
}

resource "heroku_config" "production" {
  vars = {
    SOME_VAR_1 = "value"
    # more values here...
  }

  sensitive_vars = {
    SOME_SENSITIVE_VAR_1 = var.SOME_SENSITIVE_VAR_1
    # more values here...
  }
}

resource "heroku_app_config_association" "my-app-production" {
  app_id         = heroku_app.my-app.id
  sensitive_vars = heroku_config.production.sensitive_vars
  vars           = heroku_config.production.vars
}

# We also have a bunch of heroku_addon, heroku_addon_attachment, heroku_app_feature,
# and herokux_scheduler_job but all of those are unrelated to heroku_config.

Debug Output

I am uploading only the tail of the output I got from executing TF_LOG=trace terraform apply -var-file=secrets.tfvars 2>trace.log which I think might cover what you need as it mentions heroku_config a lot and it seems to end with an error. I'd rather not upload the complete output as it could have sensitive info. But if you tell me something specific to look for I could do that.

Tail of debugging trace: https://gist.github.com/oboxodo/f463c7217a0b119669cb901b5062cdab

Panic Output

No panic.

Expected Behavior

Executing a plan after applying a change should say there's nothing to change.

Actual Behavior

The same change keeps being reported as needed, even after applying it successfully (which really does nothing). I also tried applying a real change, adding a new env var. It added the new env var and then again it continued saying there's a change to apply with no real diff.

Steps to Reproduce

1. terraform plan says there's a change to apply in heroku_config resource, but no diff.
2. terraform apply shows the same empty needed change. Type "yes" and confirm. It ends saying it applied the change.
3. terraform plan keeps showing the same empty change needed in heroku_config resource.

Here you can see an anonymized execution from my console:

$ terraform apply -var-file=secrets.tfvars
Acquiring state lock. This may take a few moments...
heroku_space.my-app-space: Refreshing state... [id=my-app-space]
heroku_config.production: Refreshing state... [id=config-1667227535]
heroku_app.my-app: Refreshing state... [id=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy]
heroku_app_config_association.my-app-production: Refreshing state... [id=config:yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy]
#...output about unrelated resources removed for brevity...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # heroku_config.production will be updated in-place
  ~ resource "heroku_config" "production" {
        id             = "config-1667227535"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

Do you want to perform these actions in workspace "my-app-workspace"?
  Terraform will perform the actions described above.
  Only 'yes' will be accepted to approve.

  Enter a value: yes

heroku_config.production: Modifying... [id=config-1667227535]
heroku_config.production: Modifications complete after 0s [id=config-1667227535]

Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

$ terraform plan -var-file=secrets.tfvars
heroku_space.my-app-space: Refreshing state... [id=my-app-space]
heroku_config.production: Refreshing state... [id=config-1667227535]
heroku_app.my-app: Refreshing state... [id=yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy]
heroku_app_config_association.my-app-production: Refreshing state... [id=config:yyyyyyyy-yyyy-yyyy-yyyy-yyyyyyyyyyyy]
#...output about unrelated resources removed for brevity...

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # heroku_config.production will be updated in-place
  ~ resource "heroku_config" "production" {
        id             = "config-1667227535"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 1 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you run "terraform apply" now.

Important Factoids

This started happening after we added a couple of env vars via Heroku's CLI, then we removed them. So no real diff should be shown. Plus, after this I've applied other changes via terraform and it keeps saying heroku_config needs an update 🤷‍♂️.

State is in Terraform Cloud. I tried both local and remote execution and result is the same.

Here's a screenshot so you see exactly what I see with both plan and apply.

[mars]

Hi @oboxodo 😄

I don't know how config vars being set directly on that app would affect this, because heroku_config is only local to Terraform. It's state propagates out to heroku_app_config_association via the dynamic attributes, and from there to the Heroku API to set them on the app.

Honestly, this heroku_config resource is pretty basic. It just saves two maps into state. I don't see what could cause such behavior.

Would you please try adding a new heroku_config.production_2 in your config, with the same values, and associate its values with the app associations, and delete the original, problematic heroku_config.production?

Hopefully the replacement will not show this behavior. Please let us know how it goes.

[oboxodo]

Hi @mars, thanks for your suggestion.

I did as you suggested... creating a new heroku_config.production_2 resource as an exact copy of heroku_config.production. And updated heroku_app_config_association.my-app-production to use the new config resource vars instead.

The new resource was created just fine. But then if I plan/apply again, having made zero changes, I get the same unexpected result :'(

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # heroku_config.production will be updated in-place
  ~ resource "heroku_config" "production" {
        id             = "config-1667227535"
        # (2 unchanged attributes hidden)
    }

  # heroku_config.production_2 will be updated in-place
  ~ resource "heroku_config" "production_2" {
        id             = "config-1668774260"
        # (2 unchanged attributes hidden)
    }

Plan: 0 to add, 2 to change, 0 to destroy.

[davidji99]

Maybe something changed in terraform core because the plan diff doesn't actually show any changes.

[mars]

Perhaps a Terraform Plugin SDK update would fix it. Currently at v2.17.0, with newer v2.24.1 available.

[mars]

@oboxodo I am unable to reproduce this issue, using the versions and configuration you documented. Thanks for the details, though!

So, all I can suggest is updating to the release v5.1.8 (or newer) which includes the newest Terraform Plugin SDK and will hopefully resolve the problem for you.

[oboxodo]

Sorry for the late reply, @mars. I wasn't able to focus on this again until now.

So, all I can suggest is updating to the release v5.1.8 (or newer) which includes the newest Terraform Plugin SDK and will hopefully resolve the problem for you.

I upgraded the heroku provider from 5.1.6 to 5.1.9, and terraform itself from 1.3.4 to 1.3.6 and the result continues being the same 😢.

[mars]

Can you reproduce this issue in an isolated, complete Terraform config? Are you positive that nothing dynamic is impacting the heroku_config, like a var that is interpolated?

I took your example config to attempt reproducing the bug using the same versions, but the configuration works correctly for me. It applies cleanly.

I also noticed that your have registry.terraform.io/davidji99/herokux v1.1.0 listed in your versions, but I do not see it being used in the example config. Is that meaningful?

Finally, is your tfvars file dynamically generated? Could it be the source of the changes?

[davidji99]

I wonder if using a sensitive variable as a value for a resource attribute causes this constant diff. What if you drop sensitive from this variable temporarily:

variable "SOME_SENSITIVE_VAR_1" {
  type      = string
  sensitive = true
}

[mars]

@davidji99 sensitivity of vars had no effect in my attempt to reproduce this issue.

[davidji99]

@davidji99 sensitivity of vars had no effect in my attempt to reproduce this issue.

Good to know!

[jasonl]

I've been experiencing a very similar issue:

Terraform will perform the following actions:

  # heroku_app.redacted-production will be updated in-place
  ~ resource "heroku_app" "redacted-production" {
        id                    = "<redacted UUID>"
        name                  = "redacted-production"
        # (12 unchanged attributes hidden)

        # (1 unchanged block hidden)
    }

I can apply the configuration, but immediately running it again flags the same resources to be updated in place. This is with Terraform 1.0.11 and heroku-provider 5.2.4

I don't use a separate heroku_config resource, but instead specify a config_vars block for the heroku_app resource.