Local file content stored in state can cause extremely slow plans/applies

[caramcc]

Terraform CLI and Provider Versions

$ terraform version
Terraform v1.3.5
on darwin_arm64

• provider registry.terraform.io/datadog/datadog v3.29.0
• provider registry.terraform.io/hashicorp/archive v2.2.0
• provider registry.terraform.io/hashicorp/aws v5.13.1
• provider registry.terraform.io/hashicorp/github v5.33.0
• provider registry.terraform.io/hashicorp/local v2.4.0
• provider registry.terraform.io/hashicorp/null v3.2.1

Terraform Configuration

data "local_file" "release_object" {
  filename = local.path_to_15mb_file
}

Expected Behavior

terraform plan and terraform apply operations are completed in a timely manner.

Actual Behavior

terraform plan and terraform apply operations take multiple minutes to complete.

Since the data.local_file.release_object object stored in Terraform state functionally stores 2 complete copies of the contents of the file (as the content and content_base64 attributes), any modestly-sized files managed with the local_file data source can cause the Terraform state file to become so large that any operations that reference the state file take several minutes to complete.

In my case, reading a 15MB file in data.local_file in two places ultimately generated a 132MB Terraform state file, which made plan and apply operations take 4-5 minutes, particularly when using a cloud-based Terraform state backend such as AWS S3.

Steps to Reproduce

1. Reference any multiple-megabyte file in the local_file data source
2. terraform apply
3. Observe the large terraform state file size and slower plan/apply time.

How much impact is this issue causing?

Low

Logs

No response

Additional Information

It would be nice if this behavior was at least documented in the local_file provider so it doesn't surprise anyone else. I was mostly able to work around this issue by using Terraform's built-in file functions.

Code of Conduct

• I agree to follow this project's Code of Conduct

[jkworth]

I concur with this being an issue. The cause of the slowness seems to be that the resource stores the content of the file in state in 2 forms. The content, in my case seems to be stored in byte form. Then it's also stored in another field called content_base64 field. In one case i had a 43MB binary file. That then gets copied in to the state twice but due to it needing to be stored in a json file format. It gets converted to something that the content property's value takes up 156MB. And the content_base64 is 58MB.

I don't know if the need to store those 2 properties in the state in really needed but just excluding them took my state file from 1.8GB, yes that is the correct size. To just 243KB. It had 8 local_file data resources in it.

I will be testing to see if the local_sensitive_file is a better fit since the attributes are all marked as much and hope they are left out of the state.

[jkworth]

Using the local_sensitive_file does not help. Still seems to store the file's contents.