SPIKE: investigate supporting easy terraform locking via cloud.gov-managed services

[rahearn]

Background

Currently, most apps utilizing terraform to manage cloud.gov infrastructure use the s3 backend for state storage, but have no mechanism to lock terraform runs, increasing the risk of bad interactions via concurrent terraform runs by multiple CI branches and/or developers

Potential options:

PG backend

https://developer.hashicorp.com/terraform/language/settings/backends/pg

Does both state storage and locking withing postgres

Expose DynamoDB as an SSB service

Continue to utilize s3 for state and make it easy to pair with DynamoDB for locking

OpenTofu DynamoDB-free locking in S3

opentofu/opentofu#599 - currently just a proposal, but would likely be the cheapest and easiest to operate if it exists by the time this is worked on.

[mogul]

Side note: There's a proposal to add a DynamoDB-free locking mechanism to the S3 backend in OpenTofu... If that happens before we work on this, then it may be not worth doing too much to encourage greater use of pg; an RDS instance is more expensive for cloud.gov to operate than an S3 bucket is. ;)