Perform ETL events via Step Functions

[coilysiren]

Ticket

Resolves #744

Changes

• Changes the ETL job from invoking ECS directly, to instead invoking Step Functions which then invokes ECS
• Makes some changes to the scheduled changes to increase their distinction from the events jobs, and the readability of their IAM roles

Context for reviewers

This is a similar PR to #745. In-fact they use many of the same resources. To goal of this PR is to run ETL events via Step Functions. Step Functions create a tracking layer that's not available when invoking ECS directly. This tracking layer allows you to see the success and failure status of your jobs.

Testing

navapbc/platform-test#136

[lorenyu]

Amazing looking really clean. just fyi template deploys to platform-test-nextjs and platform-test-flask are temporarily broken tanner is gonna fix them on monday see 🔒 https://nava.slack.com/archives/C03G1SWD9H7/p1727471310172749?thread_ts=1727463533.550949&cid=C03G1SWD9H7

[coilysiren]

Wow I forgot to merge this!

[coilysiren]

terratest error:

          	Error:      	Received unexpected error:
          	            	error while running command: exit status 2; ╷
          	            	│ Error: creating IAM Policy (app-dev-run-access): operation error IAM: CreatePolicy, https response error StatusCode: 400, RequestID: 7f8059b2-faf0-4408-ad4d-066694979b72, MalformedPolicyDocument: Policy statement must contain resources.
          	            	│ 
          	            	│   with module.service.aws_iam_policy.run_task,
          	            	│   on ../../modules/service/events_role.tf line 26, in resource "aws_iam_policy" "run_task":
          	            	│   26: resource "aws_iam_policy" "run_task" {
          	            	│ 
          	            	╵
          	            	╷
          	            	│ Error: creating IAM Policy (app-dev-scheduler): operation error IAM: CreatePolicy, https response error StatusCode: 400, RequestID: eb3c6499-a7ae-4f67-8c3f-9c30b21a3cc7, MalformedPolicyDocument: Policy statement must contain resources.
          	            	│ 
          	            	│   with module.service.aws_iam_policy.scheduler,
          	            	│   on ../../modules/service/scheduler_role.tf line 22, in resource "aws_iam_policy" "scheduler":
          	            	│   22: resource "aws_iam_policy" "scheduler" {
          	            	│ 
          	            	╵

[coilysiren]

each statement contains resources... that this worked on platform test... :/

[lorenyu]

@coilysiren interesting, looks like it fails when there are no scheduled jobs or event-based jobs defined, since the resources looks like resources = ["arn:aws:events:${data.aws_region.current.name}:${data.aws_caller_identity.current.account_id}:rule/StepFunctionsGetEventsForStepFunctionsExecutionRule"], so if the for loop is empty then resources would be an empty array

[lorenyu]

@coilysiren maybe go back to using the dynamic statement that you had previously rather than the pattern I recommended e.g.

dynamic "statement" {
    for_each = aws_sfn_state_machine.scheduled_jobs

    content {
      actions = [
        "states:StartExecution",
      ]
      resources = [statement.value.arn]
    }
  }

and add a comment along the lines of

# A dynamic statement is necessary since if we instead use a static statement
# with a dynamic resources block like resources = [for job in aws_sfn_state_machine.scheduled_jobs : "${job.arn}"]
# then if there are no jobs defined the resources block would be empty which IAM does not allow for AWS Step Functions
# policies, which would result in an error: "MalformedPolicyDocument: Policy statement must contain resources"

[coilysiren]

@lorenyu that makes sense! I'm on it

[lorenyu]

btw once you've made the changes and tested them and the tests all pass feel free to proceed to merge, no need to get another review from me unless there's anything you think will be suprising