Run s3 jobs via step functions #136
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
coilysiren
commented
Sep 13, 2024
coilysiren
commented
Sep 13, 2024
lorenyu
approved these changes
Sep 13, 2024
coilysiren
changed the title
|
Not important but curious why you closed this |
|
Yeah I dunno, shouldn't have closed this one before navapbc/template-infra#757 was merged |
coilysiren
added a commit
to navapbc/template-infra
that referenced
this pull request
Oct 18, 2024
## Ticket Resolves #744 ## Changes - Changes the ETL job from invoking ECS directly, to instead invoking Step Functions which then invokes ECS - Makes some changes to the scheduled changes to increase their distinction from the events jobs, and the readability of their IAM roles ## Context for reviewers This is a similar PR to #745. In-fact they use many of the same resources. To goal of this PR is to run ETL events via Step Functions. Step Functions create a tracking layer that's not available when invoking ECS directly. This tracking layer allows you to see the success and failure status of your jobs. ## Testing navapbc/platform-test#136
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Resolves navapbc/template-infra#744
Changes
This PR changes the Event Bridge file upload job from triggering an ECS task directly, to triggering a step function that then triggers an ECS task. We are doing this because Step Functions brings a basic observability layer (it shows failures and successes) to the triggering of ECS tasks.
Context for reviewers
This PR looks a lot like the scheduled jobs PR. It even shares a role with the scheduled jobs (
infra/modules/service/workflow_orchestrator_role.tf)Testing
(the gif runs through the whole process end to end, and ends up 70 seconds long)
Screen.Recording.2024-09-13.at.2.mp4
Preview environment
♻️ Environment destroyed ♻️