Add scheduled jobs #745
Merged
Add scheduled jobs #745
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
lorenyu
approved these changes
Sep 1, 2024
There was a problem hiding this comment.
After this is merged don't forget to:
- Check that template-only-cd.yml ran ok, if not you'll need to manually push changes to the platform-test* repos for the ones that failed. (Don't forget to update .template-version in the platform-test* repos if you need to do a manual update)
- Uncomment the cron job in platform-test (feel free to make the change directly on main, no need to open a PR)
Comment on lines
8
to
13
| scheduled_jobs = { | ||
| cron = { | ||
| task_command = ["python", "-m", "flask", "--app", "app.py", "cron"] | ||
| schedule_expression = "cron(0 * ? * * *)" | ||
| } | ||
| } |
There was a problem hiding this comment.
For the template we want this commented out since it's just an example, similar to what we did in file_upload_jobs.tf, then after this is merged in template-infra, we want to uncomment this config in platform-test so that we can exercise the functionality.
|
Another way to update platform-test if the template CD doesn't work is to:
|
coilysiren
added a commit
to navapbc/platform-test
that referenced
this pull request
Sep 27, 2024
This was referenced Sep 27, 2024
coilysiren
added a commit
to HHS/simpler-grants-gov
that referenced
this pull request
Oct 17, 2024
## Summary Relates to: #2351, #2472 ### Time to review: __5 mins__ ## Changes proposed There are two goals to this PR: 1. Make a material change that gets us closer in line with the platform. Specifically we are pulling in navapbc/template-infra#745 2. Updates the step functions in advance of adding another step function - for the terraform in ECS stuff ## Context for reviewers The majority of the diff is security related, specifically: - via guidance from @lorenyu, the permissions for managing the step function and workflow triggers were moved off of the task exec role. that's the diff on `infra/modules/service/access-control.tf` - those permissions were moved to `infra/modules/service/scheduler_role.tf` and `infra/modules/service/workflow_orchestrator_role.tf` The rest of the diff is around easy of configuration: - `infra/modules/service/scheduled_jobs.tf` represents the new, highly configurable, way to create scheduled jobs - `infra/api/app-config/env-config/scheduled_jobs.tf` shows how two scheduled jobs are configured ## Additional information <img width="1238" alt="image" src="https://github.com/user-attachments/assets/24e6e50e-74b0-4cf8-b61b-bad34af90118"> <img width="1237" alt="image" src="https://github.com/user-attachments/assets/aa297341-8d7d-4ac7-bcc9-ef102df25e05">
doug-s-nava
pushed a commit
to HHS/simpler-grants-gov
that referenced
this pull request
Oct 18, 2024
## Summary Relates to: #2351, #2472 ### Time to review: __5 mins__ ## Changes proposed There are two goals to this PR: 1. Make a material change that gets us closer in line with the platform. Specifically we are pulling in navapbc/template-infra#745 2. Updates the step functions in advance of adding another step function - for the terraform in ECS stuff ## Context for reviewers The majority of the diff is security related, specifically: - via guidance from @lorenyu, the permissions for managing the step function and workflow triggers were moved off of the task exec role. that's the diff on `infra/modules/service/access-control.tf` - those permissions were moved to `infra/modules/service/scheduler_role.tf` and `infra/modules/service/workflow_orchestrator_role.tf` The rest of the diff is around easy of configuration: - `infra/modules/service/scheduled_jobs.tf` represents the new, highly configurable, way to create scheduled jobs - `infra/api/app-config/env-config/scheduled_jobs.tf` shows how two scheduled jobs are configured ## Additional information <img width="1238" alt="image" src="https://github.com/user-attachments/assets/24e6e50e-74b0-4cf8-b61b-bad34af90118"> <img width="1237" alt="image" src="https://github.com/user-attachments/assets/aa297341-8d7d-4ac7-bcc9-ef102df25e05">
coilysiren
added a commit
that referenced
this pull request
Oct 18, 2024
## Ticket Resolves #744 ## Changes - Changes the ETL job from invoking ECS directly, to instead invoking Step Functions which then invokes ECS - Makes some changes to the scheduled changes to increase their distinction from the events jobs, and the readability of their IAM roles ## Context for reviewers This is a similar PR to #745. In-fact they use many of the same resources. To goal of this PR is to run ETL events via Step Functions. Step Functions create a tracking layer that's not available when invoking ECS directly. This tracking layer allows you to see the success and failure status of your jobs. ## Testing navapbc/platform-test#136
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Ticket
Resolves #525
Changes
Testing
Tested via navapbc/platform-test#130, and triggering a step functions job manually.