Skip to content

Commit

Permalink
[Issue #2351, #2472] Migrate to platforms step function pattern (#2506)
Browse files Browse the repository at this point in the history
## Summary

Relates to: #2351, #2472

### Time to review: __5 mins__

## Changes proposed

There are two goals to this PR:

1. Make a material change that gets us closer in line with the platform.
Specifically we are pulling in
navapbc/template-infra#745
2. Updates the step functions in advance of adding another step function
- for the terraform in ECS stuff

## Context for reviewers

The majority of the diff is security related, specifically: 

- via guidance from @lorenyu, the permissions for managing the step
function and workflow triggers were moved off of the task exec role.
that's the diff on `infra/modules/service/access-control.tf`
- those permissions were moved to
`infra/modules/service/scheduler_role.tf` and
`infra/modules/service/workflow_orchestrator_role.tf`

The rest of the diff is around easy of configuration:

- `infra/modules/service/scheduled_jobs.tf` represents the new, highly
configurable, way to create scheduled jobs
- `infra/api/app-config/env-config/scheduled_jobs.tf` shows how two
scheduled jobs are configured

## Additional information

<img width="1238" alt="image"
src="https://github.com/user-attachments/assets/24e6e50e-74b0-4cf8-b61b-bad34af90118">

<img width="1237" alt="image"
src="https://github.com/user-attachments/assets/aa297341-8d7d-4ac7-bcc9-ef102df25e05">
  • Loading branch information
coilysiren authored Oct 17, 2024
1 parent a2b220d commit 1277e32
Show file tree
Hide file tree
Showing 15 changed files with 329 additions and 310 deletions.
13 changes: 0 additions & 13 deletions infra/api/app-config/dev.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,19 +14,6 @@ module "dev_config" {
# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version
search_engine_version = "OpenSearch_2.15"

# Runs, but with everything disabled.
# See api/src/data_migration/command/load_transform.py for argument specifications.
load_transform_args = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--no-load",
"--no-transform",
"--no-set-current",
]

service_override_extra_environment_variables = {
}
}
1 change: 1 addition & 0 deletions infra/api/app-config/env-config/environment-variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ locals {
# This is a map rather than a list so that variables can be easily
# overridden per environment using terraform's `merge` function
default_extra_environment_variables = {
FLASK_APP = "src.app:create_app()"
# Example environment variables
# WORKER_THREADS_COUNT = 4
# LOG_LEVEL = "info"
Expand Down
8 changes: 4 additions & 4 deletions infra/api/app-config/env-config/outputs.tf
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,10 @@ output "service_config" {
}
}

output "scheduled_jobs" {
value = local.scheduled_jobs
}

output "incident_management_service_integration" {
value = var.has_incident_management_service ? {
integration_url_param_name = "/monitoring/${var.app_name}/${var.environment}/incident-management-integration-url"
Expand All @@ -46,7 +50,3 @@ output "incident_management_service_integration" {
output "domain" {
value = var.domain
}

output "load_transform_args" {
value = var.load_transform_args
}
54 changes: 54 additions & 0 deletions infra/api/app-config/env-config/scheduled_jobs.tf
Original file line number Diff line number Diff line change
@@ -0,0 +1,54 @@
locals {
# The `task_command` is what you want your scheduled job to run, for example: ["poetry", "run", "flask"].
# Schedule expression defines the frequency at which the job should run.
# The syntax for `schedule_expression` is explained in the following documentation:
# https://docs.aws.amazon.com/eventbridge/latest/userguide/eb-scheduled-rule-pattern.html
# The `state` is the state of the scheduled job. It can be either "ENABLED" or "DISABLED".

# See api/src/data_migration/command/load_transform.py for argument specifications.
load-transform-args = {
# Runs, but with everything disabled.
dev = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--no-load",
"--no-transform",
"--no-set-current",
],
staging = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--load",
"--transform",
"--set-current",
],
prod = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--load",
"--transform",
"--set-current",
],
}
scheduled_jobs = {
copy-oracle-data = {
task_command = ["poetry", "run", "flask", "data-migration", "copy-oracle-data"]
schedule_expression = "rate(2 minutes)"
state = "ENABLED"
}
load-transform = {
task_command = local.load-transform-args[var.environment]
schedule_expression = "rate(1 days)"
state = "ENABLED"
}
}
}
4 changes: 0 additions & 4 deletions infra/api/app-config/env-config/variables.tf
Original file line number Diff line number Diff line change
Expand Up @@ -85,7 +85,3 @@ variable "service_override_extra_environment_variables" {
EOT
default = {}
}

variable "load_transform_args" {
type = list(string)
}
12 changes: 0 additions & 12 deletions infra/api/app-config/prod.tf
Original file line number Diff line number Diff line change
Expand Up @@ -23,18 +23,6 @@ module "prod_config" {
# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version
search_engine_version = "OpenSearch_2.15"

# See api/src/data_migration/command/load_transform.py for argument specifications.
load_transform_args = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--load",
"--transform",
"--set-current",
]

service_override_extra_environment_variables = {
}
}
12 changes: 0 additions & 12 deletions infra/api/app-config/staging.tf
Original file line number Diff line number Diff line change
Expand Up @@ -14,18 +14,6 @@ module "staging_config" {
# https://docs.aws.amazon.com/opensearch-service/latest/developerguide/what-is.html#choosing-version
search_engine_version = "OpenSearch_2.15"

# See api/src/data_migration/command/load_transform.py for argument specifications.
load_transform_args = [
"poetry",
"run",
"flask",
"data-migration",
"load-transform",
"--load",
"--transform",
"--set-current",
]

service_override_extra_environment_variables = {
}
}
2 changes: 2 additions & 0 deletions infra/api/service/main.tf
Original file line number Diff line number Diff line change
Expand Up @@ -127,6 +127,8 @@ module "service" {
app_access_policy_arn = data.aws_iam_policy.app_db_access_policy[0].arn
migrator_access_policy_arn = data.aws_iam_policy.migrator_db_access_policy[0].arn

scheduled_jobs = local.environment_config.scheduled_jobs

db_vars = module.app_config.has_database ? {
security_group_ids = data.aws_rds_cluster.db_cluster[0].vpc_security_group_ids
connection_info = {
Expand Down
100 changes: 0 additions & 100 deletions infra/api/service/sfn_copy_oracle_data.tf

This file was deleted.

94 changes: 0 additions & 94 deletions infra/api/service/sfn_load_transform.tf

This file was deleted.

Loading

0 comments on commit 1277e32

Please sign in to comment.