Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add local env runtime variables override for local testing #698

Merged
merged 13 commits into from
Mar 6, 2024

Conversation

yuli-han
Copy link
Collaborator

@yuli-han yuli-han commented Feb 28, 2024

Summary

Currently Chronon set the runtime environment variables from several sources based on the below priority order:
- Environment variables existing already.
- Environment variables derived from args (like app_name)
- conf.metaData.modeToEnvMap for the mode (set on config)
- team environment per context and mode set on teams.json
- default team environment per context and mode set on teams.json
- Common Environment set in teams.json

The team environment per context mode set is a team level environment setting which defined in the json file teams.json. Team could put the environment settings(like EMR clusters and queue) based on their need, and these settings will be applied to all the jobs within the team folder.

However, sometimes our users may want to test their new job locally, the local tests may use different environment settings comparing with the prod run on airflow. Currently they need to add their job config into a different folder(like the test folder) other than the production folder(which will be parsed as context), and add context specific environment variables, but this may bring new complexity to the testing process since they still have to copy their changes to the production config to make it work in production environment.

This PR is trying to solve this problem by introducing a new arg env whose default value is dev. Users can define runtime environment variables for different environment in teams.json file, for example, for production environment and dev environment:

"production": {
    "backfill": {
        "EMR_CLUSTER": "backfill-shared-prod",
        "EMR_QUEUE": "backfill"
    },
    "upload": {
        "EMR_CLUSTER": "backfill-shared-prod",
        "EMR_QUEUE": "backfill"
    }
},
"dev": {
    "backfill": {
        "EMR_CLUSTER": "homes-shared-prod-ap",
        "EMR_QUEUE": "staging"
    }
},
  1. For production run on airflow, the job will be scheduled with --env=production, so the production environment will be used.
  2. For local testing, the default env is dev, so if the dev environment is provided in teams.json file, dev environment will be used, otherwise production environment will be used.

Why / Goal

Test Plan

  • Added Unit Tests
  • Covered by existing CI
  • Integration tested
  1. dev environment is provided in teams.json, env=production
    Local tests with
    python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=production
image
  1. dev environment is provided in teams.json, env=dev
    Local tests with
    python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=dev
image
  1. dev environment is provided in teams.json, env is not used
    Local tests with
    python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20
image
  1. dev environment is not provided in teams.json, env = dev
    Local tests with
    python3 scripts/run.py --mode=backfill --conf=production/joins/zipline_test/test_online_join_small.v2 --ds=2024-02-20 --env=dev
image

Checklist

  • Documentation update

Reviewers

@better365

@yuli-han yuli-han changed the title [Test]Add env variable in run.py for testing Add local env runtime variables override for local testing Feb 28, 2024
Copy link
Collaborator

@pengyu-hou pengyu-hou left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alt Text

Copy link
Contributor

@nikhilsimha nikhilsimha left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor comments. Congrats on your first chronon PR Yuli!

yuli-han and others added 4 commits March 6, 2024 09:45
Co-authored-by: Nikhil <[email protected]>
Signed-off-by: hanyuli1995 <[email protected]>
Co-authored-by: Nikhil <[email protected]>
Signed-off-by: hanyuli1995 <[email protected]>
Co-authored-by: Nikhil <[email protected]>
Signed-off-by: hanyuli1995 <[email protected]>
Co-authored-by: Nikhil <[email protected]>
Signed-off-by: hanyuli1995 <[email protected]>
@yuli-han yuli-han merged commit f1d7c39 into main Mar 6, 2024
2 of 6 checks passed
@yuli-han yuli-han deleted the ylh--dev-env-override-test branch March 6, 2024 17:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants