-
Notifications
You must be signed in to change notification settings - Fork 19
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
* fix `isort` hook and the imports orders * fix `sqlfluff` configuration * update `airflow.cfg` to mirror dev composer * update `sentry-sdk` to overcome warnings * add script to test all dags integrity * add script to upload dags and schema files to gcs * add requirements to run airflow from ci * add ci workflow * hardcode `dags/` since it won't change * change script name * add ci/cd to test and update files in airflow * fix `python-version` on `pre-commit` step * fix `isort` and `black` compatibility * fix imports hopefully for the last time * autofix from `pre-commit` * remove unnecessary file * rename `airflow.cfg` to specify the environment * add airflow configuration file from prod composer * rename file * add logger and env selection * update ci/cd files * fix formatting * update ci/cd workflow * fix ci/cd * rearrange dependencies * modify authentication method to gcp * trigger ci only when pr comes from official repo * lint merged files * add new variables * pause reset testnet dag when on prod composer * fix uploading `airflow.cfg` to gcs
- Loading branch information
1 parent
9e8257d
commit 3e34347
Showing
25 changed files
with
777 additions
and
261 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,113 @@ | ||
| name: CI | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: | ||
| - opened | ||
| - reopened | ||
| - synchronize | ||
| - closed | ||
| branches: | ||
| - master | ||
|
|
||
| jobs: | ||
| pre-commit: | ||
| runs-on: ubuntu-latest | ||
| if: >- | ||
| github.event.pull_request.merged == false && | ||
| github.event.pull_request.state == 'open' | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.8 | ||
|
|
||
| - uses: pre-commit/[email protected] | ||
|
|
||
| tests: | ||
| runs-on: ubuntu-latest | ||
| needs: [pre-commit] | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.8 | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| cat airflow_variables_dev.json | sed -e s/\\/home\\/airflow\\/gcs\\/dags\\/// > airflow_variables_ci.json | ||
| python -m pip install --upgrade pip | ||
| pip install -r requirements-ci.txt | ||
| - name: Init Airflow SQLite database | ||
| run: airflow db init | ||
|
|
||
| - name: Import Airflow variables | ||
| run: airflow variables import airflow_variables_ci.json | ||
|
|
||
| - name: Pytest | ||
| run: pytest dags/ | ||
|
|
||
| deploy-to-dev: | ||
| runs-on: ubuntu-latest | ||
| needs: [tests] | ||
| # deploy to dev occurs every time | ||
| # someone submits a pr targeting `master` | ||
| # from a branch at `stellar/stellar-etl-airflow` repo | ||
| if: github.repository == 'stellar/stellar-etl-airflow' | ||
| # known caveats: | ||
| # if there's more than 1 person working | ||
| # in the same file this won't behave nicely | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.8 | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| pip install --upgrade pip | ||
| pip install google-cloud-storage==2.1.0 | ||
| - name: Authenticate to test-hubble GCP | ||
| uses: google-github-actions/auth@v1 | ||
| with: | ||
| credentials_json: "${{ secrets.CREDS_TEST_HUBBLE }}" | ||
|
|
||
| - name: Upload files to dev GCS bucket | ||
| run: python dags/stellar_etl_airflow/add_files_to_composer.py --bucket $BUCKET | ||
| env: | ||
| GOOGLE_CLOUD_PROJECT: test-hubble-319619 | ||
| BUCKET: us-central1-hubble-1pt5-dev-7db0e004-bucket | ||
|
|
||
| promote-to-prod: | ||
| runs-on: ubuntu-latest | ||
| # deploy only occurs when pr is merged | ||
| if: github.event.pull_request.merged == true | ||
| permissions: | ||
| pull-requests: write | ||
|
|
||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Create pull request | ||
| run: | | ||
| gh pr create \ | ||
| --base release \ | ||
| --head master \ | ||
| --reviewer stellar/platform-committers | ||
| --title "[PRODUCTION] Update production Airflow environment" \ | ||
| --body "This PR was auto-generated by GitHub Actions. | ||
| After merged and closed, this PR will trigger an action that updates DAGs, libs and schemas files from prod Airflow." | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,67 @@ | ||
| name: release | ||
|
|
||
| on: | ||
| pull_request: | ||
| types: | ||
| - closed | ||
| branches: | ||
| - release | ||
|
|
||
| jobs: | ||
| tests: | ||
| runs-on: ubuntu-latest | ||
| if: github.event.pull_request.merged == true | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.8 | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| cat airflow_variables_dev.json | sed -e s/\\/home\\/airflow\\/gcs\\/dags\\/// > airflow_variables_ci.json | ||
| python -m pip install --upgrade pip | ||
| pip install -r requirements-ci.txt | ||
| - name: Init Airflow SQLite database | ||
| run: airflow db init | ||
|
|
||
| - name: Import Airflow variables | ||
| run: airflow variables import airflow_variables_ci.json | ||
|
|
||
| - name: Pytest | ||
| run: pytest dags/ | ||
|
|
||
| release: | ||
| runs-on: ubuntu-latest | ||
| needs: [tests] | ||
| # deploy only occurs when pr is merged | ||
| if: >- | ||
| github.event.pull_request.merged == true && | ||
| github.repository == 'stellar/stellar-etl-airflow' | ||
| steps: | ||
| - uses: actions/checkout@v3 | ||
|
|
||
| - name: Setup Python | ||
| uses: actions/setup-python@v4 | ||
| with: | ||
| python-version: 3.8 | ||
|
|
||
| - name: Install dependencies | ||
| run: | | ||
| python -m pip install --upgrade pip | ||
| pip install google-cloud-storage==2.1.0 | ||
| - name: Authenticate to hubble GCP | ||
| uses: google-github-actions/auth@v1 | ||
| with: | ||
| credentials_json: "${{ secrets.CREDS_PROD_HUBBLE }}" | ||
|
|
||
| - name: Upload files to prod GCS bucket | ||
| run: python dags/stellar_etl_airflow/add_files_to_composer.py --bucket $BUCKET --env prod | ||
| env: | ||
| GOOGLE_CLOUD_PROJECT: hubble-261722 | ||
| BUCKET: us-central1-hubble-2-d948d67b-bucket |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.