feat: Reusable Workflows (Python and Rust) [TSD-274] (#15)

* feat(python)!: Replace invocation of flake8 with delegation to taskipy

* fix(python): Formatting of run checks action

* fix(python): Add required `shell` param to run checks

* fix: Make bandit use dynamic module name. Remove cov export

* feat: add working-directory input to install-and-configure-poetry

* refactor(deny-check): convert to composite action for ssh support

* fix(deny-check): add debug finding of deny.toml

* fix(deny-check): search starting from the root

* fix(deny-check): use GITHUB_ACTION_PATH for path to deny.toml

* fix(deny-check): fix path to deny.toml

* fix(deny-check): print env for debugging

* fix(deny-check): use github context object

* fix(deny-check): allow Triax crates via clarification whitelist

* fix(deny-check): license_files -> license-files

* fix(deny-check): fake internal crates as MIT-licensed to make checks pass

* feat: create new Rust actions now that nested composite actions are allowed

* fix: rename a couple actions

* fix(rust/setup): default toolchain to stable

* refactor: remove unused/outdated action

* refactor: refer to v2, not the feature branch

* feat(rust): add a run-checks-and-tests action

The new action does not include `deny-check` as there is a high chance
of that check failing and it makes sense to keep that failure separate
from the "more important" checks.

* feat(rust): add support for working directory input

* refactor: refer to feature branch again for more testing

* fix(rust): fix manifest-path

* fix(rust): fix manifest-path again

* fix(rust): fix manifest-path again

* fix(rust): fix manifest-path again

* fix(rust): manually call cargo test-all-features

* doc: add comment indicating use of run instead of uses

* fix(rust): use v2, not feature branch

* refactor: rust/release -> rust/build-and-upload

* fix: use a rust-specific caching action to prevent issues with cache

* Created separate per-language subdirs for actions vs jobs.

* Fixed versioning

* Fix formatting error (maybe)

* Try again to fix template

* Change default working_directory param to hopefully fix JSON issue

* Added os name to artifact upload to prevent duplicate naming issues

* Fixed binary naming

* Added underscore between binary name and runner os

* Added minimum maxOS build target

* Fix format

* chore: ignore unapplicable safety warning

* chore: ignore another unapplicable safety warning

* Reverted action vs job refactor (see ticket for details)

* Revert "Reverted action vs job refactor (see ticket for details)"

This reverts commit d9826ce.

* First attempt at a reusable workflow

* Moved workflow file to proper location

* Add types for all inputs

* Changed some inputs to be secrets

* Added custom pre checks step for more reusability

* Changed default value

* Added missing input

* Fixed typo

* Removed usage of poetry tasks in favor of generalized linting

* Add additional-modules-to-lint input to workflow

* Fixed indentation

* Removed potentially unneeded quotes

* Updated some refs

* Bandit and pylint now use the local config for the project instead of predefined rules

* Added disable list for linting tests

* Made aws-default-region an input

* Some WIP changes

* Fixed workflow file name

* Fixed format errors

* Fixed publish workflow

* WIP

* Fixed inputs

* Fixed file name

* Fixed pathing

* Fixed workflow parameter

* Added missing shell values

* Fix parameter passing

* Fix parameter passing some more

* Fixed syntax

* Fix angular publish

* Added first reusable rust workflow

* Added step to use SSH key

* Updated refs

* Added build workflow, fixed typo

* Renamed workflow

* Fix default value for param

* Test

* Fixed edge case

* Removed fromJSON stuff

* Reverted changes

* Test

* Fix typo

* Fixed logic check (hopefully)

* Moved ubuntu dependency installation to Rust setup action

* Fixed ubuntu dependency installation

* Added shell

* Test

* Potential fix

* Applied fix from main

* Pin angular generator version

Co-authored-by: Dylan Anthony <[email protected]>
Co-authored-by: Michael Bryant <[email protected]>
Co-authored-by: Ethan Mann <[email protected]>

.github/workflows/python-deploy-serverless.yml

@@ -0,0 +1,69 @@
+name: Deploy
+
+on: 
+  workflow_call:
+    inputs:
+      environment:
+        required: true
+        type: string
+      cli-name:
+        required: true
+        type: string
+      dependency-hash-version:
+        required: false
+        type: string
+        default: "v1"
+      private-package-repo-url:
+        description: The url of the private package repository you want to add to poetry. You must also specify `private-package-repo-username`
+        type: string
+        required: false
+        default: ""
+      aws-default-region:
+        type: string
+        required: false
+        default: "us-east-1"
+    secrets:
+      private-package-repo-username:
+        required: false
+      private-package-repo-password:
+        required: false
+      AWS_ACCESS_KEY_ID:
+        required: true
+      AWS_SECRET_ACCESS_KEY:
+        required: true
+
+jobs:
+  build-and-deploy:
+    name: Deploy to ${{ inputs.environment }}
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Cache dependencies (Node)
+        uses: actions/cache@v2
+        with:
+          path: ./node_modules
+          key: npm-${{ hashFiles('package-lock.json') }}
+
+      - name: Setup Python and Poetry and Install Dependencies
+        uses: triaxtec/github-actions/python/setup-python-and-poetry-and-install-dependencies@TSD-274-reusable-workflows
+        with:
+          private-package-repo-url: ${{ inputs.private-package-repo-url }}
+          private-package-repo-username: ${{ secrets.private-package-repo-username }}
+
+      - name: Build Code
+        run: poetry run ${{ inputs.cli-name }} build
+
+      - name: Generate Requirements
+        run: poetry export -f requirements.txt -o requirements.txt --with-credentials --without-hashes
+
+      - name: Install Serverless and Plugins
+        run: npm install
+
+      - name: Serverless Deploy to ${{ inputs.environment }}
+        run: $(npm bin)/serverless deploy --stage ${{ inputs.environment }}
+        env:
+          AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
+          AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
+          AWS_DEFAULT_REGION: ${{ inputs.aws-default-region }}

.github/workflows/python-generate-and-publish-client-libraries.yml

@@ -0,0 +1,92 @@
+name: Build and Publish Client Libraries
+
+on: 
+  workflow_call:
+    inputs:
+      dependency-hash-version:
+        required: false
+        type: string
+        default: "v1"
+      private-package-repo-url:
+        description: The url of the private package repository you want to add to poetry. You must also specify `private-package-repo-username`
+        type: string
+        required: false
+        default: ""
+      extras:
+        description: Extras to include when installing this package
+        type: string
+        required: false
+        default: NOT_SPECIFIED
+      expected-python-client-package-name:
+        description: What you expect the name of the python client package to be
+        type: string
+        required: true
+      cli-name:
+        description: The name of the CLI for this project
+        type: string
+        required: true
+    secrets:
+      private-package-repo-username:
+        required: false
+      private-package-repo-password:
+        required: false
+      private-package-repo-push-token:
+        required: true
+
+jobs:
+  build-python-client:
+    name: Python Client Library
+    runs-on: ubuntu-latest
+    steps:
+      - name: Generate OpenAPI Document
+        uses: triaxtec/github-actions/python/generate-openapi-doc@TSD-274-reusable-workflows
+        with:
+          private-package-repo-url: ${{ inputs.private-package-repo-url }}
+          private-package-repo-username: ${{ secrets.private-package-repo-username }}
+          cli-name: ${{ inputs.cli-name }}
+
+      - name: Install pipx
+        run: python -m pip install --user pipx
+
+      - name: Generate Python Client
+        run: python -m pipx run openapi-python-client generate --config python-generator-config.yml --path openapi.json
+
+      - name: Publish Python Client
+        uses: triaxtec/github-actions/python/publish-package@TSD-274-reusable-workflows
+        with:
+          token: ${{ secrets.private-package-repo-push-token }}
+          private-package-repo-url: ${{ inputs.private-package-repo-url }}
+          dir: ./${{ inputs.expected-python-client-package-name }}
+
+  build-and-publish-angular-client:
+    name: Angular Client Library
+    runs-on: ubuntu-latest
+    steps:
+    - name: Generate OpenAPI Document
+      uses: triaxtec/github-actions/python/generate-openapi-doc@TSD-274-reusable-workflows
+      with:
+          private-package-repo-url: ${{ inputs.private-package-repo-url }}
+          private-package-repo-username: ${{ secrets.private-package-repo-username }}
+          cli-name: ${{ inputs.cli-name }}
+
+    - name: Set up Node
+      uses: actions/[email protected]
+      with:
+        node-version: '14'
+
+    - name: Generate Angular Client Library
+      uses: openapi-generators/[email protected]
+      with:
+        generator: typescript-angular
+        config-file: angular-generator-config.yml
+        generator-tag: v5.3.0
+
+    - name: Publish Angular Client Library
+      run: |
+        cd typescript-angular-client
+        echo "//npm.fury.io/triaxtec/:_authToken=${{ secrets.private-package-repo-push-token }}" > .npmrc
+        npm install
+        npm remove @types/node
+        npm add -D @types/[email protected]
+        npm run build
+        npm publish dist

.github/workflows/python-publish-package.yml

@@ -0,0 +1,57 @@
+name: Publish Python Package
+
+on:
+  workflow_call:
+    inputs:
+      dependency-hash-version:
+        required: false
+        type: string
+        default: "v1"
+      private-package-repo-url:
+        description: The url of the private package repository you want to add to poetry. You must also specify `private-package-repo-username`
+        required: false
+        type: string
+        default: ""
+      extras:
+        description: Extras to include when installing this package
+        type: string
+        required: false
+        default: NOT_SPECIFIED
+      working-directory:
+        description: The working directory where this action should run. Defaults to the root of the git repository.
+        type: string
+        required: false
+        default: "."
+    secrets:
+      private-package-repo-username:
+        required: false
+      private-package-repo-push-token:
+        required: false
+
+jobs:
+  publish:
+    name: Publish Package
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+
+    - name: Cache dependencies
+      uses: actions/cache@v2
+      with:
+        path: ./.venv
+        key: poetry-${{ inputs.dependency-hash-version }}-${{ hashFiles('poetry.lock') }}
+
+    - name: Setup Python and Poetry and Install Dependencies
+      uses: triaxtec/github-actions/python/setup-python-and-poetry-and-install-dependencies@TSD-274-reusable-workflows
+      with:
+        private-package-repo-url: ${{ inputs.private-package-repo-url }}
+        private-package-repo-username: ${{ secrets.private-package-repo-username }}
+        exclude-dev-dependencies: true
+        extras: ${{ inputs.extras }}
+
+    - name: Build and Publish
+      uses: triaxtec/github-actions/python/publish-package@TSD-274-reusable-workflows
+      with:
+        private-package-repo-url: ${{ inputs.private-package-repo-url }}
+        token: ${{ secrets.private-package-repo-push-token }}
+        dir: ${{ inputs.working-directory }}

.github/workflows/python-run-checks-and-tests.yml

@@ -0,0 +1,94 @@
+name: Run Styling/Typing/Lint Checks and Tests
+
+on: 
+  workflow_call:
+    inputs:
+      start-services-with-docker-compose:
+        required: false
+        type: boolean
+        default: false
+      custom-pre-checks-script:
+        required: false
+        type: string
+        default: ""
+      module:
+        required: true
+        type: string
+      additional-modules-to-lint:
+        description: Additional modules to be linted using the standard ruleset (i.e. not the ones used for linting unit tests).
+        required: false
+        type: string
+        default: ""
+      dependency-hash-version:
+        required: false
+        type: string
+        default: "v1"
+      private-package-repo-url:
+        description: The url of the private package repository you want to add to poetry. You must also specify `private-package-repo-username`
+        type: string
+        required: false
+        default: ""
+      exclude-dev-dependencies:
+        description: If set to true, poetry install will be called with the --no-dev
+        type: string
+        required: false
+        default: "false"
+      extras:
+        description: Extras to include when installing this package
+        type: string
+        required: false
+        default: NOT_SPECIFIED
+      working-directory:
+        description: The working directory where this action should run. Defaults to the root of the git repository.
+        type: string
+        required: false
+        default: "."
+    secrets:
+      private-package-repo-username:
+        required: false
+      private-package-repo-password:
+        required: false
+
+jobs:
+  checks:
+    name: Run Styling/Typing/Lint Checks and Tests
+    runs-on: ubuntu-latest
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v2
+
+      - name: Start Services
+        if: inputs.start-services-with-docker-compose
+        run: docker-compose up -d
+
+      - name: Cache dependencies
+        uses: actions/cache@v2
+        with:
+          path: ./.venv
+          key: poetry-${{ inputs.dependency-hash-version }}-${{ hashFiles('poetry.lock') }}
+
+      - name: Setup python and poetry and install dependencies
+        uses: triaxtec/github-actions/python/setup-python-and-poetry-and-install-dependencies@TSD-274-reusable-workflows
+        with:
+          private-package-repo-url: ${{ inputs.private-package-repo-url }}
+          private-package-repo-username: ${{ secrets.private-package-repo-username }}
+          private-package-repo-password: ${{ secrets.private-package-repo-password }}
+          exclude-dev-dependencies: ${{ inputs.exclude-dev-dependencies }}
+          extras: ${{ inputs.extras }}
+          working-directory: ${{ inputs.working-directory }}
+
+      - name: Custom pre-checks script
+        if: inputs.custom-pre-checks-script
+        run: ${{ inputs.custom-pre-checks-script }}
+
+      - name: Run Checks and Tests
+        uses: triaxtec/github-actions/python/run-checks@TSD-274-reusable-workflows
+        with:
+          module: ${{ inputs.module }}
+          additional-modules-to-lint: ${{ inputs.additional-modules-to-lint }}
+        env:
+          AWS_DEFAULT_REGION: us-east-1
+
+      - name: Uninstall package before caching
+        run: poetry run pip uninstall ${{ inputs.module }} -y

.github/workflows/rust-build-and-upload-binaries.yml

@@ -0,0 +1,33 @@
+name: Build Binaries and Upload them as Artifacts
+
+on:
+  workflow_call:
+    secrets:
+      ssh-private-key:
+        description: "SSH key used to pull private dependencies from GitHub"
+        required: true
+
+jobs:
+    build-and-upload:
+      strategy:
+        matrix:
+          os: [ windows-latest, ubuntu-latest, macos-latest ]
+      name: Build Binary for ${{ matrix.os }}
+      runs-on: ${{ matrix.os }}
+
+      steps:
+        - name: Checkout code
+          uses: actions/checkout@v2
+
+        - name: Add SSH key so cargo can access private repos
+          uses: webfactory/[email protected]
+          with:
+            ssh-private-key: ${{ secrets.ssh-private-key }}
+
+        - name: Setup Rust and Install Dependencies
+          uses: triaxtec/github-actions/rust/setup@TSD-274-reusable-workflows
+
+        - name: Build and Upload
+          uses: triaxtec/github-actions/rust/build-and-upload@TSD-274-reusable-workflows
+          with:
+            binary_name: eat