Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

packaging: Integrate github workflows to automate the air-gapping process of wave Tour #2199

Merged
merged 46 commits into from
Feb 12, 2024
Merged
Show file tree
Hide file tree
Changes from 37 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
f94114d
packaging: Initial workflow files
sulhicader Oct 29, 2023
33c3f71
packaging: changed the workflow tigger on #2163
sulhicader Oct 29, 2023
f0c6a4f
packaging: changed the workflow file names on #2163
sulhicader Oct 29, 2023
3a00254
packaging: Added to set version in app.toml on #2163
sulhicader Nov 16, 2023
5b64c3f
packaging: Added workflows from workflow-library on #2163
sulhicader Nov 16, 2023
c817498
packaging: Changed workflow trigger mode to push on #2163
sulhicader Nov 16, 2023
cd63621
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
2c6f361
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
bc9991d
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
5ee8dbf
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
e6b32d2
packaging: Added workflow directory to each step on #2163
sulhicader Nov 16, 2023
fd3610a
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
acad043
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
51a91b6
packaging: Added workflow directory to each step on #2163
sulhicader Nov 17, 2023
cdc11bc
Attached new flows to publish-university
sulhicader Nov 20, 2023
66dbde3
packaging: Attached new flows to publish-university on #2163
sulhicader Nov 20, 2023
ca86f56
packaging: Reverted some changes on #2163
sulhicader Nov 20, 2023
f1407f3
Merge remote-tracking branch 'origin/main' into feat/issue-2163
sulhicader Nov 20, 2023
dc9a1c2
packaging: Reverted some changes on #2163
sulhicader Nov 20, 2023
4ad597c
packaging: Removed generated files on #2163
sulhicader Nov 20, 2023
213e6b2
packaging: Added initial workflow files on #2180
sulhicader Nov 20, 2023
08bdc19
packaging: Added initial workflow files on #2180
sulhicader Nov 20, 2023
c7552c7
packaging: Added initial workflow files on #2180
sulhicader Nov 20, 2023
2748d8c
packaging: Changed the version on #2180
sulhicader Nov 20, 2023
31b6fa8
Changed the version of tour
sulhicader Jan 11, 2024
293a3ef
Changed the base image on #2180
sulhicader Jan 16, 2024
a2a07a6
Merge remote-tracking branch 'origin/main' into feat/issue-2163
sulhicader Jan 31, 2024
b43ef51
Handled review comments #2163
sulhicader Jan 31, 2024
72a07bd
Handled review comments #2163
sulhicader Jan 31, 2024
41bf0f7
Handled review comments #2163
sulhicader Jan 31, 2024
1ad2d02
Finalized workflow for university #2163
sulhicader Jan 31, 2024
9b5f402
Merge remote-tracking branch 'origin/main' into feat/issue-2180
sulhicader Jan 31, 2024
a112848
Merge remote-tracking branch 'origin/feat/issue-2163' into feat/issue…
sulhicader Jan 31, 2024
988b531
Combined tour image publish to one script #2180
sulhicader Jan 31, 2024
bb51d36
Added final workflow for tour app #2180
sulhicader Jan 31, 2024
d2a5e04
Removed university related changes #2180
sulhicader Jan 31, 2024
cf66838
Merge branch 'main' into feat/issue-2180
sulhicader Feb 1, 2024
ba5944b
Merge branch 'main' into feat/issue-2180
sulhicader Feb 9, 2024
6f71abb
Handled review comments
sulhicader Feb 12, 2024
b107aee
To trigger refactored workflow
sulhicader Feb 12, 2024
2d34f67
To trigger refactored workflow
sulhicader Feb 12, 2024
158e1b6
To trigger refactored workflow
sulhicader Feb 12, 2024
9080025
To trigger refactored workflow
sulhicader Feb 12, 2024
e86e170
Refactored github action for tour app
sulhicader Feb 12, 2024
b97ee49
Added an step to remove tmp folder
sulhicader Feb 12, 2024
cbdfa2f
Added an step to remove tmp folder at beginning of build-apps
sulhicader Feb 12, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish-university.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,7 +22,7 @@ jobs:

- name: Build university
run: make publish-university

- name: Publish to PyPI
uses: pypa/gh-action-pypi-publish@master
with:
Expand Down
77 changes: 76 additions & 1 deletion .github/workflows/release-wave.yml
Original file line number Diff line number Diff line change
Expand Up @@ -135,7 +135,7 @@ jobs:

- name: Build university
run: make publish-university

- name: Publish to PyPI - wave_university
uses: pypa/gh-action-pypi-publish@master
with:
Expand All @@ -153,3 +153,78 @@ jobs:
with:
packages_dir: py/h2o_lightwave_web/dist
password: ${{ secrets.PYPI_LIGTHWAVE_WEB_TOKEN }}

create-bundle:
name: Create Wave Bundle
runs-on: ubuntu-latest

outputs:
build-version: ${{ env.VERSION }}

steps:
- uses: actions/checkout@v3

- name: Download H2O CLI
working-directory: ./py/apps/tour
run: mkdir -p .bin && curl -o .bin/h2o https://h2oai-cloud-release.s3.amazonaws.com/releases/ai/h2o/h2o-cloud/latest/cli/linux-amd64/h2o

- name: Change permissions
working-directory: ./py/apps/tour
run: chmod +x .bin/h2o

- name: Set version
working-directory: ./py/apps/tour
run: sed -i -r -e "s/\{\{VERSION\}\}/${{ env.VERSION }}/g" app.toml

- name: Get App Version
id: get-build-version
run: |
echo "VERSION=${{ env.VERSION }}" >> "$GITHUB_OUTPUT"

- name: Make the folder structure
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This introduces unwanted code duplication as this code can also be found in make build-apps target. Please refactor so that we have a single make build-tour-app target that can be reused here and also in make build-apps.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good Suggestion. Thanks @mturoci

run: |
find py/apps/tour -type f -name '*.toml' -exec sed -i -e "s/{{VERSION}}/${{ env.VERSION }}/g" {} \;
find py/apps/tour -type f -name 'requirements.txt' -exec sed -i -e "s/{{VERSION}}/${{ env.VERSION }}/g" {} \;
rsync -a py/examples py/apps/tour --exclude "*.idea*" --exclude "*__pycache__*" --exclude "*.mypy_cache*" --exclude "dist" --exclude "build"
cp tools/vscode-extension/base-snippets.json py/apps/tour/examples
cp tools/vscode-extension/component-snippets.json py/apps/tour/examples
cp tools/vscode-extension/server/utils.py py/apps/tour/examples/tour_autocomplete_utils.py
cp tools/vscode-extension/server/parser.py py/apps/tour/examples/tour_autocomplete_parser.py
sed -i -r -e "s#^@app\(('|\")(.*)('|\")(.*)#@app\('/'\4#" py/apps/tour/examples/tour.py

- name: Make air-gapped bundle
working-directory: ./py/apps/tour
run: |
.bin/h2o bundle \
--docker-base-image 524466471676.dkr.ecr.us-east-1.amazonaws.com/q8s/launcher:v0.31.0-310 \
--docker-use-buildkit \
--generate-helm-charts \
--helm-chart-version ${{ env.VERSION }} \
--helm-chart-name tour \
--helm-app-bundle-image-repo 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/tour-bundle \
--helm-app-runtime-image-repo 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/tour \
--generate-dockerfile

- uses: actions/upload-artifact@v3
with:
name: wave-bundle
path: |
./py/apps/tour/*.Dockerfile
./py/apps/tour/*.wave

- uses: actions/upload-artifact@v3
with:
name: wave-bundle-helm
path: py/apps/tour/helm/

build-and-publish:
needs: create-bundle
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout

uses: ./.github/workflows/wave-bundle-docker-build-publish.yaml
with:
build-version: ${{ needs.create-bundle.outputs.build-version }}
bundle-artifact: wave-bundle
wave-app-name: tour
114 changes: 114 additions & 0 deletions .github/workflows/wave-bundle-docker-build-publish.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,114 @@
name: Build and Publish Docker image from generated Dockerfile

on:
workflow_call:
inputs:
build-version:
type: string
description: The version of the application/image to be pushed
required: true
bundle-artifact:
type: string
description: The name of the artifact containing the generated Dockerfiles and the wave bundle
required: true
working-directory:
type: string
description: Path to the working directory, where docker build will be executed
default: .
wave-app-name:
type: string
description: The name of the wave app
required: true

jobs:
docker:
name: Build and Publish App Docker Image
runs-on: ubuntu-latest
permissions:
id-token: write # This is required for requesting the JWT
contents: read # This is required for actions/checkout

steps:
- uses: actions/checkout@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v2

- name: Download Wave bundle and Dockerfiles
uses: actions/download-artifact@v3
with:
name: ${{ inputs.bundle-artifact }}
path: ./

- name: Rename Dockerfiles
run: |
mv ./*bundle.Dockerfile ./generated.bundle.Dockerfile
mv ./*runtime.Dockerfile ./generated.runtime.Dockerfile

- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v2
with:
role-to-assume: arn:aws:iam::524466471676:role/workflows-library-wave-bundling-github-actions
role-session-name: GitHub_to_AWS_via_FederatedOIDC
aws-region: us-east-1

- name: "Login to Amazon ECR"
uses: aws-actions/amazon-ecr-login@v1

- name: "Ensure ECR Repository for the bundle"
# Tries to describe the repository and if it fails, creates it
run: |
aws ecr describe-repositories \
--repository-names h2oai/${{ inputs.wave-app-name }}-bundle 2>/dev/null ||
aws ecr create-repository \
--repository-name h2oai/${{ inputs.wave-app-name }}-bundle \
--image-tag-mutability IMMUTABLE \
--tag \
Key=GithubRepo,Value=github.com/h2oai/${{ inputs.wave-app-name }} \
Key=ManagedBy,Value=GitHubActions \
Key=CreatedByWorkflow,Value=${{ github.workflow_ref }}

- name: "Ensure ECR Repository for the runtime"
# Tries to describe the repository and if it fails, creates it
run: |
aws ecr describe-repositories \
--repository-names h2oai/${{ inputs.wave-app-name }} 2>/dev/null ||
aws ecr create-repository \
--repository-name h2oai/${{ inputs.wave-app-name }} \
--image-tag-mutability IMMUTABLE \
--tag \
Key=GithubRepo,Value=github.com/h2oai/${{ inputs.wave-app-name }} \
Key=ManagedBy,Value=GitHubActions \
Key=CreatedByWorkflow,Value=${{ github.workflow_ref }}

- name: Build and Export Bundle Image
uses: docker/build-push-action@v4
id: bundle-build
with:
push: true
context: ${{ inputs.working-directory }}
file: ./generated.bundle.Dockerfile
platforms: linux/amd64
provenance: false
tags: |
524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}-bundle:${{ inputs.build-version }}

- name: Build and Export Runtime Image
uses: docker/build-push-action@v4
id: runtime-build
with:
push: true
context: ${{ inputs.working-directory }}
file: ./generated.runtime.Dockerfile
platforms: linux/amd64
provenance: false
tags: |
524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}:${{ inputs.build-version }}

- name: Published Images Summary
run: |
echo "#### Image Tags" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
echo "docker image push 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}-bundle:${{ inputs.build-version }}" >> $GITHUB_STEP_SUMMARY
echo "docker image push 524466471676.dkr.ecr.us-east-1.amazonaws.com/h2oai/${{ inputs.wave-app-name }}:${{ inputs.build-version }}" >> $GITHUB_STEP_SUMMARY
echo "\`\`\`" >> $GITHUB_STEP_SUMMARY
Loading