Task A-1: Switching to GHCR + Task A-3: Image build/push on release a…

…nd not push Task A-3: Image build/push on release instead of on every push 1. Re-building on releases instead of tags - Earlier I had set the trigger event as tags but this would mean any time we create a new tag the workflow would be triggered. - Instead, we are actually creating releases and not tags through the UI. - Hence change the trigger event to releases, specifically “published” releases. ------ 2. Both Tests and Build will happen for push events - Tests: We need tests to confirm everything works before merging. - Build: Similarly, even though tests might pass, the image build could fail due to issues in Dockerfile or any files used in Dockerfile. - Hence to ensure that the final image that is pushed is correctly built, and we can have a correct version released, running both tests and build for push events. ------ 3. Dispatch only runs on releases identified by ref_type == tag. - Dispatch depends on build and since we now have build running for both push and release events, dispatch would also run for both. - Hence restricting dispatch to run only on release events by using the ref_type check which has the value “tag” for releases and “branch” for pushes on any branch. ------- 4. Additional version increment steps + Creating incremented version tags and releases in workflow - Manually creating new release through UI would trigger the workflows for all the repos and the new tag would be updated to the tag version set in the UI. This would be updated in the .env files as well. - But for the workflow_dispatch as well, we need to handle creating a new tag and release for the dependent cascading images in the dashboard repositories. - For this, I’m fetching the latest and 2nd latest server releases and checking for the type of SemVer release (major/minor/patch). Based on the type, similar version increment would occur for the dependent images. - Depending on the event type (release / workflow_dispatch), either the tag created through the UI or the incremented tag version would be chosen as the new tag, respectively for each of the two trigger event types. - This new tag is also written to the .env file. - Finally, a new release is created based on this newly created tag in the workflow. ---------------------- Task A-1: Switch to storing images in GHCR 1. Changes to Dockerfile - Needed to add LABEL layer to ensure that the docker image is connected to the correct repository. - For dashboard images that are dependent on the server image, changed to pull from GHCR. 2. Workflow file - Login action used to login to GHCR. - Building, Renaming, Pushing images with GHCR tag. - Appropriate version tag names used depending on whether trigger event was release or workflow_dispatch.
e-mission · Oct 8, 2024 · dcdd42f · dcdd42f
1 parent 7f390ce
commit dcdd42f
Show file tree

Hide file tree

Showing 3 changed files with 176 additions and 41 deletions.
diff --git a/.github/workflows/image_build_push.yml b/.github/workflows/image_build_push.yml
@@ -2,9 +2,9 @@ name: docker image
 
 on:
   push:
-    branches: [ master, gis-based-mode-detection ]
-    tags:
-      - "v[0-9]+.[0-9]+.[0-9]+"
+    branches: [ master, gis-based-mode-detection, cleanup-cicd ]
+  release:
+    types: [published]
 
 #Dockerhub credentials are set as environment variables
 env:
@@ -26,19 +26,15 @@ jobs:
       - name: Get branch name based on ref_type
         id: get-branch-name
         run: |
-          echo "Ref: ${{ github.ref }}"
-          echo "Ref name: ${{ github.ref_name }}"
-          echo "Ref type: ${{ github.ref_type }}"
-          echo "Github event base ref: ${{ github.event.base_ref }}"
           if [ "${{ github.ref_type }}" == "tag" ]; then
-            base_ref=${{ github.event.base_ref }}
-            echo "branch_name=${base_ref#refs/heads/}" >> "$GITHUB_OUTPUT"
+            target_branch=${{ github.event.release.target_commitish }}
+            echo "branch_name=${target_branch}" >> "$GITHUB_OUTPUT"
           elif [ "${{ github.ref_type }}" == "branch" ]; then
             echo "branch_name=${GITHUB_REF##*/}" >> "$GITHUB_OUTPUT"    
           fi
 
   build:
-    if: ${{ !contains(github.event.head_commit.author.name, 'Github Actions bot to update .env with latest tags') && github.ref_type == 'tag' }}
+    if: ${{ !contains(github.event.head_commit.author.name, 'Github Actions bot') }}
     needs: [test-with-docker, test-with-manual-install, get-branch-name]
     uses: e-mission/e-mission-server/.github/workflows/reusable_image_build_push.yml@master
     with:
@@ -47,7 +43,8 @@ jobs:
     secrets: inherit
 
   dispatch:
-    needs: build
+    if: ${{ github.ref_type == 'tag' }}
+    needs: [build]
     runs-on: ubuntu-latest
 
     strategy:

diff --git a/.github/workflows/reusable_image_build_push.yml b/.github/workflows/reusable_image_build_push.yml
@@ -24,33 +24,126 @@ jobs:
           repository: e-mission/${{ inputs.repo }}
           ref: ${{ inputs.branch }}
           token: ${{ secrets.GH_FG_PAT_TAGS }}
-
+          fetch-depth: 0
+
+      - name: Fetch latest repository release tag
+        id: fetch-latest-release-tags
+        run: |
+          response=$(curl -L -H "Accept: application/vnd.github+json" \
+              -H "Authorization: Bearer ${{ secrets.GH_FG_PAT_TAGS }}" \
+              -H "X-GitHub-Api-Version: 2022-11-28" \
+              "https://api.github.com/repos/e-mission/${{ inputs.repo }}/releases/latest")
+
+          tag_name=$(echo "$response" | jq -r '.tag_name')
+          target_commitish=$(echo "$response" | jq -r '.target_commitish')
+          
+          echo "tag_name=$(echo "$response" | jq -r '.tag_name')" >> "$GITHUB_OUTPUT"
+          echo "target_commitish=$(echo "$response" | jq -r '.target_commitish')" >> "$GITHUB_OUTPUT"
+
       - name: Fetch latest server image tag
+        id: get-server-tag
         if: ${{ inputs.repo == 'op-admin-dashboard' || inputs.repo == 'em-public-dashboard' }}
         run: |
-          response=$(curl -s https://raw.githubusercontent.com/e-mission/e-mission-server/refs/heads/master/.env)
-          SERVER_IMAGE_TAG=$(echo "$response" | grep "SERVER_IMAGE_TAG=" | cut -d'=' -f2)
-          echo "SERVER_IMAGE_TAG=$SERVER_IMAGE_TAG" >> "$GITHUB_OUTPUT"
+          response=$(curl -L -H "Accept: application/vnd.github+json" \
+            -H "Authorization: Bearer ${{ secrets.GH_FG_PAT_TAGS }}" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            "https://api.github.com/repos/e-mission/e-mission-server/releases")
+
+          tags=$(echo "$response" | jq -r '.[].tag_name')
+          current_server_release_tag=$(echo "$tags" | sort -V | tail -n 1)
+          previous_server_release_tag=$(echo "$tags" | sort -V | tail -n 2 | head -n 1)
+          
+          echo "SERVER_IMAGE_TAG=$current_server_release_tag" >> "$GITHUB_OUTPUT"
+          echo "PREVIOUS_SERVER_TAG=$previous_server_release_tag" >> "$GITHUB_OUTPUT"
+      
+          echo "Latest server release tag: $current_server_release_tag"
+          echo "Previous server release tag: $previous_server_release_tag"
+
+      - name: Bump up release tag
+        if: ${{ github.event_name == 'workflow_dispatch' }} 
+        id: increment-release-tag
+        run: |
+          current_server_tag=${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }}
+          previous_server_tag=${{ steps.get-server-tag.outputs.PREVIOUS_SERVER_TAG }}
+          
+          echo "Current server tag: $current_server_tag"
+          echo "Previous server tag: $previous_server_tag"
+
+          current_server_tag_only=${current_server_tag:1}
+          previous_server_tag_only=${previous_server_tag:1}
+
+          IFS='.' read -r -a current <<< "${current_server_tag_only}"
+          IFS='.' read -r -a previous <<< "${previous_server_tag_only}"
+
+          if [[ ${#current[@]} -eq 3 && ${#previous[@]} -eq 3 ]]; then
+            if [ "${current[0]}" -gt "${previous[0]}" ]; then
+              version_change=major
+            elif [ "${current[0]}" -eq "${previous[0]}" ] && [ "${current[1]}" -gt "${previous[1]}" ]; then
+              version_change=minor
+            elif [ "${current[0]}" -eq "${previous[0]}" ] && [ "${current[1]}" -eq "${previous[1]}" ] && [ "${current[2]}" -gt "${previous[2]}" ]; then
+              version_change=patch
+            fi
+          else
+            echo "Version format error"
+            exit 1
+          fi
+
+          echo "name=version_change::$version_change"
+
+          current_repo_version=${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+          current_repo_version_only=${current_repo_version:1}
+
+          IFS='.' read -r -a version_parts <<< "${current_repo_version_only}"
 
+          case "$version_change" in
+            major)
+              version_parts[0]=$((version_parts[0] + 1))
+              version_parts[1]=0
+              version_parts[2]=0
+              ;;
+            minor)
+              version_parts[1]=$((version_parts[1] + 1))
+              version_parts[2]=0
+              ;;
+            patch)
+              version_parts[2]=$((version_parts[2] + 1))
+              ;;
+          esac
+
+          new_repo_version="v${version_parts[0]}.${version_parts[1]}.${version_parts[2]}"
+          echo "new_repo_version::$new_repo_version"
+          echo "new_repo_version=$new_repo_version" >> "$GITHUB_OUTPUT"
+  
+      - name: Print fetched tags
+        run: |
+          echo "Current image tag: ${{ steps.fetch-latest-release-tags.outputs.tag_name }} ; target_commit branch: ${{ steps.fetch-latest-release-tags.outputs.target_commitish }}"
+          if [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+            echo "Incremented image tag: ${{ steps.increment-release-tag.outputs.new_repo_version }}"
+          fi
+  
       - name: Set docker image tags
         id: set-tags
         run: |
           ls -al
           set -a; source .env; set +a
           if [ "${{ inputs.repo }}" = "e-mission-server" ]; then
             echo "SERVER_IMAGE_TAG=${SERVER_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
+            echo "CURRENT_REPO_TAG=${SERVER_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
           elif [ "${{ inputs.repo }}" = "nrel-openpath-join-page" ]; then
             echo "JOIN_IMAGE_TAG=${JOIN_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
+            echo "CURRENT_REPO_TAG=${JOIN_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
           elif [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then
             echo "ADMIN_DASH_IMAGE_TAG=${ADMIN_DASH_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
+            echo "CURRENT_REPO_TAG=${ADMIN_DASH_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
           elif [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then
+            echo "CURRENT_REPO_TAG=${PUBLIC_DASH_NOTEBOOK_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
             echo "PUBLIC_DASH_NOTEBOOK_IMAGE_TAG=${PUBLIC_DASH_NOTEBOOK_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
             echo "PUBLIC_DASH_FRONTEND_IMAGE_TAG=${PUBLIC_DASH_FRONTEND_IMAGE_TAG}" >> "$GITHUB_OUTPUT"
           fi
 
       - name: Print input docker image tags
         run: |
-          echo "Event name: ${{ github.event_name }}"
+          echo "Event name: ${{ github.event_name }} | ${{ github.ref_type }}"
           if [ "${{ inputs.repo }}" = "e-mission-server" ]; then
             echo "Current server image tag: ${{ steps.set-tags.outputs.SERVER_IMAGE_TAG }}"
           elif [ "${{ inputs.repo }}" = "nrel-openpath-join-page" ]; then
@@ -59,14 +152,17 @@ jobs:
             echo "Current admin-dash image tag: ${{ steps.set-tags.outputs.ADMIN_DASH_IMAGE_TAG }}"
             echo "Latest server image tag (${{ github.event_name }}): ${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }}"
           elif [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then            
-            echo "Current notebook image tag (push): ${{ steps.set-tags.outputs.PUBLIC_DASH_NOTEBOOK_IMAGE_TAG }}"
-            echo "Current frontend image tag (push): ${{ steps.set-tags.outputs.PUBLIC_DASH_FRONTEND_IMAGE_TAG }}"
+            echo "Current notebook image tag: ${{ steps.set-tags.outputs.PUBLIC_DASH_NOTEBOOK_IMAGE_TAG }}"
+            echo "Current frontend image tag: ${{ steps.set-tags.outputs.PUBLIC_DASH_FRONTEND_IMAGE_TAG }}"
             echo "Latest server image tag (${{ github.event_name }}): ${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }}"
           fi
 
-      - name: docker login
-        run: |
-          docker login -u $DOCKER_USER -p $DOCKER_PASSWORD
+      - name: Login to GitHub Container Registry
+        uses: docker/login-action@v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Get current date # get the date of the build
         id: date
@@ -78,9 +174,9 @@ jobs:
       - name: build docker image
         run: |
           if [ "${{ inputs.repo }}" = "e-mission-server" ]; then
-            docker build -t $DOCKER_USER/${{ inputs.repo }}:${{ inputs.branch }}_${{ steps.date.outputs.date }} .
+            docker build -t ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.fetch-latest-release-tags.outputs.tag_name }} .
           elif [ "${{ inputs.repo }}" = "nrel-openpath-join-page" ]; then
-            docker build -t $DOCKER_USER/${{ inputs.repo }}:${{ inputs.branch }}_${{ steps.date.outputs.date }} ./frontend
+            docker build -t ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.fetch-latest-release-tags.outputs.tag_name }} ./frontend
           elif [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then
             SERVER_IMAGE_TAG=${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }} docker compose -f docker-compose-prod.yml build
           elif [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then            
@@ -89,50 +185,90 @@ jobs:
           docker images
 
       - name: rename docker image
+        if: ${{ github.ref_type == 'tag' || github.event_name == 'workflow_dispatch' }}
         run: |
-          if [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then
-            docker image tag e-mission/opdash:0.0.1 $DOCKER_USER/${{ inputs.repo }}:${{ inputs.branch }}_${{ steps.date.outputs.date }}
+          if [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then 
+            if [ "${{ github.event_name }}" == "release" ]; then
+              docker tag e-mission/opdash:0.0.1 ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              docker tag e-mission/opdash:0.0.1 ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.increment-release-tag.outputs.new_repo_version }}
+            fi
           elif [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then      
-            if [ "${{ github.event_name }}" == "push" ]; then
-              docker image tag em-pub-dash-prod/frontend:latest $DOCKER_USER/${{ inputs.repo }}_frontend:${{ inputs.branch }}_${{ steps.date.outputs.date }}
+            if [ "${{ github.event_name }}" == "release" ]; then
+              docker tag em-pub-dash-prod/frontend:latest ghcr.io/e-mission/${{ inputs.repo }}_frontend:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+              docker tag em-pub-dash-prod/viz-scripts:latest ghcr.io/e-mission/${{ inputs.repo }}_notebook:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              docker tag em-pub-dash-prod/viz-scripts:latest ghcr.io/e-mission/${{ inputs.repo }}_notebook:${{ steps.increment-release-tag.outputs.new_repo_version }}
             fi
-            docker image tag em-pub-dash-prod/viz-scripts:latest $DOCKER_USER/${{ inputs.repo }}_notebook:${{ inputs.branch }}_${{ steps.date.outputs.date }}
           fi
   
       - name: push docker image
+        if: ${{ github.ref_type == 'tag' || github.event_name == 'workflow_dispatch' }}
         run: |
           if [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then      
-            if [ "${{ github.event_name }}" == "push" ]; then
-              docker push $DOCKER_USER/${{ inputs.repo }}_frontend:${{ inputs.branch }}_${{ steps.date.outputs.date }}
+            if [ "${{ github.event_name }}" == "release" ]; then
+              docker push ghcr.io/e-mission/${{ inputs.repo }}_frontend:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+              docker push ghcr.io/e-mission/${{ inputs.repo }}_notebook:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              docker push ghcr.io/e-mission/${{ inputs.repo }}_notebook:${{ steps.increment-release-tag.outputs.new_repo_version }}
+            fi
+          elif [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then
+            if [ "${{ github.event_name }}" == "release" ]; then
+              docker push ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              docker push ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.increment-release-tag.outputs.new_repo_version }}
             fi
-            docker push $DOCKER_USER/${{ inputs.repo }}_notebook:${{ inputs.branch }}_${{ steps.date.outputs.date }}
           else
-            docker push $DOCKER_USER/${{ inputs.repo }}:${{ inputs.branch }}_${{ steps.date.outputs.date }}
+            docker push ghcr.io/e-mission/${{ inputs.repo }}:${{ steps.fetch-latest-release-tags.outputs.tag_name }}
           fi
 
       - name: Update .env file 
+        if: ${{ github.ref_type == 'tag' || github.event_name == 'workflow_dispatch' }}
         run: |
           if [ "${{ inputs.repo }}" = "e-mission-server" ]; then
-            echo "SERVER_IMAGE_TAG=${{ inputs.branch }}_${{ steps.date.outputs.date }}" > .env
+            echo "SERVER_IMAGE_TAG=${{ steps.fetch-latest-release-tags.outputs.tag_name }}" > .env
           elif [ "${{ inputs.repo }}" = "nrel-openpath-join-page" ]; then
-            echo "JOIN_IMAGE_TAG=${{ inputs.branch }}_${{ steps.date.outputs.date }}" > .env
+            echo "JOIN_IMAGE_TAG=${{ steps.fetch-latest-release-tags.outputs.tag_name }}" > .env
           elif [ "${{ inputs.repo }}" = "op-admin-dashboard" ]; then
-            echo "ADMIN_DASH_IMAGE_TAG=${{ inputs.branch }}_${{ steps.date.outputs.date }}" > .env
+            if [ "${{ github.event_name }}" == "release" ]; then
+              echo "ADMIN_DASH_IMAGE_TAG=${{ steps.fetch-latest-release-tags.outputs.tag_name }}" > .env
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              echo "ADMIN_DASH_IMAGE_TAG=${{ steps.increment-release-tag.outputs.new_repo_version }}" > .env
+            fi
             echo "SERVER_IMAGE_TAG=${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }}" >> .env
           elif [ "${{ inputs.repo }}" = "em-public-dashboard" ]; then
-            echo "PUBLIC_DASH_NOTEBOOK_IMAGE_TAG=${{ inputs.branch }}_${{ steps.date.outputs.date }}" > .env
-            if [ "${{ github.event_name }}" == "push" ]; then
-              echo "Push event: Update frontend image tag"
-              echo "PUBLIC_DASH_FRONTEND_IMAGE_TAG=${{ inputs.branch }}_${{ steps.date.outputs.date }}" >> .env
-            else
-              echo "Workflow_dispatch: Reuse existing frontend image tag"
+            if [ "${{ github.event_name }}" == "release" ]; then
+              echo "PUBLIC_DASH_NOTEBOOK_IMAGE_TAG=${{ steps.fetch-latest-release-tags.outputs.tag_name }}" > .env
+              echo "PUBLIC_DASH_FRONTEND_IMAGE_TAG=${{ steps.fetch-latest-release-tags.outputs.tag_name }}" >> .env
+            elif [ "${{ github.event_name }}" == "workflow_dispatch" ]; then
+              echo "PUBLIC_DASH_NOTEBOOK_IMAGE_TAG=${{ steps.increment-release-tag.outputs.new_repo_version }}" > .env
               echo "PUBLIC_DASH_FRONTEND_IMAGE_TAG=${{ steps.set-tags.outputs.PUBLIC_DASH_FRONTEND_IMAGE_TAG }}" >> .env
             fi
             echo "SERVER_IMAGE_TAG=${{ steps.get-server-tag.outputs.SERVER_IMAGE_TAG }}" >> .env
           fi
           cat .env
 
+      - name: Create, Push new tag and Publish release
+        if: ${{ github.event_name == 'workflow_dispatch' }}
+        run: |
+          git config --local user.email "[email protected]"
+          git config --local user.name "Github Actions bot to create new tags and publish releases"
+          git tag -a ${{ steps.increment-release-tag.outputs.new_repo_version }} -m"$(git log --pretty=format:%s ${{ steps.set-tags.outputs.CURRENT_REPO_TAG }}...HEAD --no-merges)"
+          git push origin ${{ steps.increment-release-tag.outputs.new_repo_version }}
+
+          echo "Logging latest created and pushed tag"
+          git describe --abbrev=0 --tags
+
+          curl -L \
+            -X POST \
+            -H "Accept: application/vnd.github+json" \
+            -H "Authorization: Bearer ${{ secrets.GH_FG_PAT_TAGS }}" \
+            -H "X-GitHub-Api-Version: 2022-11-28" \
+            https://api.github.com/repos/e-mission/${{ inputs.repo }}/releases \
+            -d '{"tag_name":"${{ steps.increment-release-tag.outputs.new_repo_version }}","target_commitish":"${{ inputs.branch }}","name":"${{ steps.increment-release-tag.outputs.new_repo_version }}","body":"Triggered on e-mission-server new release. Bumping up release version.","generate_release_notes":true}'
+
       - name: Add, Commit, Push changes to .env file
+        if: ${{ github.ref_type == 'tag' || github.event_name == 'workflow_dispatch' }}
         run: |
           git config --local user.email "[email protected]"
           git config --local user.name "Github Actions bot to update .env with latest tags"

diff --git a/Dockerfile b/Dockerfile
@@ -3,6 +3,8 @@ FROM ubuntu:jammy-20240227
 
 MAINTAINER K. Shankari ([email protected])
 
+LABEL org.opencontainers.image.source=https://github.com/e-mission/e-mission-server
+
 WORKDIR /usr/src/app
 
 RUN apt-get -y -qq update