Helm deployment fixes (#735)

* Include helm chart and healthchecks

* Modified the traefik.yml config to separate between local and production, in order to attempt to get production traefik deployment to function as it did in another branch. Updated docker compose files to separate keycloak into a profile and remove docs service from production. Updated manage script to utilize new compose files for deployment and testing.

* Updated some docs and the manage scripts

* Merged in some minor changes from traefik-deployment-fixes branch to update app css and include other minor fixes.

* Delete docker-compose.prod-overlay.yml

* Pre-comfig fixes, update to scripts.

* Some updates to docker deployment to test if they will work in production

* Separated trafik.yml into one for production and one for local, to avoid using the conditional statements that were being used which seemed to not be working for traefik deployment. Updated the deployment script to make sure local and production custm overlays are applied last so their variables aren't overriden.

* Removed unnecessary prod_traefik file.

* Updated prod overlay to name the containers with prod to distinguish deployment container types. Updated the config helper script to generate production overlay

* Updated changelog. Added dark theme styling to the React Joyride tutorials.

* Implemented some suggestions from PR comments. Brightened the tutorial background in dark theme.

* Minor fix to compose file.

* Reverted container names to leave them be.

* Updated obs4mips url and modified the display button. Remove obsolete updateProjects.sh script and updated the README's regarding updating project data. Added the update functions to the manage_metagrid.sh scripts instead. Still need to test in production, but should work.

* Fix minor lint error. Added build step to project migration function.

---------

Co-authored-by: Zach Price <[email protected]>

.github/workflows/helm-chart.yml

@@ -0,0 +1,26 @@
+name: Helm chart
+
+on:
+  push:
+    tags:
+    - 'v*'
+
+jobs:
+  publish:
+    name: Publish
+    runs-on: ubuntu-latest
+    permissions:
+      packages: write
+    steps:
+    - uses: actions/checkout@v4
+    - name: Chart | Push
+      uses: appany/[email protected]
+      with:
+        name: Metagrid
+        repository: esgf2-us
+        tag: ${{ github.ref_name }}
+        path: helm
+        registry: ghcr.io
+        registry_username: ${{ github.actor }}
+        registry_password: ${{ secrets.GITHUB_TOKEN }}
+        update_dependencies: 'true'

.gitignore

@@ -183,8 +183,11 @@ tags
 .env
 .envs/*
 !.envs/.local/
-docker-compose.*-overlay.yml
+docker-compose*-overlay.yml
 
 # Documentation artifacts
 docs/site
 frontend/public/runtime_env.js
+
+#Helm chart deps
+helm/charts/*

.pre-commit-config.yaml

@@ -9,7 +9,7 @@ repos:
       - id: trailing-whitespace
       - id: end-of-file-fixer
       - id: check-yaml
-        exclude: traefik/traefik.yaml
+        exclude: traefik/*.yml|helm/templates/.*
 
   # Back-end
   # ------------------------------------------------------------------------------

backend/.dockerignore

@@ -1,4 +1,3 @@
 .*
 !.coveragerc
-!.env
 !.pylintrc

backend/.gitignore

@@ -90,3 +90,6 @@ ENV/
 .env
 .envs/.production/.django
 .envs/.production/.postgres
+
+# Helm
+helm/charts/*

backend/README.md

@@ -1,5 +1,12 @@
 # MetaGrid API Back-end
 
+# STEPS TO UPDATE PROJECTS, FACETS OR CATEGORIES
+
+1. Edit the initial data file with the desired changes: metagrid/backend/metagrid/initial_projects_data.py
+2. Start the manage_metagrid.sh script in the root directory (use sudo if needed): ./manage_metagrid.sh
+3. Select 'Developer Actions', then choose "Update Project Table" option. The commands to update should run.
+DONE!
+
 # Create a new python virtual environment for testing
 
 python3 -m venv backend/venv

backend/UpdateProjectData_README.txt

@@ -1,43 +1,47 @@
 STEPS TO UPDATE PROJECTS, FACETS OR CATEGORIES
 
 1. Edit the initial data file with the desired changes: metagrid/backend/metagrid/initial_projects_data.py
-2. Change to the backend directory:
-cd metagrid/backend/
-3. Make sure the docker traefik and backend containers are up and running.
-If not running, you can run the containers by going to the traefik directory first and running this command:
-sudo docker compose -f docker-compose.prod.yml up --build -d
-Then do the same in the backend directory (production backend depends on traefik)
-
-RUN UPDATE AND CLEAR TABLES
-If you need to clear tables to remove existing facets/projects or change their order then do the steps below:
+2. Go to the root directory so you can perform the steps below (you may need sudo privileges)
+
+RUN UPDATE WITHOUT CLEARING TABLES
+If your update is small and only involves minor modifications or additions to existing projects then you don't need to clear the tables.
+If there are deletions or more significant changes, run steps further below which include clearing tables, otherwise:
 
 Option 1:
-4. Use the updateProjects.sh script. Just run the script using clear option:
-sudo ./updateProject.sh --clear
+3. Start the manage_metagrid.sh script in the root directory (use sudo if needed): ./manage_metagrid.sh
+4. Select 'Developer Actions', then choose "Update Project Table" option. The commands to update should run.
 
 DONE!
 
-Option 2: Manually update without the script and clear tables:
-4. sudo docker compose -f docker-compose.prod.yml build django # Build the container
-5. sudo docker compose -f docker-compose.prod.yml run --rm django python manage.py migrate projects zero
-6. sudo docker compose -f docker-compose.prod.yml run --rm django python manage.py migrate projects
+Option 2: Manually update without the script and don't clear tables (LOCAL):
+3. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml build django # Build the container
+4. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml run --rm django python manage.py migrate --fake projects 0001_initial # Fake projects so it does migration
+5. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml run --rm django python manage.py migrate projects # Run migration
+6. docker compose --profile "*" down --remove-orphans # Stop all containers
 
-DONE!
+Option 2: Manually update without the script and don't clear tables (PRODUCTION):
+3. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml build django # Build the container
+4. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml run --rm django python manage.py migrate --fake projects 0001_initial # Fake projects so it does migration
+5. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml run --rm django python manage.py migrate projects # Run migration
+6. docker compose --profile "*" down --remove-orphans # Stop all containers
 
-RUN UPDATE WITHOUT CLEARING TABLES
-If your update is small and only involves minor modifications or additions to existing projects then you don't need to clear the tables.
-If there are deletions or more significant changes, run steps above which include clearing tables otherwise do steps below:
-
-Option 1:
-4. Use the updateProjects.sh script. Just run the default script:
-sudo ./updateProject.sh
 
 DONE!
 
-Option 2: Manually update without the script and don't clear tables:
-4. sudo docker compose -f docker-compose.prod.yml build django # Build the container
-5. sudo docker compose -f docker-compose.prod.yml run --rm django python manage.py migrate --fake projects 0001_initial
-6. sudo docker compose -f docker-compose.prod.yml run --rm django python manage.py migrate projects
+RUN UPDATE AND CLEAR TABLES, WARNING ISSUES COULD ARISE WITH THIS AND USER CARTS MAY BE AFFECTED
+If you need to clear tables to remove existing facets/projects or change their order then do the steps below:
+
+Manually update and clear tables (LOCAL):
+3. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml build django  #Build the container
+4. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml run --rm django python manage.py migrate projects zero # Zero the project table (will also affect cart items that refer to a specific project)
+5. docker compose -f docker-compose.yml -f docker-compose-local-overlay.yml run --rm django python manage.py migrate projects # Run migration
+6. docker compose --profile "*" down --remove-orphans # Stop all containers
+
+Manually update and clear tables (PRODUCTION):
+3. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml build django  #Build the container
+4. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml run --rm django python manage.py migrate projects zero # Zero the project table (will also affect cart items that refer to a specific project)
+5. docker compose -f docker-compose.yml -f docker-compose.prod.yml -f docker-compose-prod-overlay.yml run --rm django python manage.py migrate projects # Run migration
+6. docker compose --profile "*" down --remove-orphans # Stop all containers
 
 DONE!