Skip to content

production deployment

Jump to bottom
Ricardo Garcia Silva edited this page Dec 9, 2024 · 2 revisions

Production deployment plan

Deployment to production environment shall follow a semi-automated process. The initial deployments will require access to the node via SSH - later on we may lift this requirement.

The idea is to use a deployment script, coupled with a suitable configuration file, which takes care of most deployment-related tasks.

The script is located on the backend repo, at the /deployments/deploy.py relative path.

Overview

Deployments work by using suitable docker compose configuration files. The workflow is automated by the deployment script and consists of:

  1. Retrieving the latest version of the deployment-related files from the source code repository
  2. Standing down the docker compose stack, if it is running
  3. Updating system docker images by pulling their specified versions from the docker registry
  4. Standing up the stack again, with the new images
  5. Updating translations and perform any database migrations

Initial production environment bootstrap

In order to ensure a healthy deployment environment exists we need to perform some bootstrapping steps:

Have all necessary system packages installed

Ensure git and docker are installed:

sudo apt-get install --yes \
    curl \
    git
    
curl \
    -fsSL https://download.docker.com/linux/ubuntu/gpg | \
    sudo gpg \
    --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg

echo \
"deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu \
$(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

sudo apt-get update
sudo apt-get install docker-ce docker-ce-cli containerd.io

sudo groupadd docker
sudo usermod -aG docker $USER

If installing docker, be sure to log out of the current session and then log back in, which would be necessary for the new group memberships to be loaded.

Ensure there is a periodic cleaning of unused docker images

The backend source code repository contains sample systemd unit files that can be used to set up a periodic service which cleans up any unused docker images. These sample files are located at the /deployments/systemd-unit-files path. They consist of two files:

  • clean-docker-images.service - specifies that the command docker system prune --all --force shall be run
  • clean-docker-images.timer - specifies that the .service file shall run every sunday at 01h

Copy both of them on to the node's /etc/systemd/system directory. Modify the .service file, ensuring that any {CHANGE_ME} placeholder values get modified to their correct value.

Finally, enable the timer by running:

sudo systemctl start clean-docker-images.timer
Login to docker registry

The deployment workflow pulls docker images from the ghcr.io docker registry. This registry is owned by github and does not allow anonymous access. Therefore, it is necessary to log in to the registry in order to be able to download images from it.

Currently, the github registry can only be accessed by registered users and it needs a user to provide either their password or a personal access token (PAT).

Obtain a relevant PAT and then login to the registry:

docker login ghcr.io

# when instructed, provide the user email and PAT

Deployment preparation

In order for a deployment to be possible we need to perform some preparations:

Upload docker images to the github docker registry

Deployments to production use docker images and expect to be able to retrieve them from the github docker registry. Therefore, images for both the backend and the frontend applications must have been previously uploaded to the registry in order to be deployed.

The deployment script can be told which images to use for backend in frontend by one of two ways:

  1. Passing the --backend-image and --frontend-image parameters when calling the deployment script
  2. Having the main.backend_image and main.frontend_image parameters properly configured in the deployment configuration file

In either case, the value of both the backend and frontend image parameters must be the full URL of the docker image, including the registry and including the tag name.

Note

Prefer to deploy versioned docker images

In order to make it easy to always know which code is running in the production environment, always use a properly versioned docker tag. This means pushing a git tag to the respective source code repository and using the same name for the docker image tag too. Pushing a git tag and building a docker image with the same tag should optimally be done in sequence, by an automated project, like a CI pipeline.

Both the backend and frontend source code repositories have Continuous Integration (CI) pipelines running on the github actions platform. These CI pipelines will build a new docker image when a new git tag is pushed to the respective source code repository, tag the built docker image with the same tag used in git, and then push the image to the docker registry.

Create base deployment directory and copy the relevant files over

The deployment script must be able to operate on some directory as it needs to fetch files from the source code repository. Create a suitable directory:

DEPLOYMENT_ROOT=/opt/arpav-cline
sudo mkdir ${DEPLOYMENT_ROOT}
sudo chown $(id -u):$(id -g) ${DEPLOYMENT_ROOT}
cd ${DEPLOYMENT_ROOT}

Now clone the backend source code repository and get the initial version of the deployment files:

Note

This is a bootstrap procedure that only needs to be done once. Subsequent deployments will re-download the necessary files automatically.

# clone repo and get initial deployment files
git clone https://github.com/geobeyond/Arpav-PPCV-backend.git
cp Arpav-PPCV-backend/deployments/deploy.py .
cp Arpav-PPCV-backend/deployments/sample-prod-deployment.cfg .
rm -rf Arpav-PPCV-backend

Although not strictly required, you should also copy into this directory:

  • TLS-related files
  • traefik basic auth users file
  • uvicorn log config file

This is just for keeping everything together

Create or modify the deployment configuration file

The deployment configuration file is required for the deployment script to execute properly. This file must be in a format suitable for parsing with Python's configparser library. The backend code repository contains a sample configuration file, which can be copied and used as a base. This sample file is located at /deployments/sample-prod-deployment.cfg.

cd ${DEPLOYMENT_ROOT}
mv sample-prod-deployment.cfg prod-deployment.cfg

The file is expected to contain the following entries and all of them are mandatory, with two notable exceptions, as mentioned in the previous section:

section parameter description example
main backend_image full docker registry URL of the backend image ghcr.io/geobeyond/arpav-ppcv-backend/arpav-ppcv-backend:v1.0.0
main db_image_tag tag name of the postgis/postgis image to use for various DB containers 16-3.4
main deployment_root path to the directory where the deployment files reside /opt/arpav-cline
main frontend_image full docker registry URL of the frontend image ghcr.io/geobeyond/arpav-ppcv/arpav-ppcv:v1.0.0
main discord_notification_urls comma-separated list of discord-webhook URLs which will be notified when a deployment is done
main deployment_files_repo URL of the git repository where the deployment-related files reside https://github.com/geobeyond/Arpav-PPCV-backend.git
main martin_image_tag tag name of the ghcr.io/maplibre/martin image to use v0.13.0
main prefect_server_image_tag tag name of the prefecthq/prefect image to use 3.0.0rc17-python3.10
db name name of the backend db arpavppcv
db password password of the backend db. Choose something hard to guess and keep it secret.
db user username of user that connects to the backend db arpav
prefect_db name name of the prefect db prefect
prefect_db password password of the prefect db. Choose something hard to guess and keep it secret.
prefect_db user username of user that connects to the prefect db prefect
reverse_proxy image_tag tag name of the traefik image to use 3.0.2
reverse_proxy tls_cert_path local path to TLS certificate /opt/arpav-cline/cert.crt
reverse_proxy tls_cert_key_path local path to TLS certificate key /opt/arpav-cline/cert.key
reverse_proxy traefik_users_file_path local path to traefik basicauth users file. This contains users that can access the prefect UI section /opt/arpav-cline/basicauth-users.txt
tolgee_app env_tolgee_authentication_initial_password initial password for the tolgee user
tolgee_app env_tolgee_authentication_jwt_secret JWT secret key for auth-related tasks
tolgee_app env_tolgee_frontend_url public URL of the tolgee service https://tolgee.cline.arpav.it
tolgee_app image_tag tag name of the tolgee/tolgee image to use v3.71.4
tolgee_db name name of the tolgee db tolgee
tolgee_db password password of the tolgee db. Choose something hard to guess and keep it secret.
tolgee_db user username of user that connects to the tolgee db tolgee
webapp env_admin_user_password password for the user that is able to access the admin section
webapp env_admin_user_username username for the user that is able to access the admin section admin
webapp env_cors_origins comma-separated list of allowed CORS origins https://cline.arpav.it
webapp env_num_uvicorn_worker_processes how many worker processes should the uvicorn server use 4
webapp env_public_url public URL of the system https://cline.arpav.it
webapp env_session_secret_key secret key used in sessions
webapp env_thredds_server_base_url base URL of the THREDDS server https://thredds.arpav.it/thredds
webapp env_uvicorn_log_config_file local path to the uvicorn log configuration /opt/arpav-cline/prod-log-config.yaml

Deploy the system

After having gone through the steps mentioned in the previous sections, performing an actual deployment is simple: just execute the deployment script, passing it the correct configuration file and the --confirm flag:

cd ${DEPLOYMENT_ROOT}
python3 deploy.py --config-file prod-deployment.cfg --confirm
Clone this wiki locally