dev-infrastructure

Dev CDP Infrastructure

Full infrastructure setup for whole system / integration level testing.

---

Initial Comments

This is primarily used for developer stack creation and management. We have an example stack, infrastructure, pipeline, and web app available for demonstration and example data usage in our Seattle staging repo. The web page for the Seattle staging instance can be found here.

If you are just trying to process example CDP data (or for front-end: visualize example CDP data) and not upload data, it is recommended to simply point your requests at the example stack.

For in-depth details on infrastructure terminology and uses refer to the documentation found in our cookiecutter-cdp-deployment repository.

---

Dependencies

Deploying the CDP infrastructure requires having cdp-backend installed along with the contributor tools.

For more detailed information please see the project installation details and contributing details.

Account Setup

1. Create (or sign in to) a Google Cloud Platform (GCP) account. (Google Cloud Console Home)
2. Create (or re-use) a billing account and attach it to your GCP account.

Environment Setup

The only environment setup needed to run this deployment is to make sure the gcloud SDK is installed.

If this was the first time installing either of those packages, it is recommended to restart your terminal after installation.

Create a New Project and Deploy the Infrastructure

get_cdp_infrastructure_stack {dir path to store to}
just login
just init {project-name}

Attach your billing account to the GCP project you just created in the GCP console.

just setup-and-deploy {project-name} {OPTIONAL: region}

Example:

Assuming user is within the dev-infrastructure dir.

get_cdp_infrastructure_stack .
just init cdp-eva-dev-001
just setup-and-deploy cdp-eva-dev-001

Update an Existing Project

just deploy {project-name}

Or optionally passing cookiecutter yaml

just deploy {project-name} cookiecutter-yaml={path to yaml file}

Example:

just deploy cdp-eva-dev-001

Or optionally passing cookiecutter yaml

just deploy cdp-eva-dev-001 cookiecutter-yaml='some/fakepath.yaml'

Enable video / audio clipping:

just deploy-clipping {key} {region}

Example:

just deploy-clipping /home/active/cdp/cdp-eva-dev-001.json us-central

All Commands

• See Justfile commands with just or open the Justfile.

Changing the Stack

The actual infrastructure files live in the cdp_backend/infrastructure module. To make changes to the infrastructure stack, change the files in that module and then rerun get-cdp-infrastructure-stack.

Note: the database indexes are store in the cdp_backend/database/models.py module with each collection model.

Google Cloud Functions

Useful links for managing Cloud Functions:

• Pre-installed System Packages
• Execution Environments

Troubleshooting

If you receive a firebase: not found error at any point in the deployment process, make sure that you have the Firebase CLI installed.

Invalid project selection, please verify project cdp-isaac-dev-2023 exists and you have access.

• Run firebase projects:list and check whether your project is listed
• Run firebase projects:addfirebase {{project-name}}

If you are running into issues with various packages or tools not being installed, it is highly recommended to start a fresh virtual env.