-
Notifications
You must be signed in to change notification settings - Fork 21
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #48 from google/stage
Merging into Main
- Loading branch information
Showing
440 changed files
with
25,950 additions
and
14 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,21 @@ | ||
| # Getting started | ||
|
|
||
| This workflow covers the steps to setup ADO CICD pipeline for terraform. The setup involves setting up ADO repository and the corresponding CI/CD settings and variables. The pipeline triggers based on select events (like push to specific branches), authenticates to the specified service account using Workload Identity federation and runs the pipeline to deploy infrastructure using terraform in GCP.\ | ||
| A more comprehensive description of DevOps & GitOps principles can be found at [DevOps README](https://github.com/google/devops-governance/blob/GDC-phase-kickstarter-1/README.md). | ||
|
|
||
| ## Implementation Process | ||
|
|
||
| The setup consists of configuring folder-factory, project-factory and skunkworks as three ADO/gitlab repositories. The pre-requisites and the setup are detailed below for each of the repositories. | ||
|
|
||
| Workload identity federation enables applications running outside of Google Cloud to replace long-lived service account keys with short-lived access tokens. This is achieved by configuring Google Cloud to trust an external identity provider, so applications can use the credentials issued by the external identity provider to impersonate a service account.\ | ||
| To use Azure Devops with GCP Deployments, we can leverage on [Workload Identity Federation](https://cloud.google.com/iam/docs/workload-identity-federation) between Azure Devops and Google Cloud. This will be possible by configuring the Workload Identity Federation to trust OIDC tokens generated for a specific workflow in Azure Devops. | ||
|
|
||
| ## ADO Prerequisites | ||
|
|
||
| - Create a Project in ADO | ||
| - Add a repo called "folder factory" and copy code from [devops folder factory repo](../../../examples/guardrails/azuredevops/folder-factory) into it | ||
| - Add repo called "project factory" and copy code from [devops project factory repo](../../../examples/guardrails/azuredevops/project-factory) into it | ||
| - Add a repo called "skunkworks" and copy code from [devops skunkworks repo](../../../examples/guardrails/azuredevops/skunkworks) into it. | ||
|
|
||
| > .\ | ||
| Once ADO set is completed go to [Folder Factory](../../../examples/guardrails/azuredevops/folder-factory) |
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,96 @@ | ||
| # Folder Factory | ||
|
|
||
| This is a template for a DevOps folder factory. | ||
|
|
||
| It can be used with [https://github.com/google/devops-governance/tree/main/examples/guardrails/project-factory](https://github.com/google/devops-governance/tree/main/examples/guardrails/project-factory) and is intended to house the folder configurations: | ||
|
|
||
| <img width="1012" alt="Screenshot 2023-03-10 at 02 52 08" src="https://user-images.githubusercontent.com/94000358/224204373-f17024c0-1a2c-474b-82c2-affd8119cc05.png"> | ||
|
|
||
| Using Keyless Authentication the project factory connects a defined Github repository with a target service account and project within GCP for IaC. | ||
|
|
||
| The idea is to enable developers of the "skunkworks" repository to deploy into the "skunkworks" project via IaC pipelines on Github. | ||
|
|
||
|
|
||
| ## Setting up folders | ||
|
|
||
| The folder factory will: | ||
| - create a folders with defined organisational policies | ||
|
|
||
| It uses YAML configuration files for every folder with the following sample structure: | ||
| ``` | ||
| parent: folders/XXXXXXXXX | ||
| org_policies: | ||
| policy_boolean: | ||
| constraints/compute.disableGuestAttributesAccess: true | ||
| constraints/iam.disableServiceAccountCreation: false | ||
| constraints/iam.disableServiceAccountKeyCreation: false | ||
| constraints/iam.disableServiceAccountKeyUpload: false | ||
| constraints/gcp.disableCloudLogging: false | ||
| policy_list: | ||
| constraints/compute.vmExternalIpAccess: | ||
| inherit_from_parent: null | ||
| status: true | ||
| suggested_value: null | ||
| values: | ||
| iam: | ||
| roles/resourcemanager.projectCreator: | ||
| - serviceAccount:XXXXX@XXXXXX | ||
| ``` | ||
|
|
||
| Every folder is defined with its own yaml file located in the following [Folder](data/folders). | ||
| Copy "folder.yaml.sample" to "folder_name.yaml"; Name of the yaml file will be used to create folder with the same name. | ||
| Once folder_name.yaml file is created update yaml file | ||
| * parent - can be another folder or organization | ||
| * ServiceAccount | ||
| data/folders can have multiple yaml files and a folder will be created for each yaml file. | ||
|
|
||
|
|
||
| ## How to run this stage | ||
| ### Prerequisites | ||
|
|
||
| Workload Identity setup between the folder factory code repositories and the GCP Identity provider configured with a service account containing required permissions to create folders and their organizational policies. There is a sample code provided in “folder.yaml.sample” to create a folder and for terraform to create a folder minimum below permissions are required. | ||
| “Folder Creator” or “Folder Admin” at org level | ||
| “Organization Policy Admin” at org level | ||
|
|
||
|
|
||
| ### Installation Steps | ||
| From the folder-factory edit the azure-pipeline.yml | ||
|
|
||
| * CI/CD variables | ||
|
|
||
| Add the variables to the pipeline as described in the table below. | ||
|
|
||
|
|
||
| ### Terraform config validator | ||
| The pipeline has an option to utilise the integrated config validator (gcloud terraform vet) to impose constraints on your terraform configuration. You can enable it by setting the CI/CD Variable $TERRAFORM_POLICY_VALIDATE to "true" and providing the policy-library repo URL to $POLICY_LIBRARY_REPO variable. See the below for details on the Variables to be set on the CI/CD pipeline. | ||
|
|
||
|
|
||
| | Variable | Description |Sample value | | ||
| |--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------| | ||
| | GCP_PROJECT_ID | The GCP project ID of your service account | sample-project-1122 | | ||
| | GCP_SERVICE_ACCOUNT | The Service Account to be used for creating folders | [email protected] | | ||
| | GCP_WORKLOAD_IDENTITY_PROVIDER | The Workload Identity provider URI configured with the Service Account and the repository | projects/${PROJECT_NUMBER}/locations/global/workloadIdentityPools/${POOL_NAME}/providers/${PROVIDER_NAME} | | ||
| | STATE_BUCKET | The GCS bucket in which the state is to be centrally managed. The Service account provided above must have access to list and write files to this bucket. Use a seed project if running this as part of Foundations or create a new GCS Bucket. | sample-terraform-state-bucket | | ||
| | TF_LOG | The terraform env variable setting to get detailed logs. Supports TRACE,DEBUG,INFO,WARN,ERROR in order of decreasing verbosity | WARN | | ||
| | TF_ROOT | The directory of the terraform code to be executed. | $CI_PROJECT_DIR | | ||
| | TF_VERSION | The terraform version to be used for execution. The specified terraform version is downloaded and used for execution for the workflow. | 1.3.6 | ||
| | TERRAFORM_POLICY_VALIDATE | Set this value as true if terraform vet is to be run against the policy library repository set in $POLICY_LIBRARY_REPO variable | true | | ||
| | POLICY_LIBRARY_REPO | The policy library repository URL which will be cloned using git clone to run gcloud terraform vet against. | https://github.com/GoogleCloudPlatform/policy-library | ||
|
|
||
| * Once the prerequisites are set up, trigger the pipeline manually or set the trigger as per your need. | ||
|
|
||
|
|
||
| * .gcp-auth script should run successfully in the pipeline if the workload identity federation is configured as required. | ||
|
|
||
| ### Pipeline Workflow Overview | ||
| The complete workflow comprises of 4-5 stages | ||
| * Stages: | ||
| * gcp-auth : creates the wif credentials by impersonating the service account. | ||
| * terraform init : initializes terraform in the specified TF_ROOT directory | ||
| * setup-terraform : Downloads the specified TF_VERSION and passes it as a binary to the next stages | ||
| * validate: Runs terraform fmt check and terraform validate. This stage fails if the code is not run against terraform fmt command | ||
| * plan: Runs terraform plan and saves the plan and json version of the plan as artifacts | ||
| * policy-validate: Runs gcloud terraform vet against the terraform code with the constraints in the specified repository. | ||
| * apply: once the plan is successful. | ||
| Runs terraform apply and creates the infrastructure specified. | ||
|
|
Oops, something went wrong.