Skip to content

Latest commit

 

History

History
124 lines (93 loc) · 6.81 KB

README.md

File metadata and controls

124 lines (93 loc) · 6.81 KB

terraform-google-airbyte-infra

A Terraform module to deploy an Airbyte server on a GCP compute engine instance.

Deployed Airbyte version: 0.52.0

Prerequisites

  • Terraform. Tested with v1.5.3
  • A GCP project and an authenticated gcloud CLI

Permissions

  • Broad roles that will work, but not recommended for service accounts or even people.
    • roles/owner
  • Recommended roles to respect the least privilege principle.
    • roles/compute.admin
    • roles/iam.serviceAccountAdmin
    • roles/resourcemanager.projectIamAdmin
  • Granular permissions required to build a custom role specific for this deployment.
    • compute.addresses.create
    • compute.addresses.delete
    • compute.disks.create
    • compute.firewalls.create
    • compute.firewalls.delete
    • compute.instances.create
    • compute.instances.delete
    • compute.instances.setMetadata
    • compute.instances.setServiceAccount
    • compute.networks.create
    • compute.networks.delete
    • compute.networks.updatePolicy
    • compute.routers.create
    • compute.routers.delete
    • compute.routers.update
    • compute.routes.create
    • compute.routes.delete
    • compute.subnetworks.create
    • compute.subnetworks.delete
    • compute.subnetworks.use
    • iam.serviceAccounts.create
    • iam.serviceAccounts.delete
    • resourcemanager.projects.setIamPolicy

Usage

Go to the examples directory to view deployment code samples.

Once deployment is successfully finished, you will need to SSH tunnel to your Airbyte instance:

$ gcloud compute ssh airbyte -- -L 8000:localhost:8000 -N -f

You will then be able to access it via your browser.

You may also Terraform the ELT flows themselves using the Artefactory terraform-google-airbyte-flows module, or the Airbyte community provider.

Architecture

This module will provision the following resources.

As a security precaution, the Airbyte VM is not reachable from the internet. The only way to access it is through gcloud-authenticated SSH. Egress traffic is of course possible to access remote data sources to pull data from.

The Airbyte service account has fairly high level of privilege on GCS and BQ (roles/storage.objectAdmin, and roles/bigquery.dataEditor), allowing it to read/write on any table or bucket. These permissions are required to allow Airbyte to create temporary datasets and tables. If that is an issue for you, you can isolate this deployment in a dedicated project.

Auto-generated module documentation

Requirements

No requirements.

Providers

Name Version
google n/a

Modules

No modules.

Resources

Name Type
google_compute_address.airbyte_external_ip resource
google_compute_firewall.allow_internal_traffic resource
google_compute_firewall.allow_ssh_from_iap resource
google_compute_instance.airbyte_vm resource
google_compute_network.airbyte_vpc resource
google_compute_route.internet_route resource
google_compute_router.router resource
google_compute_router_nat.airbyte_nat resource
google_compute_subnetwork.airbyte_subnet resource
google_project_iam_member.airbyte_iam resource
google_service_account.airbyte resource

Inputs

Name Description Type Default Required
config Configuration for the Airbyte VM 
object({
    vm_machine_type = optional(string, "e2-standard-2")
    vm_disk_size    = optional(number, 30)
    airbyte_vm_name = optional(string, "airbyte")
    airbyte_sa_name = optional(string, "airbyte")
    labels          = optional(map(string), {})

    vpc_name            = optional(string, "airbyte-vpc")
    subnet_name         = optional(string, "airbyte-subnet")
    ip_cidr_range       = optional(string, "10.0.1.0/24")
    router_name         = optional(string, "airbyte-router")
    external_ip_name    = optional(string, "airbyte-ip")
    nat_name            = optional(string, "airbyte-nat")
    internet_route_name = optional(string, "airbyte-internet-route")
    network_tags        = optional(list(string), [])
  })
 {} no
project_id GCP project id string n/a yes
region GCP region string n/a yes
zone GCP zone string n/a yes

Outputs

Name Description
airbute_nat n/a
airbyte_address n/a
airbyte_router n/a
airbyte_service_account n/a
airbyte_subnet n/a
airbyte_vm_name n/a
airbyte_vpc n/a