Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Doc Updates #159

Merged
merged 47 commits into from
Oct 31, 2024
Merged
Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
5bda45f
Move files from local
yanmastin-astro Oct 18, 2024
deaf6fc
Commit generated docs
yanmastin-astro Oct 18, 2024
4e0559c
Updates from review
yanmastin-astro Oct 21, 2024
6c5dddc
Update generated docs
yanmastin-astro Oct 21, 2024
179deeb
Update templates/guides/import-script.md
yanmastin-astro Oct 22, 2024
4facb11
Updates from review
yanmastin-astro Oct 22, 2024
03cecc1
Formatting fixes
yanmastin-astro Oct 22, 2024
4c88e5f
Update generated docs
yanmastin-astro Oct 22, 2024
e18ca2b
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
f55c12c
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
10873a1
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
1f4eac3
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
a7c9ffb
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
735720f
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
ae35d61
Update templates/guides/get-started.md
yanmastin-astro Oct 23, 2024
0983feb
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
0206161
Update templates/guides/import-script.md
yanmastin-astro Oct 23, 2024
cd56eb5
Update from review
yanmastin-astro Oct 23, 2024
e7053f0
Add prereqs
yanmastin-astro Oct 23, 2024
5a9f76b
Add note
yanmastin-astro Oct 23, 2024
b7ca225
Update generated docs
yanmastin-astro Oct 23, 2024
50291fc
Misc update
yanmastin-astro Oct 25, 2024
a2b4a7e
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
c1c6aff
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
9431cf7
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
87ee5a8
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
ce179de
Update templates/guides/get-started.md
yanmastin-astro Oct 25, 2024
d549319
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
9b51a95
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
2066346
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
6e14481
Update templates/guides/import-script.md
yanmastin-astro Oct 25, 2024
6a6bcf2
Add clarity to note
yanmastin-astro Oct 25, 2024
a4e5cc4
Remove description
yanmastin-astro Oct 28, 2024
7e7407d
Update generated docs
yanmastin-astro Oct 28, 2024
ec5d92b
Update templates/guides/import-script.md
yanmastin-astro Oct 29, 2024
84d78f1
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
c254fbe
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
cc81d65
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
1d69fb1
Update templates/guides/import-script.md
yanmastin-astro Oct 30, 2024
2288384
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
237ce3f
Update templates/guides/get-started.md
yanmastin-astro Oct 30, 2024
13f6d20
Make active voice
yanmastin-astro Oct 30, 2024
a07942c
Update generated docs
yanmastin-astro Oct 30, 2024
3dbb93f
Update generated docs
yanmastin-astro Oct 30, 2024
ccc558d
Misc update
yanmastin-astro Oct 31, 2024
8e6e32c
Update templates/guides/import-script.md
yanmastin-astro Oct 31, 2024
a07c495
Update generated docs
yanmastin-astro Oct 31, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
216 changes: 216 additions & 0 deletions docs/guides/get-started.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,216 @@
---
page_title: "Get started with Astro Terraform Provider"
---

# Get started with Astro Terraform Provider
In this guide, we will automate the onboarding of a new team by creating and managing an Astro Workspace and Deployment. By the end of this tutorial, you will have a fully automated setup that is reproducible and easily scalable to more teams.

## Step 1: Create Your Terraform Working Directory
1. Create a folder, `my-data-platform` for your Terraform project.
2. Save the following code in a file named `terraform.tf`:
```
terraform {
required_providers {
astro = {
source = "astronomer/astro"
version = "1.0.0"
}
}
}

provider "astro" {
organization_id = <your-organization-id>
}
```
3. Insert your organization's ID for `<your-organization-id>`. The working directory will contain all your Terraform code, and all Terraform commands will be run from this directory.

## Step 2: Initialize the Terraform Working Directory
1. Run `terraform init`. You will see Terraform downloading and installing the Astro Terraform provider to your local computer:
```
$ terraform init

Initializing the backend...

Initializing provider plugins...
- Finding astronomer/astro versions matching "1.0.0"...
- Installing astronomer/astro v1.0.0...
- Installed astronomer/astro v1.0.0 (signed by a HashiCorp partner, key ID F5206453FDEA33CF)

...

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
```
2. The versions and hashes of providers are stored in a generated file `.terraform.lock.hcl`. Store this file in version control.

## Step 3: Authenticate with Astro
1. [Create an API token](https://www.astronomer.io/docs/astro/automation-authentication#step-1-create-an-api-token) in Astro. Since you are creating a Workspace, you need an [Organization API token](https://www.astronomer.io/docs/astro/organization-api-tokens) with [Organization Owner permissions](https://www.astronomer.io/docs/astro/user-permissions#organization-roles).
2. Configure the API token as an environment variable `ASTRO_API_TOKEN` to run Terraform commands:
`export ASTRO_API_TOKEN=<your-api-token>`

Alternatively, users can set their API token value in the provider block:
```
provider "astro" {
organization_id = "&lt;your-organization-id&gt;"
token = "&lt;your-api-token&gt;"
}
```

## Step 4: Define Resources in Terraform
In a file `main.tf`, define two resources, an `astro_workspace` and an `astro_deployment`. These resources will represent an Astro workspace and Astro Deployment, defined in Terraform code:
```
# Create a new workspace
resource "astro_workspace" "my_first_tf_workspace" {
name = "My first TF workspace"
description = "My first Terraform-created workspace"
cicd_enforced_default = false
}

# Create a new Deployment
resource "astro_deployment" "my_first_tf_deployment" {
name = "My first TF deployment"
description = "My first Terraform-created deployment"
type = "STANDARD"
cloud_provider = "AWS"
region = "us-east-1"
contact_emails = []
default_task_pod_cpu = "0.25"
default_task_pod_memory = "0.5Gi"
executor = "CELERY"
is_cicd_enforced = true
is_dag_deploy_enabled = true
is_development_mode = false
is_high_availability = false
resource_quota_cpu = "10"
resource_quota_memory = "20Gi"
scheduler_size = "SMALL"
workspace_id = astro_workspace.my_first_tf_workspace.id
environment_variables = []
worker_queues = [{
name = "default"
is_default = true
astro_machine = "A5"
max_worker_count = 10
min_worker_count = 0
worker_concurrency = 1
}]
}
```
-> One of the key characteristics (and benefits) of using Terraform is that it's *declarative*. For example, `workspace_id = astro_workspace.my_first_tf_workspace.id` tells Terraform to configure the Workspace ID in the Deployment. This means the Workspace must be created first, producing an ID which is a generated value and unknown at the time of writing. You don't have to instruct Terraform to create resources in a certain order, you only have to instruct what to create. The resources above can be defined in any order. Terraform takes the relationships between resources into account when deciding the order of creating resources.

## Step 5: (Optional) Define Outputs
In a file `outputs.tf`, define values you'd like to log after creating the infrastructure. We'll output the Workspace and Deployment IDs:
```
output "terraform_workspace" {
description = "ID of the TF created workspace"
value = astro_workspace.my_first_tf_workspace.id
}

output "terraform_deployment" {
description = "ID of the TF created deployment"
value = astro_deployment.my_first_tf_deployment.id
}
```

## Step 6: Preview the Terraform changes
You should now have 3 files: `terraform.tf`, `main.tf` and `outputs.tf` (optional).
1. Run [`terraform plan`](https://developer.hashicorp.com/terraform/cli/commands/plan) to let Terraform create an execution plan and preview the infrastructure changes that Terraform will make. You should see the following text:
```
$ terraform plan

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

# astro_deployment.my_first_tf_deployment will be created
+ resource "astro_deployment" "my_first_tf_deployment" {
...
}

# astro_workspace.my_first_tf_workspace will be created
+ resource "astro_workspace" "my_first_tf_workspace" {
...
}

Plan: 2 to add, 0 to change, 0 to destroy.

Changes to Outputs:
+ terraform_deployment = (known after apply)
+ terraform_workspace = (known after apply)
```
2. Verify the generated plan contains the text `Plan: 2 to add, 0 to change, 0 to destroy.` This validates that the plan is to create two resources, which are the Workspace and Deployment as defined in `main.tf`.

## Step 7: Apply the Terraform Plan
Run `terraform apply` and select `yes` to execute the plan. This creates the Astro resources and will print their ids, as you defined in `outputs.tf`:
```
$ terraform apply

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
+ create

Terraform will perform the following actions:

...

Do you want to perform these actions?
Terraform will perform the actions described above.
Only 'yes' will be accepted to approve.

Enter a value: yes

astro_workspace.my_first_tf_workspace: Creating...
astro_workspace.my_first_tf_workspace: Creation complete after 0s [id=clxn24vhy0as101ohmeg9i72o]
astro_deployment.my_first_tf_deployment: Creating...
astro_deployment.my_first_tf_deployment: Creation complete after 1s [id=clxn24vs00afv01ot12ph6ejb]

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

Outputs:

terraform_deployment = "clxn24vs00afv01ot12ph6ejb"
terraform_workspace = "clxn24vhy0as101ohmeg9i72o"
```
The resources were created and will now be visible in Astro.

## Step 8: Clean Up Terraform-Created Resources
Run `terraform destroy` and select `yes`:
```
$ terraform destroy
astro_workspace.my_first_tf_workspace: Refreshing state...
[id=clxn24vhy0as101ohmeg9i72o]
astro_deployment.my_first_tf_deployment: Refreshing state...
[id=clxn24vs00afv01ot12ph6ejb]

Terraform used the selected providers to generate the following execution plan. Resource actions are indicated with the following symbols:
- destroy

Terraform will perform the following actions:

...

Plan: 0 to add, 0 to change, 2 to destroy.

Changes to Outputs:
- terraform_deployment = "clxn24vs00afv01ot12ph6ejb" -> null
- terraform_workspace = "clxn24vhy0as101ohmeg9i72o" -> null

Do you really want to destroy all resources?
Terraform will destroy all your managed infrastructure, as shown above.
There is no undo. Only 'yes' will be accepted to confirm.

Enter a value: yes

astro_deployment.my_first_tf_deployment: Destroying...
[id=clxn24vs00afv01ot12ph6ejb]
astro_deployment.my_first_tf_deployment: Destruction complete after 1s
astro_workspace.my_first_tf_workspace: Destroying...
[id=clxn24vhy0as101ohmeg9i72o]
astro_workspace.my_first_tf_workspace: Destruction complete after 0s

Destroy complete! Resources: 2 destroyed.
```
The output shows two destroyed resources which are the Workspace and Deployment that you first created.
132 changes: 132 additions & 0 deletions docs/guides/import-script.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
---
page_title: "Use Terraform Import Script to migrate existing resources"
---

# Use Import Script to migrate existing resources
The Astro Terraform Import Script is a tool designed to help you import existing Astro resources into your Terraform configuration.

In this guide, we will migrate an existing Workspace, API token and Team into Terraform using the Terraform Import Script.

### Import Script options
- `-resources`: Comma-separated list of resources to import. Accepted values are
`workspace`, `deployment`, `cluster`, `api_token`, `team`, `team_roles`,and `user_roles`. If not provided, all resources are imported.

-> Ensure you have the necessary permissions in your Astro organization to access the resources you're attempting to import.

- `-token`: API token to authenticate with the Astro platform. If not provided, the script will attempt to use the ASTRO_API_TOKEN environment variable.
- `-organizationId`: Organization ID to import resources from.
- `-runTerraformInit`: Run terraform init after generating the import configuration. Used for initializing the Terraform state in our GitHub Actions.
- `-help`: Display help information.


## Prerequisites
- An [Astro](https://www.astronomer.io/product/) organization with a Workspace, Team, and API token
- An initialized Terraform working directory

## Step 1: Download the Import Script
1. Download the `terraform-provider-astro-import-script` executable file from [releases](https://github.com/astronomer/terraform-provider-astro/releases) based on your OS and architecture. For this guide, the script will be `terraform-provider-astro-import-script_v0.1.3_darwin_arm64`.

## Step 2: Run the Import Script

-> Make sure you run `terraform init` before using the import script, or use the `-runTerraformInit` option when running the import script.

1. Authenticate with Astro by creating an [API token](https://www.astronomer.io/docs/astro/organization-api-tokens#create-an-organization-api-token) with the **organization owner** role and configure it as an `ASTRO_API_TOKEN` environment variable:
```
export ASTRO_API_TOKEN=&lt;your-api-token&gt;
```

2. If you are using a Unix-based systems, add execute permission to the script file:
```
chmod +x terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt;
```
3. Run the import script. Insert the script's version, your computer's operating system, and your computer's architecture for `<version-number>`, `<os>` and `<arc>`.

- On Unix-based systems:
```
./terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt; [options]
```
- On Windows:

```
.\terraform-provider-astro-import-script_&lt;version-number&gt;_&lt;os&gt;_&lt;arc&gt;.exe [options]
```

To import your existing Workspace, API token and Team, specify those resources with the `-resources` option. The other option you will need to specify is `-organizationId`:
```
./terraform-provider-astro-import-script_v0.1.3_darwin_arm64 -organizationId &lt;your-organization-id&gt; -resources api_token,team,workspace
```

You should see the following output:
```
Terraform Import Script Starting
Resources to import: [api_token team workspace]
Using organization ID: &lt;your-organization-id&gt
Terraform version 1.9.7 is installed and meets the minimum required version.
Importing teams for organization &lt;your-organization-id&gt
Importing API tokens for organization &lt;your-organization-id&gt
Importing workspaces for organization &lt;your-organization-id&gt
Importing Workspaces: [&lt;workspace-id&gt]
Successfully handled resource workspace
Importing API Tokens: [&lt;api_token-id&gt]
Successfully handled resource api_token
Importing Teams: [&lt;team-id&gt]
Successfully handled resource team
Successfully wrote import configuration to import.tf
Successfully deleted generated.tf
terraform.tfstate does not exist, no need to delete
astro_team.team_&lt;team-id&gt: Preparing import... [id=&lt;team-id&gt]
astro_api_token.api_token_cm2c9slvb08p601n3lb4jljps: Preparing import... [id=cm2c9slvb08p601n3lb4jljps]
astro_workspace.workspace_&lt;workspace-id&gt: Preparing import... [id=&lt;workspace-id&gt]
astro_workspace.workspace_&lt;workspace-id&gt: Refreshing state... [id=&lt;workspace-id&gt]
astro_api_token.api_token_&lt;api_token-id&gt: Refreshing state... [id=&lt;api_token-id&gt]
astro_team.team_&lt;team-id&gt: Refreshing state... [id=&lt;team-id&gt]

Terraform will perform the following actions:

...

Plan: 3 to import, 0 to add, 0 to change, 0 to destroy.

Terraform has generated configuration and written it to generated.tf. Please review the configuration and edit it as
necessary before adding it to version control.

Note: You didn't use the -out option to save this plan, so Terraform can't guarantee to take exactly these actions if you
run "terraform apply" now.
Import process completed. Summary:
Resource workspace processed successfully
Resource api_token processed successfully
Resource team processed successfully
```
-> If you import Deployments, they will not count torwards the `Plan: 3 to import, 0 to add, 0 to change, 0 to destroy` line of the output even when the Deployments are successfully imported.

## Step 3: Review output
The script will generate two main files:
- `import.tf`: Contains the Terraform import blocks for the specified resources.
- `generated.tf`: Contains the Terraform resource configurations for the imported resources.
The generated Terraform configurations may require some manual adjustment to match your specific requirements or to resolve any conflicts.

## Step 4: Extract and organize resources
The `generated.tf` file that is created by the import script will contain all of the specified resources in one file. It is recommended that you extract and modularize the resources so they are easily maintained and reusable. This is an example of a well structured Terraform project for managing Astro infrastructure:
```
terraform-astro-project/
├── environments/
│ ├── dev/
│ │ ├── main.tf # Root module for development
│ │ ├── variables.tf # Dev-specific variables
│ │ ├── outputs.tf # Dev-specific outputs
│ │ └── dev.tfvars # Variable values for development
│ ├── prod/
│ │ ├── main.tf # Root module for production
│ │ ├── variables.tf # Prod-specific variables
│ │ ├── outputs.tf # Prod-specific outputs
│ │ └── prod.tfvars # Variable values for production
├── modules/
│ ├── astro/
│ │ ├── main.tf # Entry point for the module
│ │ ├── workspace.tf # Defines Astro workspaces
│ │ ├── deployment.tf # Defines Astro Deployments
│ │ ├── users.tf # Defines user roles and access
│ │ ├── variables.tf # Variables used in the astro module
│ │ └── outputs.tf # Outputs from the astro module
└── cloud_provider.tf
```
24 changes: 14 additions & 10 deletions docs/index.md
Original file line number Diff line number Diff line change
@@ -1,17 +1,21 @@
---
# generated by https://github.com/hashicorp/terraform-plugin-docs
page_title: "astro Provider"
subcategory: ""
description: |-

page_title: "Astro Provider"
description:
Astro Provider for Terraform allows you to manage Astro resources from a Terraform script. The Provider uses the Astro API.
---

# astro Provider
# Astro Provider
[Astro](https://www.astronomer.io/product/) is a unified data platform built on Apache Airflow® that ensures data is delivered on time, securely, and accurately.
This is the official [Astro Terraform Provider](https://github.com/astronomer/terraform-provider-astro), built to allow you to automate, scale, and manage your Astro infrastructure.
Reduce manual effort, minimize errors, and create a more streamlined approach to managing data orchestration.

## Authentication
Create an [API token](https://www.astronomer.io/docs/astro/automation-authentication#step-1-create-an-api-token) with the minimum required [permissions](https://www.astronomer.io/docs/astro/user-permissions) to securely use Terraform, ensuring it only has access to the resources necessary.
You can create a [Deployment](https://www.astronomer.io/docs/astro/deployment-api-tokens), [Workspace](https://www.astronomer.io/docs/astro/workspace-api-tokens), or [Organizaton](https://www.astronomer.io/docs/astro/organization-api-tokens) API token.
An Organizaton token is the most flexible option to authenticate for high level changes accross multiple different resources.
Ensure the API token is configured as an environment variable, `ASTRO_API_TOKEN` when running Terraform commands.


## Example Usage

## Example usage
```terraform
provider "astro" {
organization_id = "cljzz64cc001n01mln1p12345"
Expand All @@ -28,4 +32,4 @@ provider "astro" {
### Optional

- `host` (String) API host to use for the provider. Default is `https://api.astronomer.io`
- `token` (String, Sensitive) Astro API Token. Can be set with an `ASTRO_API_TOKEN` env var.
- `token` (String, Sensitive) Astro API Token. Can be set with an `ASTRO_API_TOKEN` env var.
Loading