TerraformDevKit

Set of scripts to ease development and testing with Terraform.

The script collection includes support for:

• Managing AWS credentials
• Backing up the state from a failed Terraform execution
• Executing external commands
• Simple configuration management
• Simple reading and writing to AWS DynamoDB
• Multiplatform tools
• Making simple HTTP requests
• Retrying a block of code
• Terraform environment management
• Locally installing Terraform
• Filtering Terraform logging messages

Most of these scripts exist to provide support to a module development and testing environment for Terraform: TerraformModules. But, they might be useful for other purposes too.

Currently this repository is tightly coupled with AWS and has not been tested to work with other providers. We are actively working to change this and hope to have a more generic solution soon. If you would like to see support for your favourite cloud provider please have submit a pull request implementing support and we will be more than happy to merge your changes in.

Installation

Add this line to your application's Gemfile:

gem 'TerraformDevKit'

And then execute:

$ bundle

Or install it yourself as:

$ gem install TerraformDevKit

Usage

To use the library simply import it as:

require 'TerraformDevKit'

Managing Terraform Environments

TerraformDevKit provides a set of Rake tasks and Ruby scripts to ease the management of multiple Terraform environments. Three major environment types are supported: dev, test and prod.

There might be many development environments (dev), each one with its own name. Development environments use a local Terraform backend. They are intented to be used by developers while adding features to the infrastructure.

Testing (test) and production (prod) environment use a remote backend. Thus, the Terraform state file is not kept in the local disk, but on S3. This allows multiple developers to easily work on the same infrastructure instance. For safety reasons, operations that affect testing and production environments require manual user input. This is not the case for development environments.

TerraformDevKit expects templated versions (using Mustache) of the Terraform files. Such files might contain placeholders for several fields such as Environment, (AWS) Region or (AWS) Profile, among others. TerraformDevKit uses the template files to generate the final files that will be consumed by Terraform. As an example, for the production environment, the resulting files are placed in a directory named envs/prod.

Configuration Files

Configuration files must be placed in a directory named config. Each environment type requires a different configuration file. Thus, the following three files must be placed in the config directory:

• config-dev.yml
• config-test.yml
• config-prod.yml

The first one contains the configuration for all the development environments that are created. The other two contain the configuration for the test and production environments, respectively.

A sample configuration files is shown next:

terraform-version: 0.12.24
project-name: my super cool project
aws:
  profile: myprofile
  region: eu-west-1

The AWS profile must not be specified for test and production accounts, as users are required to manually type the profile name.

A Minimal Rakefile

ROOT_PATH = File.dirname(File.expand_path(__FILE__))

spec = Gem::Specification.find_by_name 'TerraformDevKit'
load "#{spec.gem_dir}/tasks/devkit.rake"

task :custom_test, [:env] do |_, args|
  # Add some tests here
end

Overrides

It's possible to override the location of your config files by setting the variable CONFIG_FILE in the top level Rakefile

# %s will be substituted with the environment name.
# File is exected to live in /c/path/to/root/config/config-dev.yml
CONFIG_FILE = File.join(ROOT_PATH, 'config', 'config-%s.yml')

Tasks and Hooks

TerraformDevKit provides a set of generic tasks to perform:

• prepare: prepares the environment
• plan: shows the plan to create the infrastructure
• apply: creates the infrastructure
• destroy: destroys the infrastructure
• clean: cleans the environment (after destroying the infrastructure)
• test: tests a local environment
• preflight: creates a temporary infrastructure and runs the test task

Additionally, TerraformDevKit allows users to define a set of hooks that will be called during the different steps required to complete the previous list of tasks. The following hooks are available:

• pre_apply: invoked before apply task runs
• post_apply: invoked after apply task runs
• pre_destroy: invoked before destroy task runs
• post_destroy: invoked after destroy task runs
• custom_prepare: invoked during the preparation process, before terraform is initialized and the environment folder is created
• custom_test: invoked during as part of the test task, right after apply completes.

Sample Terraform Templates

The following file (main.tf.mustache) contains the infrastructure configuration (a single S3 bucket) as well as information related to the AWS provider.

locals {
  env    = "{{Environment}}"
}

# See example below for how to configure a remote backend

provider "aws" {
  profile = "{{Profile}}"
  region  = "{{Region}}"
}

resource "aws_s3_bucket" "raw" {
  bucket = "foo-${local.env}"
  acl    = "private"

{{#LocalBackend}}
  force_destroy = true
{{/LocalBackend}}
{{^LocalBackend}}
  lifecycle {
    prevent_destroy = true
  }
{{/LocalBackend}}
}

The config file requires a project-name to be set. This project name is then use to generate the S3 bucket and dynamodb lock table required by terraform to mamage remote state. To use the remote state feature of TerraformDevKit you must add the following section to your main.tf.mustache file:

terraform {
  {{#LocalBackend}}
    backend "local" {}
  {{/LocalBackend}}
  {{^LocalBackend}}
    backend "s3" {
      bucket         = "{{ProjectName}}-{{Environment}}-state"
      key            = "{{ProjectAcronym}}-{{Environment}}.tfstate"
      dynamodb_table = "{{ProjectAcronym}}-{{Environment}}-lock-table"
      encrypt        = true
      profile        = "{{Profile}}"
      region         = "{{Region}}"
    }
  {{/LocalBackend}}
}

Injecting Additional Variables into Template Files

In addition to the default variables that are passed to Mustache when rendering a template file, users can provide additional variables. To do so, users must register a procedure that receives the environment as a parameter and returns a map with the extra variables and their values. An example is shown next:

TDK::TerraformConfigManager.register_extra_vars_proc(
  proc do
    { SumoLogicEndpoint: TDK::Configuration.get('sumologic')['endpoint'] }
  end
)

Additional Template Sources

TerraformDevKit looks for mustache templates in the root directory, and creates final versions of these files after injecting values such as the environment name or the project name.

In addition to doing this for the infrastructure description files (.tf files), TerraformDevKit allows users to specify additional template sources. This feature could be used, for instance, to version other types of files whose content depends on values such as the environment name.

The following example shows how to instruct TerraformDevKit to read templates from different source folders, and write the resulting files into their corresponding destination folders located within an environment folder (e.g., envs/<env-name>/):

terraform-version: 0.12.24
project-name: my super cool project
aws:
  profile: myprofile
  region: eu-west-1
template-dirs:
  template-dest-folder: ../template-src-folder
  template-dest-folder: ../another-template-src-folder
  another-template-dest-folder: yet-another-template-src-folder

Copying Files to the Environment Folder

Sometimes it may be desirable to copy files that exist in a given folder into the environment folder. The copy-files field makes this possible. See the following configuration for an example:

terraform-version: 0.12.24
project-name: my super cool project
copy-files:
  dest-folder: ../src-folder
  another-dest-folder: ../../another-src-folder
  dest-file: src-file

Updating Modules

Terraform will get the necessary modules every time a new environment is created. Once the modules are cached, there is generally no need to keep updating the modules each time Terraform is executed. When using a module repository it is possible to select a specific version to use (as shown here). In such a case, Terraform will automatically update the modules whenever the version number is changed.

When using local modules (e.g., during development process) it might be desirable to update the modules every time Terraform runs. This can be achieved by setting the environment variable TF_DEVKIT_UPDATE_MODULES to true.

Development

After checking out the repo, run bin/setup to install dependencies. Then, run rake spec to run the tests. You can also run bin/console for an interactive prompt that will allow you to experiment.

To install this gem onto your local machine, run bundle exec rake install. To release a new version, update the version number in version.rb, and then run bundle exec rake release, which will create a git tag for the version, push git commits and tags, and push the .gem file to rubygems.org.

Contributing

Bug reports and pull requests are welcome on GitHub at https://github.com/vistaprint/TerraformDevKit.

License

The gem is available as open source under the terms of the Apache License, Version 2.0.