Summary: In this case, ‘Atomic’ refers to small components rather than nuclear power. This post will discuss the strategy of using many small Terraform plans to build an environment, rather than one large plan. Although this creates a Terraform plan management burden, its primary goal is to reduce Terraform’s blast radius (the amount of damage Terraform can do if its actions are unexpected – granted due typically to user error). In order to try to maintain the same level of automation, we will use PowerShell to automate Terraform provisioning.

When Terraform runs, it assumes that it knows the desired state of an environment and will make any changes necessary to get to this known state. This can be problematic when an environment is changed outside of Terraform, or Terraform’s own state files are not up to date, or changed outside of Terraform (e.g. misconfigured remote state).

We will use PowerShell to perform the following functions:

• Run multiple Terraform plans with a single command.
• Run multiple Terraform plans in parallel using PowerShell Jobs.
• Verify most recent modules are referenced for the Terraform plan.
• Verify Terraform remote state is properly configured before taking action.
• Verify Terraform will not ‘change’ or ‘destroy’ anything when provisioning a new instance (unless overridden).

Our environment will contain a single Terraform plan in a sub-directory that represents a single instance. Therefore if an environment has 5 servers and 1 network, there will be 6 sub-directories containing 6 plans.

Note: A future blog post will show how to use PowerShell to programmatically generate all Terraform plans for an environment in order to further reduce the plan management burden for an Atomic Terraform implementation.

Example 1 – Automate Multiple Terraform ‘Applys’: See the above list of actions that PowerShell will take before executing a Terraform Apply.

# Version:: 0.1.5 (10/21/16)
# Script Description:: This script is a wrapper for running multiple terraform commands in the background using PowerShell jobs.
#
# Author(s):: Otto Helweg
#

param($limit)

# Display help
if (($Args -match "-\?|--\?|-help|--help|/\?|/help") -or (!($limit))) {
  Write-Host "Usage: Terraform-Apply-All.ps1"
  Write-Host "     -limit [instance(s)/none]   (required) Specify a specific instance (e.g. VM), or 'none' to run against all servers"
  Write-Host "     -force                      Force the Apply even if elements will be changed or destoryed"
  Write-Host ""
  Write-Host "Examples:"
  Write-Host "     Terraform-Apply-All.ps1 -limit network01,server01 -force"
  Write-Host "     Terraform-Apply-All.ps1 -limit none"
  Write-Host ""
  exit
}

if ($limit -eq "none") {
  $limit = $false
} elseif ($limit) {
  $limit = $limit.Split(",")
}

$instances = (Get-ChildItem -Directory -Exclude "modules").Name
$currentDir = (Get-Item -Path ".\" -Verbose).FullName

$startTime = Get-Date
$jobIds = @{}
foreach ($instance in $instances) {
  $planChecked = $false
  $remoteChecked = $false
  if (($limit -and ($instance  -in $limit)) -or (!($limit))) {
    Write-Host "Working in Terraform directory $instance"
    if (Test-Path $instance) {
      # Check to make sure remote state file is configured
      Set-Location "$instance"
      terraform get
      $remoteOutput = terraform remote pull 2>&1
      Set-Location "..\"
      if ($remoteOutput -notlike "*not enabled*") {
        $remoteChecked = $true
      } else {
        Write-Host -ForegroundColor RED "Error: Remote state file pointer is not configured."
      }

      # Check to make sure nothing will be changed or destroyed (unless forced)
      if ($remoteChecked) {
        Set-Location "$instance"
        $planOutput = terraform plan
        Set-Location "..\"
        if (($planOutput -like "* 0 to change*") -and ($planOutput -like "* 0 to destroy*")) {
          $planChecked = $true
        } else {
          if ($Args -contains "-force") {
            $planChecked = $true
            Write-Host -ForegroundColor YELLOW "Warning: Terraform Apply will change or destroy existing elements. Force detected."
          } else {
            Write-Host -ForegroundColor YELLOW "Warning: Terraform Apply will change or destroy existing elements. Skipping $instance"
          }
        }
      }
      if ($planChecked -and $remoteChecked) {
        $jobInfo = Start-Job -ScriptBlock { Set-Location "$Args"; terraform apply -no-color } -ArgumentList "$currentDir\$instance"
        $jobIds["$instance"] = $jobInfo.Id
        Write-Host "  Creating job $($jobInfo.Id)"
      }
    } else {
      Write-Host -ForegroundColor RED "Error: $instance plan does not appear to exist, consider running 'Build-TerraformEnv.ps1' in ..\setup."
    }
  }
}

$waiting = $true
while ($waiting) {
  $elapsedTime = (Get-Date) - $startTime
  $allJobsDone = $true
  foreach ($instanceKey in $jobIds.Keys) {
    $jobState = (Get-Job -Id $jobIds[$instanceKey]).State
    Write-Host "($($elapsedTime.TotalSeconds) sec) Job $serverKey - $($jobIds[$instanceKey]) status: $jobState"
    if ($jobState -eq "Running") {
      $allJobsDone = $false
    }
  }
  if ($allJobsDone) {
    $waiting = $false
  } else {
    Sleep 10
  }
}

$jobState = @{}
foreach ($instanceKey in $jobIds.Keys) {
  $jobOutput = Receive-Job -Id $jobIds[$instanceKey]
  if ($jobOutput -like "*Apply complete!*") {
    Write-Host -ForegroundColor GREEN "Job $serverKey - $($jobIds[$instanceKey]) output:"
    $jobState[$instanceKey] = "Succeeded"
  } else {
    Write-Host -ForegroundColor RED "Error: Job $serverKey - $($jobIds[$instanceKey]) failed. Output:"
    $jobState[$instanceKey] = "Failed"
  }
  Write-Output $jobOutput
}

Write-Host -ForegroundColor GREEN "Job status summary:"
foreach ($instanceKey in $jobState.Keys) {
  Write-Host "$instanceKey - $($jobState[$instanceKey])"
}

    if ($remoteOutput -notlike "*not enabled*") {
      $jobInfo = Start-Job -ScriptBlock { Set-Location "$Args"; terraform destroy -force -no-color } -ArgumentList "$currentDir\$instance"
      $jobIds["$instance"] = $jobInfo.Id
      Write-Host "  Creating job $($jobInfo.Id)"
    } else {
      Write-Host -ForegroundColor RED "Error: Remote state file pointer is not configured."
    }

And replace the success check with:

  if ($jobOutput -like "*Destroy complete!*") {
    Write-Host -ForegroundColor GREEN "Job $instanceKey - $($jobIds[$instanceKey]) output:"
    $jobState[$instanceKey] = "Succeeded"
  } else {
    Write-Host -ForegroundColor RED "Error: Job $instanceKey - $($jobIds[$instanceKey]) failed. Output:"
    $jobState[$instanceKey] = "Failed"
  }

Enjoy!