Integrate Jenkins, Hashicorp Vault, and PowerShell

Summary: Passwords, Secrets, and Credentials, stored in a Hashicorp Vault server, can easily be leveraged by Jenkins Projects (including projects that leverage PowerShell for the automation – or pure Microsoft shops). There is a common tension between automation and security and this example will show how they can co-exist.

The following steps are used to enable this automation:

  • Save a Vault access token as a Jenkins Credential
  • Bind the Jenkins Credential to a Jenkins Project
  • Access the Jenkins Secret as an environment variable from PowerShell
  • Run a Jenkins Project (PowerShell) that loads all of the Vault Secrets for the project

When a credential is stored in Jenkins, it is encrypted and the credential secret value cannot be viewed after the fact, outside of a Jenkins project. When the credential is bound to a Jenkins project, it is loaded as an Environment Variable when the project is executed and can be accessed by the automation (PowerShell) in the manner. If the credential or secret is exposed in the StdOut of the automation, Jenkins will mask the credential value when it logs the output (see below).

Step 1: Add the Credential (Vault Secret)

From the Jenkins home, select Credentials, hover over the down arrow next to the domain “(global)”, and select “Add credentials”.

Jenkins-Add-Creds

Step 2: Add the Credential (Vault Secret)

Add the credential as a “Secret text” item.

Jenkins-Add-Creds-2

Step 3: Bind the Credential to the Project

Bind the credential (Vault Secret) to the Jenkins Project.

Jenkns-Bind-Secret

Step 4: Reference the Credential in PowerShell

Reference the Jenkins Secret via an environmental variable within the PowerShell automation.

 

Jenkins-PowerShell

The following PowerShell script is an example which will download and list all Vault Secrets within a particular path. Of course displaying secrets used during automation is not advisable, but serve as an example and launching point for using them in code.

Pull-Vault.ps1 (example):

# Version:: 0.1.5 (3/16/2017)
# Script Description:: Pulls Vault secrets into environmental variables.
#
# Author(s):: Otto Helweg
#
param($token,$vaultSvr,$path)

# Display help
if ($Args -match "-\?|--\?|-help|--help|/\?|/help") {
  Write-Host "Usage: Pull-Vault.ps1"
  Write-Host "     -path [path to credentials]             Path the list of credentials."
  Write-Host "     -token [Vault auth token]               Authentication token for accesing the Vault server."
  Write-Host "     -vaultSvr [Vault server name or IP]     Vault server name or IP address."
  Write-Host ""
  Write-Host "Examples:"
  Write-Host "     Pull-Vault.ps1 -token '770da5b6-eff1-6fd6-f074-1e2604987340'"
  Write-Host "     Pull-Vault.ps1 -token '770da5b6-eff1-6fd6-f074-1e2604987340' -vaultSvr '10.102.76.4'"
  Write-Host ""
  exit
}

if (!($env:VAULT_TOKEN) -or !($env:VAULT_ADDR)) {
  if (!($token)) {
    $token = Read-Host -Prompt "Enter Token for Vault authentication" -AsSecureString
    $token = (New-Object PSCredential "token",$token).GetNetworkCredential().Password
  }

  if (!($vaultSvr)) {
    $vaultSvr = Read-Host -Prompt "Enter Vault Server"
  }

  $env:VAULT_ADDR = "http://$($vaultSvr):8200"
  $env:VAULT_TOKEN = $token
}

if (!($path)) {
  $path = Read-Host -Prompt "Enter Secrets Path"
}

$keys = vault list -format=json $path | ConvertFrom-Json

foreach ($key in $keys) {
  $vaultKey = "TF_VAR_$key"
  $value = vault read -format=json "$($path)/$($key)" | ConvertFrom-Json
  if ($Args -contains "-debug") {
    Write-Host "  $($path)/$($key) : $($value.data.value)"
  }
  Write-Host "Loading env var: $vaultKey"
  Set-Item -path "env:$vaultKey" -value "$($value.data.value)"
}

Note: The output from the Jenkins Project will mask out any output that matches the Jenkins Secret.

...
VAULT_ADDR http://10.10.10.10:8200 
VAULT_ROOT_TOKEN **** 
VAULT_TOKEN **** 
windir C:\Windows 
WINSW_EXECUTABLE C:\Program Files (x86)
...

Enjoy!

Using AWS SSM With Windows Instances

Summary: Late 2015, AWS introduced a new feature called SSM (Simple System Manager) which lets you remotely execute commands on Windows (and Linux) server instances within AWS EC2. Unlike Windows Remote Management, SSM leverages the EC2 infrastructure to directly interact with the server instance, bypassing the need for WinRM ports to be opened up. In addition, SSM commands are interacting with the EC2Config service running on the server instance.

SSM supports several methods on the remote instance including running PowerShell commands as well as a very powerful Windows Update method (which also manages rebooting the server instance). Here’s a list of the available Windows methods in SSM:

  • AWS-JoinDirectoryServiceDomain: join an AWS Directory
  • AWS-RunPowerShellScript: run PowerShell commands or scripts
  • AWS-UpdateEC2Config: update the EC2Config service
  • AWS-ConfigureWindowsUpdate: configure Windows Update settings
  • AWS-InstallApplication: install, repair, or uninstall software using an MSI package
  • AWS-InstallPowerShellModule: install PowerShell modules
  • AWS-ConfigureCloudWatch: configure Amazon CloudWatch Logs to monitor applications and systems
  • AWS-ListWindowsInventory: collect information about an EC2 instance running in Windows
  • AWS-FindWindowsUpdates: scan an instance and determines which updates are missing
  • AWS-InstallMissingWindowsUpdates: install missing updates on your EC2 instance
  • AWS-InstallSpecificWindowsUpdates: install one or more specific updates

Note: SSM commands are run from the Local System account on the EC2 server instance, meaning they are run as Administrator.

The following examples show how to leverage SSM via the AWS CLI utility. AWS CLI must first be installed and configured with the proper credentials for these examples to work. These commands can be run from either a CMD or PowerShell prompt.

Example 1 – Run a PowerShell command with SSM: This demonstrates using PowerShell to modify a firewall rule using SSM on an EC2 instance. Where using User-Data can be used to run PowerShell commands when EC2 creates instances, SSM can be run anytime after the instance is running.

aws ssm send-command --instance-ids "i-12345d8d" --document-name "AWS-RunPowerShellScript" --comment "Update Firewall Rule" --parameters commands="Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any"

Example 2 – Install all missing updates: This is a very powerful method in SSM where all missing updates can be applied to an EC2 instance with a single command. This method also manages rebooting the instance after the updates are installed, if necessary.

aws ssm send-command --instance-ids "i-12345a86" --document-name "AWS-InstallMissingWindowsUpdates" --comment "Install Windows Upates" --parameters UpdateLevel="All"

Note: All SSM PowerShell commands that are run on an instance are saved in ‘C:\programdata\Amazon\Ec2Config\Downloads\aws_psModule’. This can be useful for troubleshooting commands or should be considered if sensitive information is used within SSM PowerShell commands.

Once an SSM command is executed, the job details are passed back in JSON to allow for monitoring the job state. This allows for automation to query the job status and apply logic for further action.

For example, the job details can be assigned to a PowerShell variable as follows (PowerShell v.4+ is required when using the ConvertFrom-Json cmdlet):

$ssmJob = (aws ssm send-command --instance-ids "i-12345d8d" --document-name "AWS-RunPowerShellScript" --comment "Update Firewall Rule" --parameters commands="Set-NetFirewallRule -Name WINRM-HTTP-In-TCP-PUBLIC -RemoteAddress Any") | ConvertFrom-JSON

The details of the job can be viewed by inspecting the $ssmJob object as follows:

$ssmJob.Command

You can query for the status of an SSM job using the following example:

$ssmJobStatus = (aws ssm list-command-invocations --command-id $ssmJob.Command.CommandId) | ConvertFrom-Json
$ssmJobStatus.CommandInvocations.Status

Enjoy!

 

CloudStack API PowerShell Example

Summary: This script uses a simple command, sent via PowerShell and REST to a CloudStack API, in order to list all visible Virtual Machines as well as their current state (‘Running’, ‘Stopped’, etc.).

CloudStack is an Apache Open Source project for managing a cloud service (much like OpenStack, but it’s been around a LOT longer). CloudStack also has an impressive list of customers (check out their Wikipedia post above). Like most cloud services, CloudStack has an API (REST) for programmatically interacting with the service.

Note: A valid account’s API public key and secret are required for this script.

This API authenticates requests by using an account’s public key and secret to create a signature for the API request. In order the create the signature, the command being signed must be broken down into key/value pairs, sorted, reassembled for signing, then converted to lower case.

On-going problems: This script mostly works. But it (or the API) has problems with certain command strings. And PowerShell doesn’t seem to like the API’s output in JSON (so this script uses the default XML). If you find a solution, please reach out.

Datapipe’s documentation of their implementation of the CloudStack API can be found here: https://docs.cloud.datapipe.com/developers/api/developers-guide

Here is a sample list of command strings that leverage the listVirtualMachines method (your mileage may vary):

command=listVirtualMachines
command=listVirtualMachines&state=Running
command=listVirtualMachines&account=General
command=listVirtualMachines&zoneid=13
command=listVirtualMachines&groupid=21&account=General
command=listVirtualMachines&groupid=21&account=General&state=Running
command=listVirtualMachines&state=Stopped&response=json
command=listVirtualMachines&id=3a57b2d3-b95a-4892-903a-34f4662ed475&response=json
command=listVirtualMachines&id=3a57b2d3-b95a-4892-903a-34f4662ed475
command=listVirtualMachines&state=Running&response=json

Note: This script requires PowerShell v.3+ due to the use of the Invoke-RestMethod cmdlets.

openstackApiExample.ps1

#
# Name:: openstackApiExample.ps1
# Version:: 0.1.0 (6/27/2016)
# Script Description:: Queries the Datapipe Cloud (Cloud-Stack) REST API
#
# API Documentation:: https://docs.cloud.datapipe.com/developers/api/developers-guide
#
# Author(s):: Otto Helweg
#
# PowerShell v.3+ is required for the Invoke-RestMethod cmdlet
#
# Parameters: -k = apikey, -s = secret (both are required)
# Example: .\apiExample.ps1 -k "F2rrzJiluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJ" -s "iluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJiluwK39LpD6PvyF2rrzJiluwK3"

param($k,$s)

$uri=@{}
$baseUri = "https://cloud.datapipe.com/api/compute/v1?"
$command = "command=listVirtualMachines"
# The following is a command string that should work, but doesn't
# $command = "command=listVirtualMachines&state=Running"
$uri["apikey"] = $k
$secret = $s

# Build the Command String for getting an authorization signature
# First extract all key/value pairs in the command into the uri hash
$subCommand = $command.split("&")
foreach ($item in $subCommand | Sort-Object) {
  if ($item -like "*=*") {
    $items = $item.Split("=")
    $uri[$items[0]] = $items[1]
  } else {
    $uri[$item] = ""
  }
}

# Build the signing String by sorting the command key/values then make lowercase for signing
$signString = ""
foreach ($key in $uri.Keys | Sort-Object) {
  if ($uri[$key]) {
    $signString = $signString + $key + "=" + $uri[$key] + "&"
  } else {
    $signString = $signString + $key + "&"
  }
}
$signString = $signString.ToLower()
$signString = $signString.TrimEnd("&")

# Get the HMAC SHA-1 signature for the specific Command String
$hmacSha = New-Object System.Security.Cryptography.HMACSHA1
$hmacSha.key = [Text.Encoding]::ASCII.GetBytes($secret)
$signature = $hmacSha.ComputeHash([Text.Encoding]::ASCII.GetBytes($signString))
$signature = [Convert]::ToBase64String($signature)

# Build the signed REST URI
$newUri = $baseUri + $command + "&apiKey=" + $uri["apikey"] + "&signature=" + $signature
Write-Host "URI: $newUri"
Write-Host "signString: $signString"
Write-Host "Signature: $signature"

# Query for a list of all VMs
Write-Host "Querying the Cloud-Stack API..."
[xml]$vmList = Invoke-RestMethod -Method GET -Uri $newUri -ContentType "application/xml"

# List all VMs visible by the account
Write-Host ""
Write-Host "Virtual Machines:"
foreach ($vm in $vmList.listvirtualmachinesresponse.virtualmachine) {
  if ($vm.state -eq "Stopped") {
    $color = "yellow"
  } else {
    $color = "white"
  }
  Write-Host -ForegroundColor $color "$($vm.displayname) - $($vm.state)"
}

 

Enjoy!

 

Use PowerShell to Refresh CenturyLink Cloud VM Snapshots

Summary: VMs created in the CenturyLink Cloud, can have a snapshot (only 1 per VM), and that snapshot has a maximum life of 10 days. Therefore if it’s necessary to maintain a perpetual snapshot (like for test VMs that have a baseline configuration), VMs need to have their snapshots routinely refreshed. The following PowerShell script leverages the powerful CenturyLink Cloud REST v2 API to automate this process. This script relies on the presence of a ‘bearer token’ for authentication into the CLC API. Details on how to create this ‘bearer token’ can be found in this blog post (as well as details on how to discover your Group ID). The complete reference to CenturyLink’s Cloud REST API can be found here: https://www.ctl.io/api-docs/v2/

This script basically leverages CLC APIs to restore, delete, and create a VM snapshot. After each API is called, the script will wait until the task is complete, by querying the status of the task requested.

Note: This script sample requires PowerShell v4+

Refresh-Snapshot.ps1

#
# Name:: Refresh-Snapshot.ps1
# Version:: 0.1.2 (3/5/2016)
# Script Description:: Refreshes a server's snapshot by restoring the snapshot, deleting it, then creating a new one.
#
# API Documentation:: https://www.ctl.io/api-docs/v2/
#
# Author(s):: Otto Helweg
#

param($s)

# Display help
if (($Args -match "-\?|--\?|-help|--help|/\?|/help") -or !($s)) {
  Write-Host "Usage: Refresh-Snapshot.ps1"
  Write-Host "     -s [server name]       (Required) Server's name according to Centurylink Cloud"
  Write-Host "     -r                     Revert VM to existing snapshot before refreshing"
  Write-Host ""
  Write-Host "EXAMPLES:"
  Write-Host "     Refresh-Snapshot.ps1 -s CA3TESTTEST01 -r"
  Write-Host ""
  exit
}

$goodStatus = @("notStarted","executing","succeeded","resumed")

# PowerShell v4 is required for the REST cmdlets
if (!($psversiontable.PSVersion.Major -ge 4)) {
  Write-Host -ForegroundColor "red" "Requires PowerShell v.4 or greater. Please install the Windows Management Framework 4 or above."
  exit
}

# Check to make sure the Bearer Token is less than 2 weeks old
if (!(Test-Path .\bearerToken.txt)) {
  Write-Host -ForegroundColor Red "Error: Bearer Token file is missing. Run Save-BearerToken.ps1"
  exit
} else {
  $fileInfo = dir .\bearerToken.txt
  if ($fileInfo.LastWriteTime -lt (Get-Date).AddDays(-11)) {
    Write-Host -ForegroundColor Yellow "Warning: Bearer Token file is almost out of date. Run Save-BearerToken.ps1"
  }
  if ($fileInfo.LastWriteTime -lt (Get-Date).AddDays(-13)) {
    Write-Host -ForegroundColor Red "Error: Bearer Token file is out of date. Run Save-BearerToken.ps1"
    exit
  }
}

$bearerTokenInput = Get-Content ".\bearerToken.txt"
$accountAlias = "SXSW"
$bearerToken = " Bearer " + $bearerTokenInput
$header = @{}
$header["Authorization"] = $bearerToken
$serverName = $s

Write-Host -ForegroundColor Green "Getting server properties for $serverName..."
$requestUri = "https://api.ctl.io/v2/servers/$accountAlias/$serverName"
$serverProperties = Invoke-RestMethod -Method GET -Headers $header -Uri $requestUri -ContentType "application/json"

if (!$serverProperties) {
  Write-Host -ForegroundColor Red "Error: $serverName does not appear to exist!"
  exit
}

if (($Args -contains "-r") -and $serverProperties.details.snapshots) {
  Write-Host -ForegroundColor Green "Restoring snapshot for $serverName..."
  $startTime = Get-Date
  $requestUri = "https://api.ctl.io$($serverProperties.details.snapshots.links.href[0])/restore"
  $restoreResult = Invoke-RestMethod -Method POST -Headers $header -Uri $requestUri -ContentType "application/json"
  $statusUri = "https://api.ctl.io/v2/operations/$accountAlias/status/$($restoreResult.id)"
  $continue = $true
  Write-Host "Waiting for snapshot restore to complete..."
  while ($continue) {
    $statusResult = Invoke-RestMethod -Method GET -Headers $header -Uri $statusUri -ContentType "application/json"
    [int]$elapsedSeconds = ($(Get-Date) - $startTime).TotalSeconds
    Write-Host "   [$elapsedSeconds seconds] $($statusResult.status)"
    if ($statusResult.status -eq "succeeded") {
      $continue = $false
    }
    if ($statusResult.status -notin $goodStatus) {
      Write-Host -ForegroundColor Red "Error: Restoring snapshot for $serverName failed!"
      exit
    }
    Sleep 2
  }
}

if ($serverProperties.details.snapshots) {
  Write-Host -ForegroundColor Green "Deleting snapshot for $serverName..."
  $startTime = Get-Date
  $requestUri = "https://api.ctl.io$($serverProperties.details.snapshots.links.href[0])"
  $restoreResult = Invoke-RestMethod -Method DELETE -Headers $header -Uri $requestUri -ContentType "application/json"
  $statusUri = "https://api.ctl.io/v2/operations/$accountAlias/status/$($restoreResult.id)"
  $continue = $true
  Write-Host "Waiting for snapshot delete to complete..."
  while ($continue) {
    $statusResult = Invoke-RestMethod -Method GET -Headers $header -Uri $statusUri -ContentType "application/json"
    [int]$elapsedSeconds = ($(Get-Date) - $startTime).TotalSeconds
    Write-Host "   [$elapsedSeconds seconds] $($statusResult.status)"
    if ($statusResult.status -eq "succeeded") {
      $continue = $false
    }
    if ($statusResult.status -notin $goodStatus) {
      Write-Host -ForegroundColor Red "Error: Deleting snapshot for $serverName failed!"
      exit
    }
    Sleep 2
  }
}

Write-Host -ForegroundColor Green "Creating snapshot for $serverName..."
$startTime = Get-Date
$createBody = "{
  ""snapshotExpirationDays"":""10"",
  ""serverIds"":[
      ""$serverName""
    ]
}"

$requestUri = "https://api.ctl.io/v2/operations/$accountAlias/servers/createSnapshot"
$restoreResult = Invoke-RestMethod -Method POST -Headers $header -Body $createBody -Uri $requestUri -ContentType "application/json"
$statusUri = "https://api.ctl.io/v2/operations/$accountAlias/status/$($restoreResult.links.id)"

$continue = $true
Write-Host "Waiting for snapshot creation to complete..."
while ($continue) {
  $statusResult = Invoke-RestMethod -Method GET -Headers $header -Uri $statusUri -ContentType "application/json"
  [int]$elapsedSeconds = ($(Get-Date) - $startTime).TotalSeconds
  Write-Host "   [$elapsedSeconds seconds] $($statusResult.status)"
  if ($statusResult.status -eq "succeeded") {
    $continue = $false
  }
  if ($statusResult.status -notin $goodStatus) {
    Write-Host -ForegroundColor Red "Error: Creating snapshot for $serverName failed!"
    exit
  }
  Sleep 2
}

 

Enjoy!