A Golden AMI factory, from nothing to GovCloud

The Virtual Appliance Builder: one pipeline that builds the image, proves upgrades still work, and lands it in every commercial region customers need plus both AWS GovCloud regions.

client
Anchore
role
Principal DevOps Engineer
when
2024 to present
drawn by
Jesse Tamburino
BUILD ACCOUNTImage buildGolden AMIbuilt from scratch01Validatevalidation frameworkparallel upgrades02upgrade runupgrade runupgrade runupgrades run side by side,not one after anotherRELEASE ACCOUNTCopyto release accountownership transfer03Sharecommercial regionsboth GovCloud regions04COMMERCIALregionregionregionGOVCLOUD PARTITIONus-gov-west-1us-gov-east-1SHARED TERRAFORM MODULE LIBRARY · AWS + CLOUDFLARE05providerappDNStunnelIP accessEKS
Build, validate, promote, distribute. The critical path is lit.Hover a step below to find it on the sheet. Scroll sideways on small screens.
2GovCloud regions
1release account
0manual copy steps

the problem

Shipping a virtual appliance means shipping an image customers trust, in every region they run, including two AWS GovCloud regions, and the upgrade from the last version must not break. There was no pipeline for it. I built one from scratch.

Constraints

  • GovCloud is its own partition with its own accounts. Anything that works in commercial has to be done again there, deliberately.
  • Customers upgrade. An image that boots clean on a fresh install proves very little.
  • The image customers launch should not live in the account where builds churn.
  • Every step a human has to remember is a step that eventually gets skipped.

Stack

AWS AMIAWS GovCloudMulti-account AWSTerraformCloudflareEKS

how it is built

  1. 01

    Build

    An end-to-end Golden AMI factory: the Virtual Appliance Builder pipeline, built from scratch.

  2. 02

    Validate

    An AMI validation framework that runs upgrade paths in parallel, so the gate is thorough without being slow enough to tempt anyone to skip it.

  3. 03

    Promote

    Copy to a dedicated release account with ownership transfer. The published artifact belongs to release, not to the build account.

  4. 04

    Distribute

    Multi-region sharing across commercial regions and into both GovCloud regions, as pipeline stages rather than a runbook.

  5. 05

    Codify

    A shared Terraform module library across AWS and Cloudflare: provider, app, DNS, tunnel, IP access policy and EKS modules.

calls I made

Why it looks like this.

01

Promote by copy, not by share

Copying into a release account and transferring ownership means the build account can be rebuilt or torn down without touching anything a customer launched.

02

Test the upgrade, not the boot

The failure customers feel is the upgrade. Running upgrade paths in parallel buys that coverage at a cost the pipeline can carry on every build.

03

GovCloud is a target, not a follow-up

Both GovCloud regions are stages in the same pipeline. Nobody hand-carries an image across a partition boundary.

Want the longer version, with the parts that went wrong?

Ask me about it