The Colo-to-Cloud Migration That Nobody Plans For — Castles of Light Blog
The hardest infrastructure pattern in communications: real-time media that spans bare-metal colo for the hot path and cloud for everything else. The CI/CD pipeline that deploys to both simultaneously without breaking either one.
TLDR
When Zoom says "colocation data centers, AWS, OCI, and other cloud providers" in the same breath as "real-time media systems" and "strict latency SLAs," they are describing the hardest infrastructure pattern in communications: a hot path that lives on bare-metal colo because cloud-only latency breaks the media stream, surrounded by everything else that moved to AWS and OCI years ago. The migration nobody plans for is the gap between those two worlds, and the CI/CD pipeline that has to deploy to both simultaneously without breaking either one.
The Pattern Nobody Names
I have seen this architecture three times. At Spotify, the audio delivery path spanned CDN edge nodes and bare-metal transcoding clusters while the control plane lived in Google Cloud. At Google Cloud Professional Services, I architected multi-cloud deployments for media clients where the encode pipeline stayed on colo for latency and the orchestration layer lived in AWS. At Raytheon, the defense-grade real-time data distribution system spanned air-gapped colo and classified cloud regions.
The pattern is always the same. There is a hot path that cannot tolerate more than a few milliseconds of jitter, and there is everything else. The hot path stays on bare metal in a colocation facility because the physics of packet round-trip time to a cloud region add latency that breaks the real-time constraint. Everything else, the control plane, the observability stack, the CI/CD orchestration, the config management, the service mesh, lives in AWS or OCI or both.
The JD calls this "hybrid infrastructures spanning colocation data centers, AWS, OCI, and other cloud providers." It is the most under-hired role in infrastructure because the skill set is a Venn diagram with almost no overlap. Cloud-native engineers do not know colo. Colocation engineers do not know Kubernetes. Almost nobody has done both at production scale.
The Architecture
graph TB
subgraph "Colocation (Hot Path)"
COLO_MMR["MMR Media Mixer
Bare-metal K8s
Sub-5ms inter-frame"]
COLO_WEBRTC["WebRTC SFU
Bare-metal
DPDK bonded NICs"]
COLO_LB["Hardware LB
L3/L4 packet capture
BGP routing"]
COLO_DNS["Colo DNS
Anycast
Hardware resolver"]
end
subgraph "AWS (Control Plane)"
AWS_API["API Gateway
Lambda + ALB
Control orchestration"]
AWS_K8S["EKS Cluster
Terraform-managed
30+ microservices"]
AWS_OBS["Observability
CloudWatch + Datadog
SLI aggregation"]
end
subgraph "OCI (Data Plane)"
OCI_STREAM["Media Recording
Object Storage
MMR stream archive"]
OCI_TF["Terraform State
OCI Backend
Drift detection"]
end
subgraph "CI/CD"
GIT["GitOps Repo
Unified deploy spec
Colo + Cloud"]
TF["Terraform
AWS + OCI modules
Ansible for colo"]
end
COLO_LB --> COLO_MMR
COLO_MMR --> COLO_WEBRTC
COLO_DNS --> COLO_LB
AWS_API --> AWS_K8S
AWS_K8S -.controls.-> COLO_MMR
COLO_WEBRTC -.streams to.-> OCI_STREAM
GIT --> TF
TF --> AWS_K8S
TF -.provision.-> COLO_MMR
TF -.provision.-> OCI_STREAM
AWS_OBS -.scrapes.-> COLO_MMR
AWS_OBS -.scrapes.-> COLO_WEBRTC
style COLO_MMR fill:#f96,stroke:#333,stroke-width:3px
style COLO_WEBRTC fill:#f96,stroke:#333,stroke-width:3px
style GIT fill:#6cf,stroke:#333,stroke-width:2px
The hot path is in orange. Everything it touches, the MMR media mixer, the WebRTC SFU, the hardware load balancer with its BGP routing and L3/L4 packet capture, the anycast DNS resolver, all of it lives on bare metal in a colocation facility within single-digit milliseconds of the media endpoint. The control plane in AWS orchestrates the hot path but does not sit in its data path. OCI holds the recording archive and the Terraform state backend. The CI/CD pipeline has to deploy to all three.
What Most Teams Ship First (The Naive Approach)
Most teams start with a cloud-only deployment and add colo later. The first attempt at unifying the deploy looks like this:
# Naive: deploy everything from one pipeline, no awareness of target
#!/bin/bash
set -euo pipefail
# Deploy AWS services
aws eks update-kubeconfig --name prod-cluster
kubectl apply -f k8s/ --recursive
terraform apply -target=module.aws_control_plane
# Deploy colo services (this is where it breaks)
ssh colo-prod-01 "docker pull registry/mmr-mixer:latest && docker run -d mmr-mixer"
ssh colo-prod-02 "docker pull registry/webrtc-sfu:latest && docker run -d webrtc-sfu"
# No rollback path for colo. No health check. No blast radius isolation.
echo "Deploy complete. Pray nothing breaks."
The problem is not that SSH and docker pull do not work. They work. The problem is that there is no rollback path for the colo side, no health check gate, no blast radius isolation, and no way to deploy the colo hot path independently of the AWS control plane. When the MMR mixer breaks on colo-prod-01, the entire pipeline, including the AWS control plane that was fine, is now blocked. A colo hardware issue takes down the cloud deploy.
The Right Abstraction: A Unified Deploy Spec
The fix is a deploy-unit definition that treats colo and cloud as first-class targets in the same Terraform-managed pipeline. Terraform handles AWS and OCI. Ansible handles the colo bare-metal provisioning because Terraform cannot manage hardware it cannot reach via a cloud API. Both are driven from the same GitOps repo with the same deploy spec.
# deploy-units/mmr-hot-path.yaml
apiVersion: castlesoflight.io/v1
kind: DeployUnit
metadata:
name: mmr-hot-path
labels:
criticality: hot-path
latency-budget: 5ms
sla-tier: tier-0
spec:
targets:
- name: colo-mmr-mixer
provider: ansible
inventory: colo/prod-01,prod-02
playbook: playbooks/mmr-mixer.yml
healthCheck:
type: grpc
endpoint: ":9090/healthz"
timeout: 2s
retries: 3
rollback:
enabled: true
strategy: blue-green
previousImage: "{{ .PreviousTag }}"
- name: aws-control-plane
provider: terraform
module: modules/aws-control-plane
workspace: prod
healthCheck:
type: http
endpoint: "https://api.castlesoflight.io/healthz"
timeout: 5s
retries: 3
- name: oci-recording-archive
provider: terraform
module: modules/oci-recording
workspace: prod
deployOrder:
- aws-control-plane
- colo-mmr-mixer
- oci-recording-archive
rollbackOrder:
- colo-mmr-mixer
- aws-control-plane
observability:
sli:
- name: media-stream-latency
query: histogram_quantile(0.99, rate(mmr_stream_latency_bucket[5m]))
threshold: 5ms
- name: webrtc-session-success
query: rate(webrtc_sessions_success_total[5m]) / rate(webrtc_sessions_total[5m])
threshold: 0.999
slo:
- name: tier-0-availability
target: 99.99
window: 30d
The deploy-unit is the abstraction that makes colo and cloud deployable from one pipeline without coupling them. Each target has its own provider (Ansible for colo, Terraform for AWS and OCI), its own health check, and its own rollback strategy. The deploy order ensures the AWS control plane is healthy before the colo hot path rolls, because the colo mixer needs the control plane to register its capacity. The rollback order is the reverse of deploy, but only for the targets that actually failed, not the whole pipeline.
The Placement Algorithm
The harder problem is deciding what goes on colo and what goes in cloud. The naive answer is "everything latency-sensitive goes on colo." The real answer is a constraint solver that accounts for the latency budget, the colo rack footprint, the cloud egress cost, and the failover topology.
"""
Latency-aware placement solver for hybrid colo/cloud deployments.
Determines which services must run on bare metal vs which can
tolerate cloud round-trip latency.
"""
from dataclasses import dataclass, field
from typing import Optional
@dataclass
class Service:
name: str
latency_budget_ms: float
depends_on: list[str] = field(default_factory=list)
colo_rack_units: int = 0
cloud_compatible: bool = True
egress_cost_monthly: float = 0.0
@dataclass
class PlacementDecision:
service: str
target: str # "colo" or "cloud"
reason: str
latency_headroom_ms: float
COLO_RTT_TO_EDGE = 1.2 # ms, measured bare-metal to edge switch
CLOUD_RTT_TO_EDGE = 8.5 # ms, measured AWS us-west-2 to edge
COLO_RACK_CAPACITY = 40 # total rack units available
def solve_placement(services: list[Service]) -> list[PlacementDecision]:
decisions: list[PlacementDecision] = []
used_rack_units = 0
for svc in sorted(services, key=lambda s: s.latency_budget_ms):
# Walk the dependency chain to compute cumulative latency
cumulative_latency = sum(
next(d for d in decisions if d.service == dep).latency_headroom_ms
for dep in svc.depends_on
if any(d.service == dep for d in decisions)
)
cloud_latency = cumulative_latency + CLOUD_RTT_TO_EDGE
colo_latency = cumulative_latency + COLO_RTT_TO_EDGE
# Decision: colo if latency budget is tight OR colo rack space available
must_colo = svc.latency_budget_ms < cloud_latency
can_colo = used_rack_units + svc.colo_rack_units <= COLO_RACK_CAPACITY
if must_colo and not can_colo:
raise RuntimeError(
f"Placement infeasible: {svc.name} needs colo "
f"(budget {svc.latency_budget_ms}ms < cloud {cloud_latency:.1f}ms) "
f"but rack is full ({used_rack_units}/{COLO_RACK_CAPACITY}U)"
)
if must_colo:
target = "colo"
reason = f"latency budget {svc.latency_budget_ms}ms < cloud RTT {cloud_latency:.1f}ms"
used_rack_units += svc.colo_rack_units
headroom = svc.latency_budget_ms - colo_latency
elif svc.colo_rack_units > 0 and not svc.cloud_compatible:
target = "colo"
reason = "service not cloud-compatible"
used_rack_units += svc.colo_rack_units
headroom = svc.latency_budget_ms - colo_latency
else:
target = "cloud"
reason = f"latency budget {svc.latency_budget_ms}ms > cloud RTT {cloud_latency:.1f}ms"
headroom = svc.latency_budget_ms - cloud_latency
decisions.append(PlacementDecision(
service=svc.name, target=target, reason=reason,
latency_headroom_ms=headroom
))
return decisions
The solver walks the service dependency graph in latency-budget order (tightest budget first) and places each service on colo or cloud based on whether the cumulative latency through its dependency chain fits within the budget. It fails loudly when the placement is infeasible, which is the single most important property of a placement solver. A silent fallback to cloud when the colo rack is full produces a latency regression that nobody notices until the media stream starts dropping frames in production.
Blast Radius: What Does This Change Actually Affect
The second hardest problem is blast radius isolation. When you change a Terraform module that provisions the colo MMR mixer, does that change also affect the AWS control plane? The naive answer is "no, they are different modules." The real answer requires computing the transitive dependency graph and checking whether any colo change transitively affects a cloud resource, or vice versa.
"""
Blast radius computation for hybrid deploy-unit changes.
Given a changed file in the GitOps repo, determines which
deploy targets are affected and whether a colo change
transitively impacts cloud resources (or vice versa).
"""
from dataclasses import dataclass, field
from pathlib import Path
import re
@dataclass
class BlastRadius:
changed_files: list[str]
directly_affected: list[str] # target names
transitively_affected: list[str]
cross_boundary: bool # colo change affecting cloud, or vice versa
risk_level: str # "low", "medium", "high"
reasoning: list[str] = field(default_factory=list)
# Map file paths to deploy targets
TARGET_PATTERNS = {
r"modules/aws-.*": "aws-control-plane",
r"modules/oci-.*": "oci-recording-archive",
r"playbooks/.*": "colo-mmr-mixer",
r"deploy-units/.*": None, # affects all targets
}
# Cross-boundary dependencies: colo depends on cloud
CROSS_BOUNDARY_DEPS = {
"colo-mmr-mixer": ["aws-control-plane"],
"colo-webrtc-sfu": ["aws-control-plane", "oci-recording-archive"],
}
def compute_blast_radius(changed_files: list[str]) -> BlastRadius:
directly_affected: set[str] = set()
affects_all = False
for path in changed_files:
matched = False
for pattern, target in TARGET_PATTERNS.items():
if re.match(pattern, path):
if target is None:
affects_all = True
matched = True
break
directly_affected.add(target)
matched = True
if not matched:
# Unknown file, assume it affects everything
affects_all = True
if affects_all:
directly_affected = {"aws-control-plane", "oci-recording-archive", "colo-mmr-mixer"}
# Propagate cross-boundary dependencies
transitively_affected: set[str] = set(directly_affected)
for target in list(directly_affected):
for dep in CROSS_BOUNDARY_DEPS.get(target, []):
transitively_affected.add(dep)
cross_boundary = (
any("colo" in t for t in directly_affected) and
any("aws" in t or "oci" in t for t in transitively_affected)
)
if cross_boundary:
risk_level = "high"
reasoning = [
f"Change affects colo target(s) {directly_affected} which",
f"transitively depends on cloud target(s) {transitively_affected - directly_affected}.",
"Colo hot-path changes that transitively impact cloud resources",
"require staged deployment with health-check gates between stages.",
]
elif len(directly_affected) > 1:
risk_level = "medium"
reasoning = [f"Change affects {len(directly_affected)} targets.", "Deploy in dependency order."]
else:
risk_level = "low"
reasoning = [f"Change isolated to {directly_affected.pop()}.", "Safe to deploy independently."]
return BlastRadius(
changed_files=changed_files,
directly_affected=list(directly_affected),
transitively_affected=list(transitively_affected),
cross_boundary=cross_boundary,
risk_level=risk_level,
reasoning=reasoning,
)
The key insight is that a colo change can transitively affect cloud resources through the control-plane dependency. If the MMR mixer depends on the AWS control plane for capacity registration, a change to the Ansible playbook that provisions the mixer does not directly affect the control plane, but the blast radius computation flags it as cross-boundary because a failed mixer health check will trigger a rollback that the control plane has to coordinate. This is the kind of coupling that produces 3 AM pages when nobody is thinking about it.
The Report
The blast radius computation produces a report that the CI/CD pipeline consumes:
# blast-radius-report.yaml (generated by CI/CD pre-deploy gate)
changed_files:
- playbooks/mmr-mixer.yml
- modules/aws-control-plane/variables.tf
directly_affected:
- colo-mmr-mixer
- aws-control-plane
transitively_affected:
- colo-mmr-mixer
- aws-control-plane
cross_boundary: true
risk_level: high
reasoning:
- "Change affects colo target(s) {'colo-mmr-mixer'} which"
- "transitively depends on cloud target(s) {'aws-control-plane'}."
- "Colo hot-path changes that transitively impact cloud resources"
- "require staged deployment with health-check gates between stages."
recommended_strategy:
deploy_order:
- aws-control-plane
- colo-mmr-mixer
health_gate_between_stages: true
rollback_on_failure: true
notify_oncall: true
slack_channel: "#infra-deploy"
The CI/CD Gate
The deploy gate ties it together. It reads the blast radius report, enforces the deploy order, runs health checks between stages, and rolls back on failure. The gate is a Python script that runs in CI and orchestrates Terraform, Ansible, and Kubernetes in the right order with the right health checks.
"""
CI/CD deploy gate for hybrid colo/cloud deploy-units.
Reads blast-radius-report.yaml, enforces deploy order,
runs health checks between stages, rolls back on failure.
"""
import subprocess, yaml, sys, json, time
from pathlib import Path
def run_deploy_gate(report_path: str = "blast-radius-report.yaml"):
report = yaml.safe_load(Path(report_path).read_text())
strategy = report["recommended_strategy"]
print(f"Deploy gate: risk={report['risk_level']} cross_boundary={report['cross_boundary']}")
for reason in report["reasoning"]:
print(f" {reason}")
if report["risk_level"] == "high":
print("High-risk deploy detected. Notifying on-call.")
notify_oncall(report)
for i, target in enumerate(strategy["deploy_order"]):
print(f"\n[{i+1}/{len(strategy['deploy_order'])}] Deploying {target}...")
if target.startswith("aws-") or target.startswith("oci-"):
exit_code = run_terraform(target)
elif target.startswith("colo-"):
exit_code = run_ansible(target)
else:
print(f"Unknown target type: {target}")
sys.exit(1)
if exit_code != 0:
print(f"Deploy failed for {target}. Rolling back.")
rollback(target, strategy.get("rollback_on_failure", True))
sys.exit(1)
# Health check gate between stages for cross-boundary deploys
if strategy.get("health_gate_between_stages") and i < len(strategy["deploy_order"]) - 1:
print(f"Health gate: waiting for {target} to stabilize...")
if not wait_for_health(target, timeout=120):
print(f"Health check failed for {target}. Rolling back.")
rollback(target, strategy.get("rollback_on_failure", True))
sys.exit(1)
print(f" {target} healthy. Proceeding to next stage.")
print("\nDeploy complete. All targets healthy.")
post_deploy_summary(report)
def run_terraform(target: str) -> int:
module = f"modules/{target.replace('-', '_', 1)}"
return subprocess.call([
"terraform", "apply", "-auto-approve",
f"-target=module.{target}", f"-var-file=env/prod.tfvars"
])
def run_ansible(target: str) -> int:
playbook = f"playbooks/{target.replace('colo-', '')}.yml"
return subprocess.call([
"ansible-playbook", playbook, "-i", "inventory/prod.ini",
"--limit", target
])
def wait_for_health(target: str, timeout: int = 120) -> bool:
# In production, this calls the service's gRPC/HTTP health endpoint
# and checks SLIs against thresholds from the deploy-unit spec
deadline = time.time() + timeout
while time.time() < deadline:
result = subprocess.run(
["python3", "scripts/health-check.py", "--target", target],
capture_output=True, text=True
)
if result.returncode == 0:
return True
time.sleep(5)
return False
def rollback(target: str, enabled: bool = True):
if not enabled:
print("Rollback disabled. Manual intervention required.")
return
if target.startswith("colo-"):
subprocess.call([
"ansible-playbook", f"playbooks/{target.replace('colo-', '')}-rollback.yml",
"-i", "inventory/prod.ini", "--limit", target
])
else:
subprocess.call(["terraform", "apply", "-auto-approve",
f"-target=module.{target}", "-var-file=env/prod.tfvars"])
def notify_oncall(report: dict):
# Post to Slack #infra-deploy
msg = f":rotating_light: High-risk deploy started\n"
for r in report["reasoning"]:
msg += f" {r}\n"
msg += f"Targets: {', '.join(report['directly_affected'])}"
subprocess.call(["python3", "scripts/slack-notify.py", "--message", msg])
def post_deploy_summary(report: dict):
print(f"\nDeploy summary:")
print(f" Risk: {report['risk_level']}")
print(f" Targets deployed: {len(report['directly_affected'])}")
print(f" Cross-boundary: {report['cross_boundary']}")
print(f" All health checks passed.")
if __name__ == "__main__":
run_deploy_gate()
The gate does four things that the naive SSH-and-docker-pull script does not. It computes blast radius before touching anything. It deploys in dependency order so the control plane is healthy before the hot path rolls. It runs health checks between stages so a failed colo deploy does not cascade. It rolls back automatically, including the Ansible rollback playbook for colo targets that Terraform cannot reach.
When NOT to Use This Pattern
This architecture is overkill if your latency budget is above 50ms. A web application with a 200ms p99 SLA does not need a colo hot path. The pattern is for real-time media (MMR, WebRTC), high-frequency trading pipelines, defense-grade telemetry, or any system where the physics of cloud round-trip time breaks the product. If your services tolerate cloud latency, use cloud-native Kubernetes and skip the colo entirely.
The pattern is also wrong if your team has never operated bare metal. Colocation introduces a class of failure modes that cloud-native teams do not encounter: hardware provisioning lead times, switch configuration drift, BGP routing changes, physical rack space constraints, and on-site hands for hardware replacement. If your team is cloud-only and the latency budget does not demand colo, do not add colo to solve a problem you do not have.
What This Looks Like Done Right
When the hybrid colo/cloud pattern is deployed correctly, the MMR media mixer runs on bare metal with sub-5ms inter-frame latency, the WebRTC SFU runs on dedicated hardware with DPDK-bonded NICs, and the control plane in AWS orchestrates capacity registration, scaling, and failover without sitting in the media data path. The CI/CD pipeline deploys to colo and cloud from the same GitOps repo, with blast radius isolation that prevents a colo hardware issue from blocking a cloud deploy, and a placement solver that fails loudly when the colo rack is full rather than silently degrading to cloud latency.
The observability stack tracks SLIs that matter for real-time media: media stream latency at p99, WebRTC session success rate, MMR mixer jitter buffer depth, and SFU packet loss. The SLOs are tiered: tier-0 (hot path) at 99.99% availability, tier-1 (control plane) at 99.9%. Postmortems are driven by the SLI data, not by user reports. The CI/CD gate enforces the deploy order, runs health checks between stages, and rolls back automatically on failure.
This is the infrastructure that Zoom is building. It is the infrastructure I have built at Spotify, Google ProServe, and Raytheon. It is the hardest infrastructure pattern to hire for, because the skill set spans colo and cloud, networking and Kubernetes, hardware and GitOps, and almost nobody on the market has done both at production scale.
If your team is hitting the wall between colo and cloud, between real-time latency and cloud-native architecture, I run a 3-day infrastructure Sprint that maps the hot path, builds the unified deploy spec, and gets your CI/CD pipeline deploying to both worlds without breaking either one. I also operate as a fractional CTO for teams that need ongoing infrastructure leadership across the hybrid boundary. Learn more at castlesoflight.com.
Next: SLIs Before SLOs, the observability stack that measures real-time media latency without drowning in dashboards. Subscribe for the next piece.