Teardown: Resilient Network Graphs and the Next-Generation AI Network

For most of cloud computing history, data center networks were designed for microservice east-west traffic: many short connections, reasonable tolerance for variable latency, and relatively unpredictable traffic patterns that benefit from static ECMP. The model works well for OLTP, video streaming, and REST APIs. It fails categorically for distributed AI model training.

The reason is structural. In a collective All-Reduce training job — the dominant pattern in frameworks like PyTorch with NCCL — all workers must synchronize gradients at every optimization step. This generates an incast traffic pattern: hundreds or thousands of flows converge simultaneously on the same aggregation switches. Buffers overflow. Packets are dropped. NCCL enters retransmission. The job that should finish in 12 hours finishes in 18 — or doesn't finish at all.

The congestion problem is compounded by three AI-specific factors: (1) tensor size — a single All-Reduce on a 70B parameter model can move hundreds of gigabytes per step; (2) synchronicity — unlike web traffic, workers fire simultaneously because SGD is synchronous by design; (3) jitter sensitivity — tail latency kills efficiency because the step only advances when the last worker finishes. A single congested link can degrade GPU utilization from 90% to 40%.

Beyond congestion, there is the failure topology problem. In a classic Clos network with static ECMP, a spine switch failure can create load asymmetry that the data plane does not detect quickly. For long-running workloads (multi-day jobs), this means silent degradation — the job continues, but slowly, with no obvious alarm. Detecting and re-routing in seconds, not minutes, is a production requirement.

Finally, there is the energy cost dimension. A 16,000-GPU H100 cluster consumes on the order of 50-80 MW in compute alone. The interconnect network adds 10-20% on top of that. High-radix switches with deep buffers and high-speed SerDes are power-hungry. The choice of topology, port speed, and buffer policy has a direct impact on PUE and the energy bill — and therefore on Amazon's sustainability goals.

The core architecture is a multi-stage Clos network — specifically a three-layer fat-tree variant: ToR (Top-of-Rack), Aggregation, and Spine. This is not novel; Clos has been the industry standard since the 2000s. What differentiates AWS's approach for AI is what happens above the data plane.

Graph-based control plane. The network is modeled as a weighted directed graph where nodes are switches and edges are links with attributes for capacity, current utilization, and health state. A centralized controller (analogous to an SDN controller, but with graph semantics) maintains this representation in memory and runs shortest-path algorithms with capacity constraints — essentially a Dijkstra or Bellman-Ford variant with multiple objectives: minimize tail latency, maximize bisection throughput, and avoid degraded links. When in-band telemetry (INT) reports a link above 70% utilization, the controller recalculates alternative routes and injects them into the data plane via P4 or OpenConfig, without human intervention. The target convergence time is sub-second — critical for training jobs that cannot pause.

RoCEv2 and DCQCN. Data transport between GPUs uses RDMA over Converged Ethernet (RoCEv2), which eliminates kernel memory copy overhead and reduces latency to tens of microseconds. The classic RoCEv2 problem is that it is lossless by design — it uses PFC (Priority Flow Control) to pause upstream transmitters when buffers fill. Misconfigured PFC generates PFC deadlock, where pauses propagate through the network and stall unrelated traffic. AWS mitigates this with DCQCN (Data Center Quantized Congestion Notification), a congestion control algorithm based on ECN that reduces the sender's injection rate before the buffer overflows, keeping the network in a non-lossy regime without relying on PFC as the primary mechanism.

In-band telemetry (INT). Each packet carries latency and queue utilization metadata collected by every switch in the path. The controller aggregates this data in real time to build a network heat map. This is fundamentally different from periodic SNMP polling — the temporal granularity is per-flow, not per-interface, and the observability latency is milliseconds, not minutes. For a training job with All-Reduce every 100ms, this makes a real operational difference.

Checkpointing and job resilience. The network alone does not guarantee job resilience. AWS integrates periodic checkpointing to S3/EFS directly into training frameworks (via SageMaker or Trainium SDK). If a node fails or a link degrades to the point of job abort, training restarts from the last checkpoint, not from scratch. Checkpoint frequency is a trade-off: more frequent checkpoints reduce lost work but increase network traffic and I/O cost — especially relevant for 100B+ parameter models where a single checkpoint can be hundreds of GB.

Amazon achieved 100% renewable energy in 2023 — ahead of the original 2025 target. This is relevant to this teardown not as an ESG footnote, but because sustainability is becoming a first-class design constraint in AI network architecture.

The argument is straightforward: a 10,000-accelerator AI training cluster consumes on the order of 30-60 MW. The interconnect network represents 10-20% of that consumption — meaning 3-12 MW in switches and transceivers alone. The choice between 112G PAM4 vs. 56G NRZ SerDes for 400G links has a direct impact on consumption per bit. The choice of deep buffers (necessary to absorb incast) vs. shallow buffers (more energy-efficient) is an explicit trade-off between network efficiency and energy consumption.

AWS is addressing this on three fronts. First, PUE of ~1.2 in new data centers — achieved via direct liquid cooling in GPU racks and airflow optimization. For context: a PUE of 1.2 means 83% of energy goes to compute; a PUE of 1.5 (typical of legacy data centers) means only 67%. The difference in a 50 MW cluster is 8 MW — enough to power a small city.

Second, dynamic power capping integrated with the network controller. When data center power demand approaches a limit, the controller can reduce clock frequencies of non-critical switches or consolidate traffic on fewer physical links (shutting down the rest). This is analogous to what processors do with DVFS, but applied to the network.

Third, co-location with renewable sources. AWS is building data centers near solar and wind farms to minimize transmission losses and ensure temporal matching of renewable energy — not just annual credits (RECs), but real hourly matching. This influences region location choices and, indirectly, inter-region network latency.

The point I want to make clear: for systems architects designing AI workloads, network energy efficiency is not the AWS infrastructure team's problem — it is a parameter that affects cost per trained token and therefore the economic viability of the product. Choosing instances with more efficient networking (Trainium2 vs. P4d, for example) is an architectural decision with measurable financial impact.

This teardown is not just about what AWS does internally. The architectural decisions of AWS's network infrastructure propagate directly to the choices systems architects make when designing AI workloads in the cloud.

Instance choice is not just about FLOP/$. The instance's network topology matters as much as the number of accelerators. A P4de instance with EFA in a cluster placement group has fundamentally different network behavior from a G5 instance in a generic AZ. For distributed training jobs above 8 GPUs, instance and placement group selection should be guided by analysis of the model's communication pattern — not just compute cost.

Model partitioning has network implications. Tensor parallelism (TP) generates high-frequency all-to-all traffic between GPUs on the same node — ideal for NVLink, not Ethernet. Pipeline parallelism (PP) generates point-to-point traffic between stages — tolerant of higher latency. Data parallelism (DP) generates periodic All-Reduce — the case for which the network described in this teardown is optimized. An architect who understands network topology can choose the parallelism strategy that minimizes the communication bottleneck for their specific model.

Network monitoring is part of the training SLA. In financial systems, I would never accept an availability SLA without network latency monitoring. The same principle applies to AI clusters: the job completion SLA must include network metrics (link utilization, RDMA retransmission rate, All-Reduce latency). CloudWatch with EFA metrics and VPC Flow Logs are the minimum; INT via tools like AWS Network Manager is the state of the art.

Network cost is visible and optimizable. Cross-AZ traffic has an explicit cost on AWS. A poorly partitioned training job that generates cross-AZ traffic can have network costs that exceed compute costs. This is an architecture bug, not an infrastructure problem. The solution is cluster placement groups (guarantees co-location in the same AZ and ideally the same rack) and traffic analysis before scaling.