Zone-Aware Routing in ECS Service Connect: The End of the Cost vs. Resilience Trade-off

ECS Service Connect operates as a lightweight Envoy-based service mesh, injecting a sidecar proxy into each task and registering endpoints in an AWS Cloud Map namespace. Before this feature, load balancing across endpoints was topology-agnostic: a task in us-east-1a could route to replicas in us-east-1b or us-east-1c with equal probability, depending only on health and configured weight.

With zone-aware routing, each task's Envoy proxy receives zone metadata and prioritizes endpoints in the same AZ with a dynamic weight. The mechanism is not a static 100% local routing — it is a weight algorithm that adjusts based on locally available capacity. If the local AZ has sufficient healthy endpoints to absorb the load, traffic stays local. If local endpoints fall below a capacity threshold or become unhealthy, the algorithm automatically redistributes to neighboring AZs, without manual intervention and without overloading any specific zone.

This behavior mirrors what Kubernetes implements with topologyAwareHints in EndpointSlice (GA in Kubernetes 1.27) and what ALB does with cross-zone load balancing disabled. The difference is that in ECS Service Connect this is transparent — no annotations required, no hints configuration, and it works with native Service Connect DNS without any application code change.

On payment platforms I operate or review as reference, the typical pattern is an ingress gateway (ALB or API Gateway) distributing to an orchestration service, which in turn calls authorization, anti-fraud, ledger, and notification services in sequence or parallel. With 3 AZs and 10 tasks per service (roughly 3-4 per AZ), cross-AZ traffic before zone-aware routing was statistically ~66% of calls — two out of every three calls crossed a zone.

With zone-aware routing, that number drops to near zero under normal conditions. For a platform processing 100 GB/hour of payload in internal calls (not uncommon in settlement systems with 1-5 KB messages at 50k req/s), the direct saving is ~$48/hour in eliminated cross-AZ transfer — approximately $35k/month. Not the largest cost item on an enterprise platform, but a saving that requires zero architectural change.

More important than cost, however, is the impact on compound latency. In synchronous call chains with 6-8 hops (standard in payment systems with separation of concerns), each 1-2ms of cross-AZ latency multiplies. Reducing that to intra-AZ latency (~0.3ms) under normal conditions measurably improves the p95 of the full chain — especially during peak hours when latency variance increases.

The important caveat for financial environments is capacity asymmetry between AZs during scaling events. If autoscaling reacts faster in one AZ than others (which happens when CPU/memory triggers fire at slightly different moments), the AZ with more capacity receives more local traffic — and the AZ with less capacity redistributes to neighbors. This behavior is correct and expected, but it needs to be modeled in incident response runbooks.

The announcement recommends VPC Flow Logs with AZ metadata to validate routing effectiveness. This is a necessary starting point, but insufficient for financial production environments. Flow Logs operate at the network level — they confirm traffic is flowing within the AZ, but do not correlate with application latency, error rate, or retry behavior.

For a complete observability stack, I add three layers on top of Flow Logs. First, Envoy metrics exposed via ECS Service Connect: the Envoy proxy already collects upstream_cx_total, upstream_rq_retry, and latency histograms per destination cluster. With ECS Container Insights enabled and an Envoy metrics exporter (or via AWS Distro for OpenTelemetry), these metrics reach CloudWatch or Datadog with AZ granularity. Second, distributed traces with zone context propagation: by adding the az.id attribute to each call's span (via OpenTelemetry SDK), it is possible to build latency heatmaps per AZ origin-destination pair in X-Ray or Datadog APM — making it visible exactly when and how much cross-AZ traffic still occurs (during failovers). Third, zone redistribution alarms: a composite CloudWatch alarm that simultaneously detects a drop in healthy endpoints in one AZ AND a p99 latency increase in the destination service is a reliable signal that redistribution is active — and allows distinguishing expected degradation from a real failure.

The cost of Flow Logs in high-frequency environments is not trivial. At 50k req/s with 1 KB payloads, flow log volume can reach 10-20 GB/hour. I recommend using VPC Flow Logs with custom format including only necessary fields (srcaddr, dstaddr, az-id, bytes, start, end) and directing to S3 with hourly and AZ partitioning — which reduces storage cost and enables efficient Athena queries without going through CloudWatch Logs.

Zone-aware routing solves a real problem, but introduces three failure modes that need to be on every serious architect's radar.

1. AZ Imbalance Amplified by Asymmetric Autoscaling. If autoscaling of a downstream service scales one AZ faster than others (common when CPU metrics have 1-minute evaluation windows and tasks do not scale at exactly the same time), the AZ with more tasks receives more local traffic — which increases its load and may trigger more scaling, while AZs with fewer tasks remain underutilized. The antidote is configuring target tracking scaling with ALB RequestCountPerTarget metrics instead of CPU, and ensuring the deployment's minHealthyPercent and maximumPercent are calibrated to maintain at least 1 task per AZ during deploys.

2. Thundering Herd During AZ Recovery. When an AZ recovers after a failure (endpoints become healthy again), Envoy begins redistributing traffic back to it. If recovery is abrupt (all endpoints return simultaneously), there may be a traffic spike to the newly recovered AZ before it is fully warmed up — particularly problematic for JVM services that need JIT warmup. The mitigation is using slow start mode in Envoy (configurable via ECS Service Connect task definition) with a 30-60 second window, gradually increasing the weight of newly healthy endpoints.

3. Redistribution Invisibility in Aggregated Dashboards. Latency and error metrics aggregated by service mask zone redistributions. An overloaded AZ with p99 of 200ms can be invisible if the other two AZs have p99 of 20ms — the aggregated average lands at ~80ms, below any alarm threshold. The solution is to mandatorily break metrics by AZ dimension in CloudWatch (using AvailabilityZone as a dimension in custom metrics) and define SLOs per AZ, not just per service.

This feature closes an important competitive gap between ECS Service Connect and service mesh alternatives. AWS App Mesh (now in maintenance mode, with migration recommended to Service Connect) never had native zone-aware routing. Istio on EKS has had localityLbSetting with configurable failover priorities for years. Native Kubernetes has had topologyAwareHints in EndpointSlice since 1.21 (beta) and 1.27 (GA). ECS Service Connect was behind on this specific dimension — and now reaches functional parity, with the advantage of being zero-config.

For teams evaluating ECS vs. EKS for new platforms, this launch removes one argument in favor of EKS. The relevant question is no longer 'EKS has zone-aware and ECS does not' — it is 'what is the operational cost of managing a Kubernetes cluster vs. using ECS with Fargate and Service Connect?' For most financial platforms that do not need deep scheduler customization or specific Kubernetes operators, ECS with Service Connect now offers a comparable feature set with significantly less operational overhead.

The caveat is that ECS Service Connect still does not expose granular locality failover controls like Istio does. In Istio, it is possible to define explicit priorities (same AZ = priority 0, same region = priority 1, other region = priority 2) with configurable spillover percentages. In ECS Service Connect, the algorithm is managed by AWS — which is excellent for operational simplicity, but can be limiting for scenarios where you need fine-grained control over degradation behavior. For those cases, App Mesh (while still supported) or Istio on EKS remain the more appropriate choice.