Cloudflare (2026): when a single-AZ dependency takes down an 'HA' cluster

On February 20, 2026, Cloudflare recorded a failure in internal zone PDX-04, located in Portland, Oregon. PDX-04 is one of the internal datacenters Cloudflare operates to support its control infrastructure — distinct from the global edge network that delivers HTTP traffic, DNS, and security to its customers.

The zone failure itself was not the central problem. The problem was what lived exclusively in that zone: Kafka instances responsible for log event ingestion, and ClickHouse instances responsible for storing and serving analytics queries. These two systems had no replicas in other zones. They were, in practice, single-AZ — despite being part of an architecture that, on paper, was described as highly available.

When PDX-04 became unavailable, the control plane services that depended on Kafka to publish events and on ClickHouse to query analytics data simply stopped working or entered error states. Cloudflare's global data plane — which processes real customer traffic — continued operating normally, as it is architecturally independent. But the ability to observe, control, and analyze that traffic was compromised.

What makes this incident particularly interesting from an architectural standpoint is the nature of the dependency: it was implicit. The HA cluster services apparently had no explicit documentation stating they depended on single-AZ components. The dependency mapping had not captured this relationship. And resilience tests — if they existed — had not simulated the isolated loss of PDX-04.

There is a critical distinction between declared HA and verified HA. Declared HA is what appears in architecture documentation: "our cluster has N nodes in M zones". Verified HA is what you get when you map all transitive dependencies of each service and confirm that each of them also satisfies the same availability SLA.

In Cloudflare's case, the control plane cluster satisfied declared HA. But Kafka and ClickHouse — which are critical supporting dependencies, not optional ones — were single-AZ. This creates what I call HA with an asterisk: the system survives individual compute node failures, but collapses completely if the underlying data service fails.

This pattern is surprisingly common. It appears when:

1. Infrastructure services grow organically: Kafka and ClickHouse were likely initially provisioned as smaller supporting services, without the same resilience rigor applied to primary production services. Over time, the dependency grew, but the criticality classification was not revisited.

1. Dependency mapping is not maintained: In rapidly evolving systems, it is common for the real dependency graph to diverge from the documented one. Without automatic dependency discovery tools or structured periodic reviews, these gaps accumulate silently.

1. Resilience testing is incomplete: Chaos engineering and game days tend to test failures in services we already know are critical. Supporting dependencies — like observability pipelines — are often excluded because they are considered "not business-critical". But when the control plane fails, the distinction between critical and non-critical collapses.

There is also a coupling between data plane and observability plane dimension worth noting. In this incident, Cloudflare's global traffic continued flowing — the data plane was healthy. But the ability to observe, diagnose, and potentially intervene in that traffic was compromised. In security or incident response scenarios, this loss of visibility can turn a manageable problem into a critical one.

Cloudflare's post-mortem identifies corrective actions across several dimensions. I analyze each from the perspective of someone who has implemented similar systems:

1. Multi-zone replication for Kafka and ClickHouse

The most direct action: ensure Kafka brokers and ClickHouse nodes exist in multiple zones, with replication configured between them. For Kafka, this means configuring min.insync.replicas and replication.factor so that losing a zone does not result in data loss or write unavailability. For ClickHouse, it means configuring replicas in distinct zones with synchronous or asynchronous replication depending on the acceptable consistency SLA.

The practical challenge here is cost and latency. ClickHouse with synchronous cross-zone replication introduces write latency. The decision between asynchronous and synchronous replication is a real trade-off that needs to be explicitly documented — not assumed.

2. Dependency audit and mapping

Identify all services that depend on single-AZ components and classify them by criticality. This sounds simple, but in organizations at Cloudflare's scale, the dependency graph can have hundreds of services. The practical approach I recommend is a combination of:

• Distributed tracing (like Jaeger or Zipkin) for automatic runtime dependency discovery
• Structured architecture reviews before any service promotion to production
• A dependency registry maintained as code (dependency manifests), audited periodically

3. Isolation between data plane and observability plane

This is the most sophisticated lesson from the incident. The observability plane — logs, metrics, analytics — should not share infrastructure dependencies with the data plane it monitors. If the log pipeline uses the same Kafka as the control plane, a failure in that Kafka simultaneously blinds the operational system and the diagnostic system.

The architectural solution is to treat the observability plane as a system of equal or higher criticality than the data plane, with its own set of isolated dependencies. In AWS, for example, this means log pipelines in separate accounts, with strictly controlled cross-account IAM and no shared infrastructure dependencies.

4. Chaos engineering with expanded scope

Explicitly include supporting dependencies — Kafka, ClickHouse, cache systems, ETL pipelines — in chaos engineering scenarios. The question that should guide experiment design is not "what happens if this service fails?" but "what happens if any component this service depends on, directly or transitively, fails?"