OpenAI (2024): How a New Telemetry Service Took Down the Kubernetes Control Plane

On December 11, 2024, OpenAI's infrastructure team initiated the rollout of a new telemetry collection service. The intent was legitimate and common: improve internal observability of the Kubernetes fleet that underpins all inference infrastructure for ChatGPT, the public API, and Sora. The agent was packaged as a DaemonSet — Kubernetes' standard mechanism for running a pod on every node in the fleet — and the deployment was triggered broadly, hitting a very large number of nodes simultaneously.

The problem was not in the agent's code itself. It was in the emergent behavior of thousands of instances of that agent initializing at the same time and, in doing so, each establishing persistent connections to the Kubernetes API servers to observe cluster state — the Kubernetes API watch pattern. At OpenAI's production fleet scale, this meant hundreds or thousands of watch connections were opened within a very short time window. The API servers, which are the central control point for the entire cluster, began to saturate.

What makes this incident particularly instructive is the role of DNS caching. During the first minutes after the rollout began, the impact was not immediately visible in end-user service health indicators. The cluster's internal DNS was still resolving addresses from cached entries, meaning application traffic continued flowing normally for a period. This created a window of false safety — latency and error alerts for product services did not fire immediately, which delayed recognition that something was fundamentally wrong at the infrastructure layer.

When DNS caches began to expire and new resolutions needed to pass through Kubernetes' service discovery mechanisms (which depend on the already-saturated API servers), the impact cascaded rapidly to the data plane. Pods could not be scheduled, services could not be discovered, and user traffic began failing at global scale.

One of the most technically interesting aspects of this incident is the role of the cluster's internal DNS cache — and how it transformed a detectable problem into one that revealed itself too late for a fast response.

In a Kubernetes cluster, service name resolution is handled by CoreDNS (or equivalent). When a pod needs to communicate with another service, it resolves the service's DNS name, which points to the corresponding ClusterIP. These resolutions are cached locally in pods and intermediate resolvers with a defined TTL — typically between 5 and 30 seconds for cluster DNS, but the actual implementation depends on how ndots, search, and CoreDNS TTLs are configured.

At the moment the telemetry DaemonSet began saturating the API servers, application traffic between product services (ChatGPT, API, Sora) continued working normally because DNS resolutions were already cached. Pods did not need to query CoreDNS for each request — they already had the resolved IP addresses. This created a time window — estimated at 15 to 20 minutes based on the incident's progression — during which the control plane was severely degrading, but product SLIs (error rate, latency) remained within normal bounds.

This window of false normality is a classic trap in distributed systems: the caching layer absorbs the initial impact and masks degradation in the underlying layer. When caches expired and new resolutions needed to pass through CoreDNS — which in turn depends on Kubernetes endpoints updated by the saturated API servers — the impact became immediate and total. There was no gradual degradation visible to end users: it was an abrupt transition from "everything working" to "nothing working".

This has direct implications for alert design: monitoring only product SLIs is insufficient when there is a caching layer between the user and the infrastructure. Control plane health must be monitored independently, with alerts that fire well before the cache expires — not after.

The remediation phase of this incident exposes one of the most dangerous properties of Kubernetes control plane failures: the system you use to fix the problem is the same system that is broken.

When the team identified the telemetry DaemonSet as the cause and attempted to execute the rollback — whether via kubectl rollout undo, via a GitOps operator like ArgoCD or Flux, or via direct API calls — all of these operations require the Kubernetes API servers to accept and process write requests. An API server saturated by thousands of active watch connections has no available processing capacity to reliably accept new control commands. Rollback requests were queued, hit timeouts, or failed entirely.

This is not a Kubernetes bug — it is an expected architectural consequence of a centralized control plane design. etcd, which persists cluster state, was also under elevated read pressure due to the volume of watch connections, which further degraded the API servers' ability to process anything.

Recovery was only possible gradually: as some nodes managed to process the DaemonSet rollback (reducing the number of active watch connections), pressure on the API servers decreased incrementally, freeing capacity to process more rollback operations. It was a self-limiting recovery process — each successful rollback enabled the next, but progress was slow because each operation competed with the remaining watch connections.

The operational lesson here is that any incident response plan for a large-scale Kubernetes environment must include remediation mechanisms that do not depend exclusively on the API servers. This can include: direct node access via SSH or SSM to manually remove DaemonSet pods; emergency scripts that operate outside the normal Kubernetes reconciliation cycle; or, in extreme cases, cordon and drain procedures that can be executed with partial control plane connectivity. None of these mechanisms replace prevention — but the absence of an alternative remediation plan turns a serious incident into a catastrophic one.