Datadog (2023): how a systemd security patch took down 5 regions simultaneously

On the morning of March 8, 2023, Datadog began receiving alerts of severe degradation across multiple products — metrics ingestion, logs, APM, and synthetics. The most disturbing characteristic of the incident was not the failure itself, but its simultaneity: five independent production regions began exhibiting problems at the same time, which immediately ruled out localized failure hypotheses and pointed to a common cross-cutting cause.

The investigation revealed that Ubuntu, the OS of the compute nodes, had automatically applied a security update to the systemd package. Specifically, the update affected systemd-networkd, the daemon responsible for managing network interfaces at the OS level. Upon being updated, systemd-networkd was restarted — expected and documented behavior for applying security patches to network components.

The critical problem lay in the interaction between this restart and Cilium, the CNI plugin (Container Network Interface) used by Datadog to manage Kubernetes pod networking. Cilium operates in kernel space via eBPF and maintains connectivity state that depends on stable network interfaces. When systemd-networkd restarted and reconfigured the interfaces, Cilium lost pod connectivity state — effectively severing communication between running workloads, without the pods themselves being restarted or reporting failure.

The result was a silent and diffuse failure: pods running, nodes apparently healthy, but with no ability to communicate. Ingestion pipelines stopped receiving data. Internal control services lost connectivity with each other. The Kubernetes control plane continued functioning, but the data plane was effectively broken.

Immediate remediation involved three parallel fronts. First, the team blocked new auto-updates on all remaining nodes to prevent the problem from spreading to nodes not yet affected. Second, the recovery process for already-affected nodes began: in some cases, restarting the Cilium agent was sufficient to restore connectivity after systemd-networkd had stabilized; in others, nodes needed to be drained and replaced via rolling replacement, which is an inherently slow process when dealing with tens of thousands of nodes.

Scale was the primary obstacle to rapid recovery. In a localized incident — say, a single region with hundreds of nodes — the drain-and-replace procedure would complete in minutes. With tens of thousands of nodes distributed across five regions, the same procedure took hours. This illustrates an important point: the blast radius of a change affects not only the severity of the initial impact, but also the recovery time. The larger the blast radius, the slower and more costly the remediation.

In the medium term, Datadog implemented a set of structural controls. Unrestricted auto-update was replaced by a managed OS update pipeline, with progressive rollout per region and health metric observation between each step. The interaction between systemd-networkd and Cilium was documented and added to OS patch validation criteria. Compatibility testing between OS updates and the CNI plugin was now executed in staging before any production rollout. Additionally, faster detection mechanisms for node-level network connectivity failures were implemented — specifically to capture the pattern of 'pod running, node Ready, but no connectivity', which is a false positive from the kubelet's perspective.

One of the fundamental principles of resilience in distributed systems is failure domain isolation. Distinct geographic regions exist precisely so that a failure in one does not propagate to others. Datadog, as an observability company, certainly had this principle embedded in its product architecture. The problem was that the infrastructure update pipeline did not respect those same boundaries.

This reveals a common asymmetry in mature engineering organizations: significant investment in failure isolation at the application plane — circuit breakers, bulkheads, retry with backoff, multi-region deployments — but the underlying infrastructure plane may have global couplings that are not treated with the same rigor. unattended-upgrades did not know it was operating in a multi-region system with blast radius requirements. It was just a system daemon doing its job.

The lesson here is that any mechanism that can modify state across multiple production nodes must be treated as a deployment, with all the guarantees that implies: canary release, progressive rollout, health gates between steps, and rollback capability. This applies to OS patches, agent updates, configuration changes via Ansible/Chef/Puppet, and any other form of infrastructure change at scale. The distinction between 'application change' and 'infrastructure change' is operational, not architectural — from a risk perspective, both require the same controls.

There is also an aspect of observability of the change process itself that deserves attention. Datadog is an observability company — its products are used by other companies to detect exactly this type of problem. The irony of the incident is that root cause detection took time because the monitoring system itself was partially degraded by the failure. This reinforces the importance of having out-of-band observability — monitoring mechanisms that do not depend on the infrastructure they are monitoring.