Meta 2021: How a Maintenance Command Took Facebook Down for 6 Hours

To understand the incident, one must first understand how Meta structures its global network. The company operates one of the largest private networks in the world, with points of presence (PoPs) in dozens of countries connected by a proprietary backbone. This backbone is advertised to the internet via BGP — the inter-autonomous-system routing protocol that, simply put, tells the rest of the internet "these IP prefixes belong to Meta, send traffic here."

On the morning of October 4, an engineering team was performing maintenance work on the backbone infrastructure to audit available capacity. They used an internal network management tool that, among other functions, allows modifying router configurations across the network spine. The issued command was intended to perform a capacity scan — but due to a bug in the tool or an incorrect command configuration (Meta did not publicly specify which), the system interpreted the instruction as an order to disable BGP sessions on all backbone routers simultaneously.

The result was immediate and total: all of Meta's IP prefixes were withdrawn from global BGP. From the internet's perspective, Meta simply ceased to exist. No DNS resolver in the world could reach Meta's authoritative name servers to resolve facebook.com, instagram.com, or whatsapp.com — because those servers sat behind the same infrastructure that had just been disconnected. DNS itself wasn't broken; it was simply unreachable.

What made the situation even more critical was the collateral effect on Meta's own internal systems. Monitoring tools, remote access systems, and even physical data center badge readers depended on services that also went offline. Engineers attempting to diagnose the problem remotely lost access to the very tools needed to fix it. The team had to dispatch technicians physically to the data centers — a process that took hours, especially since physical access control systems were also partially affected.

One of the most technically interesting aspects of this incident is the specific role of DNS — and why it could not recover automatically even when parts of the infrastructure might have been restored more quickly.

DNS works in layers. When a recursive resolver (like Google's 8.8.8.8 or your ISP's resolver) needs to resolve facebook.com, it first queries root servers to find who is authoritative for .com, then queries TLD servers to find who is authoritative for facebook.com, and finally queries Meta's name servers directly. Those name servers (a.ns.facebook.com, b.ns.facebook.com, etc.) are the ones that return the actual IP addresses of application servers.

The problem is that Meta's authoritative name servers are IP addresses within Meta's own prefixes — the same ones withdrawn from BGP. When a resolver tried to reach them, packets simply had no route. This wasn't a DNS failure in the sense of "the server responded with an error" — it was a network reachability failure. The server was there, running, but no packets could reach it.

This has an important implication for DNS TTL (Time To Live). Meta's DNS records had relatively short TTLs — in the range of minutes. This means recursive resolvers worldwide quickly discarded their cached entries and attempted revalidation. Every revalidation attempt resulted in a timeout. The result was a storm of DNS queries failing globally, with no fallback possibility, because there were no alternative authoritative servers outside the affected perimeter.

A resilience practice that would have mitigated this would be maintaining authoritative DNS servers on separate ASNs (Autonomous System Numbers), with independent BGP prefixes, advertised by different transit providers. This is exactly what large managed DNS operators (like Cloudflare, AWS Route 53, NS1) do by design — they distribute their authoritative servers across multiple ASNs so that the failure of one doesn't compromise domain resolvability.

The natural question when hearing about this incident is: "if the problem was a command that disabled BGP sessions, why wasn't it enough to just re-enable the BGP sessions?" The answer reveals the second layer of systemic failure: the recovery control plane was inside the compromised perimeter.

Meta operates infrastructure of extraordinary scale — dozens of data centers, hundreds of points of presence, thousands of routers. Managing this infrastructure is done through sophisticated internal tools, accessible via corporate networks that, in turn, depend on the same backbone systems that were offline. When the backbone went down, remote engineers lost access to network management tools. It wasn't possible to simply open a terminal and re-enable BGP sessions — the tools to do so were inaccessible.

The solution was to physically dispatch engineers to data centers. This introduced significant delays for several reasons: the logistics of mobilizing people to physical locations takes time; some data centers are in remote or controlled-access locations; and, ironically, physical access control systems (badge readers, door authentication systems) also partially depended on services that were offline, requiring manual emergency authorization procedures.

Once physical access was established, engineers could connect directly to routers via console and manually re-enable BGP sessions. But even at that point, recovery had to be carefully staged. When Meta's services became reachable again, billions of clients (mobile apps, browsers, third-party systems) attempted to reconnect simultaneously. An abrupt restoration could generate a thundering herd — an avalanche of requests that would overwhelm systems still recovering. The team opted for a gradual restoration, progressively re-adding BGP prefixes and monitoring load at each step.

This experience led Meta to announce structural changes: clearer separation between the management control plane and production infrastructure, creation of out-of-band emergency access channels that don't depend on production services, and revision of validation processes for high-impact commands on network infrastructure.