Roblox 2021: 73 Hours of Downtime, Consul and the Load Effect

Roblox operates a massive and predominantly on-premise infrastructure. Unlike most platforms of similar scale that have migrated to public cloud, Roblox maintains its own and colocated datacenters, meaning the entire service discovery, orchestration, and distributed coordination layer is an internal responsibility. At the center of this infrastructure sits HashiCorp Consul — used for both service discovery and health checking of thousands of services.

In the days leading up to the incident, Roblox's infrastructure team enabled a new Consul feature: telemetry streaming, designed to reduce polling load on Consul agents and servers by replacing the pull model with a push-based streaming model. The intention was good — reduce overhead. The timing was fatal.

The traffic context matters here. Late October coincides with Halloween, one of the most significant seasonal peaks for Roblox. The platform was processing a volume of sessions and events substantially above average. More sessions mean more service registrations, more health checks, more read and write operations on the distributed state maintained by Consul.

Consul, in its default production configuration for large clusters, uses BoltDB as the storage backend for Raft state. BoltDB is an embedded key-value database, written in Go, with a fundamental architectural characteristic: it uses a single writer, multiple readers model with explicit locking. Writes acquire an exclusive lock on the entire database file. This works well under normal loads, but under elevated contention — many concurrent writes or long writes blocking reads — throughput degrades non-linearly.

The telemetry streaming feature, once enabled, significantly increased the volume of write operations to BoltDB. Combined with elevated Halloween traffic generating more health check and service registration events, the Consul cluster began experiencing increasing latencies in Raft operations. High Raft latencies mean health checks start failing due to timeouts. Failing health checks cause services to be marked as unhealthy. Unhealthy services are removed from Consul DNS. And at that point, the cascade effect becomes irreversible without manual intervention.

A legitimate question is: why can't a Consul cluster simply be restarted? The answer reveals the real complexity of operating distributed coordination infrastructure at scale.

First, Consul's state is the platform's state. Thousands of services depend on Consul to know where their peers are, which instances are healthy, and how to route traffic. An abrupt cluster restart without a state recovery plan can result in an even worse situation: services attempting to register simultaneously, generating a write storm that would reproduce exactly the original problem.

Second, diagnosis was not immediate. The Roblox team needed to correlate BoltDB behavior with the newly enabled telemetry streaming. This is non-trivial in an environment with hundreds of variables. BoltDB contention logs are not, by default, prominently exposed. Raft latency is a symptom that can have multiple causes — network problems, saturated CPU, slow disk. Isolating streaming as the amplifying factor required careful investigation.

Third, recovery needed to be gradual. Once the cause was identified and streaming disabled, the Consul cluster needed to be rebuilt in a controlled manner. This means bringing nodes up one by one, verifying Raft state convergence, and only then allowing services to begin re-registering — in an order that doesn't reproduce the write storm. With thousands of services and workloads managed by Nomad, this process is inherently slow.

Finally, there is the human and organizational factor. A 73-hour incident requires team rotation, constant stakeholder communication, decisions under pressure, and accumulated fatigue. The quality of technical decisions in a prolonged incident is directly affected by these factors. Roblox, to the team's credit, published a detailed and honest post-mortem — which is rare and valuable.

The post-mortem published by Roblox in January 2022 details a robust set of corrective actions. I'll analyze them critically, because not every remediation action is equally effective.

Migration from BoltDB to bbolt with concurrency improvements: The original BoltDB was archived; the bbolt project (fork maintained by the etcd/CoreOS community) offers incremental improvements, but the fundamental single-writer limitation persists. The real remediation here is migration to alternative backends — Consul supports backends like etcd or can use WAL-based storage. Roblox also evaluated and implemented changes to Consul configuration to reduce the frequency of unnecessary write operations.

Disabling telemetry streaming and reviewing the feature rollout process: This is the most direct and correct action. Streaming was disabled and the process for enabling new features in production was revised to include impact assessment on I/O operations and specific load testing for the storage backend.

Improving Consul cluster observability: Before the incident, BoltDB contention metrics were not prominently monitored. Post-incident, specific alerts for Raft operation latency, BoltDB write queue size, and lock acquisition time were added. This is essential — you cannot respond to what you cannot see.

Reviewing Consul dependency architecture: The incident exposed that Consul was a single point of failure for the entire platform. Post-incident, Roblox worked to reduce the critical dependency surface — implementing more aggressive DNS caching on clients, circuit breakers for services that depend on service discovery, and graceful degradation plans for when Consul is degraded but not completely unavailable.

What I would do differently: Migration to a storage backend with a better concurrency model should be the number one priority — not an incremental improvement to bbolt. For a Consul cluster of Roblox's size, the ideal backend would be WAL-based with support for concurrent writes. Additionally, I would implement Consul-specific chaos engineering — injecting latency into BoltDB, simulating lock contention — before any feature rollout that affects I/O volume.