Discord: How to Store Trillions of Messages — Cassandra → ScyllaDB Migration Teardown

Discord adopted Apache Cassandra early, and the choice made sense: wide-column data model, horizontal scalability without a single point of failure, and a mature ecosystem. For a messaging platform where each Discord server (guild) has channels, and each channel has a message history that needs to be read efficiently by timestamp, Cassandra's partition model is almost perfect in theory. You partition by (channel_id, bucket) and cluster by message_id (Snowflake, monotonically increasing), which gives time-range reads within a single partition. Elegant.

The problem wasn't the data model. It was the JVM.

Cassandra runs on the JVM and, with large heaps — necessary to keep indexes and caches in memory for trillions of messages — the garbage collector becomes the enemy. GC pauses of seconds are not hypothetical at this scale; they are routine. Discord reported completely unpredictable p99 and p999 latencies, with spikes reaching seconds on operations that should take milliseconds. On a real-time communication platform, this is perceptible. Users see message history take a long time to load. The experience breaks.

There is also the compaction problem. Cassandra uses LSM trees (Log-Structured Merge-tree) internally, and the compaction process — necessary to maintain read performance over time — competes with production operations for I/O and CPU. In large clusters under continuous load, compaction windows become operational risk events. The Discord team described hot nodes causing tail pressure on the entire cluster, because Cassandra has no per-operation resource isolation — a slow node contaminates the cluster's latency percentile.

The obvious alternative would be to tune the GC, use ZGC or Shenandoah, reduce the heap, fragment the clusters. The team tried these approaches. The problem is that they buy time, they don't solve the root cause: you are running a mission-critical database on top of a runtime with non-deterministic garbage collection. At some point, tuning engineering becomes accumulated technical debt, and the cost-benefit ratio inverts.

The core data model remained essentially the same after the migration — and that is an important point. Discord did not need to redesign the schema to move from Cassandra to ScyllaDB, because ScyllaDB is compatible with the CQL (Cassandra Query Language) protocol. The messages table uses (channel_id, bucket) as the partition key, where bucket is derived from the message timestamp to avoid unbounded partitions (an active channel over years would have a gigantic partition without bucketing). The clustering key is message_id, a Snowflake ID that encodes the timestamp, guaranteeing native chronological ordering and efficient range reads.

What changed was the engine underneath. ScyllaDB is a reimplementation of Cassandra in C++ with the Seastar framework, which uses a shard-per-core model: each CPU core has its own data set and its own I/O queue, with no shared memory between cores. This eliminates the lock contention that is endemic in multi-threaded systems with a shared heap. There is no GC. Memory management is deterministic. The practical result is that tail latencies — p99, p999 — collapse to much smaller values and, more importantly, much more predictable ones.

The rewrite of data services in Rust was complementary and equally deliberate. The previous Python/Go services introduced their own latency variability: Go's GC, while much better than the JVM's, still introduces pauses; Python has the GIL. Rust offers deterministic performance without a GC runtime, with compile-time memory safety. For a service that is on the critical path of every message read and write, the choice makes clear technical sense. The cost is the learning curve and slower development velocity — a trade-off Discord evaluated as acceptable given the load profile.

The migration itself was conducted with dual-write and backfill. During the transition phase, new messages were written to both Cassandra and ScyllaDB. A background migration process read historical data from Cassandra and wrote it to ScyllaDB. When data parity was confirmed, read traffic was gradually shifted to ScyllaDB. This approach is conservative and correct for mission-critical data — you never do a big-bang migration on production message storage. The risk of data loss or inconsistency is too high. The cost of dual-write (temporarily doubling write load) is a reasonable price for operational safety.

The rewrite of data services in Rust deserves separate analysis because it reveals an important engineering philosophy: when you have an aggressive tail latency requirement, every layer of the stack needs to be evaluated as a potential source of jitter. Discord didn't just swap the database — they audited the critical path end-to-end and identified that the service layer was also a source of variability.

The technical argument for Rust in this context is solid. Rust's ownership model guarantees no runtime garbage collection — allocations and deallocations are deterministic and predictable. For a service processing millions of read and write operations per second, the absence of GC pauses — even the sub-millisecond ones from Go — translates to lower and more stable latency percentiles. Additionally, Rust has performance close to C/C++ without the memory safety risks that would make C++ impractical for a product team.

The real cost of Rust is development velocity and the size of the engineer pool. Rust has a significant learning curve — the borrow checker is a real obstacle for developers coming from GC languages. For a company of Discord's size, this means the base of engineers who can contribute to these services is smaller, and the onboarding time for new engineers is longer. This is a trade-off that makes sense for low-level infrastructure services with extreme performance requirements, but would be questionable for high-level business services where iteration speed matters more than tail latency.

What Discord did correctly was to limit Rust usage to the critical path — the data services that sit directly in front of the database. They did not rewrite everything in Rust. This containment is mature: you apply the most expensive tool (in terms of development cost) where the return is highest.