GitHub 2018: 43 Seconds of Partition, 24 Hours of MySQL Split-Brain

On October 21, 2018, at 16:00 UTC, GitHub's network engineering team initiated the replacement of a network device in the primary datacenter located on the US East Coast. The procedure was routine — the kind of maintenance that happens dozens of times a year in large-scale infrastructure. During the swap, there was a 43-second connectivity interruption between the US East and US West datacenters.

This interval was sufficient for the GitHub Orchestrator, the MySQL topology management tool used by GitHub (which GitHub itself helped develop as an open source project), to interpret the loss of connectivity to the MySQL primary in US East as a real node failure. Following its automatic promotion logic, Orchestrator elected a replica in US West as the new primary and redirected write traffic to it.

The problem: the original primary in US East had not failed. It was perfectly operational — just temporarily isolated. When connectivity was restored after 43 seconds, the system found itself in a classic split-brain state: two MySQL nodes believing themselves to be the legitimate primary, both accepting writes, both silently diverging in terms of data.

From that moment on, every write arriving at the US East primary and every write arriving at the new US West primary created conflicting versions of reality. Issues created on one side didn't appear on the other. Pull requests updated in one region diverged from the other. Memcached, which serves as a caching layer on top of MySQL, began serving inconsistent data — because the underlying data was inconsistent.

GitHub's team quickly detected that something was wrong — replication alerts fired within minutes. But the fix was not trivial. You can't simply "reconnect" two diverging primaries and expect MySQL to resolve conflicts. MySQL's binlog-based replication is unidirectional and has no native conflict resolution. To restore consistency, the team needed to identify the exact point of divergence in the binlogs, discard writes from the side being demoted, and rebuild the replication topology — all while the production site was still serving traffic.

When GitHub's team confirmed the split-brain, the first critical decision was: which side of the divergence to preserve? This is not a trivial question. In a system with millions of active users, both sides of the split-brain had legitimate writes from real users. Choosing US East as the source of truth meant discarding writes that had happened in US West after the promotion — writes that users had made in good faith and expected to be persisted.

The team decided to use US East as the source of truth, based on the fact that it was the original primary and had the longest and most reliable replication history. Conflicting writes from US West were identified through binlog analysis — a manual and meticulous process of comparing transaction logs from both sides to find the exact point where the histories diverged.

The recovery process was deliberately slow and cautious. The team couldn't simply run CHANGE MASTER TO and hope for the best. Each MySQL cluster needed to be verified individually. Memcached needed to be completely invalidated to prevent stale data from continuing to be served after resynchronization. Services that depended on eventual consistency needed to be paused or placed in read-only mode while the topology was rebuilt.

An important detail the official post-mortem reveals: during the recovery process, the team disabled Orchestrator to prevent it from making further automatic decisions while the topology was in an inconsistent state. This is an implicit acknowledgment that automation, at that moment, was an additional risk, not an aid. The recovery was essentially manual — experienced engineers navigating MySQL binlogs at 2 AM, making surgical decisions about which transactions to preserve.

The total cost was approximately 24 hours of service degradation, with varying impact across different features. Some features were completely unavailable; others operated in degraded mode with the possibility of inconsistency. GitHub's public communication during the incident was exemplary — frequent updates on the status page, transparency about the nature of the problem, and a detailed post-mortem published 9 days later.

It's important not to demonize Orchestrator. It did exactly what it was configured to do: detect primary unavailability and promote the best available replica. In 99% of real failure scenarios — where the primary actually died, where a disk failed, where a process hung — this behavior is correct and saves operational lives.

The problem is that Orchestrator, like any automatic failover system, operates under an implicit assumption: if it can't reach the primary, the primary is dead. This assumption is reasonable but not universal. Transient network partitions — especially during planned maintenance — create exactly the scenario where the assumption fails.

The technical solution to this problem has been known for decades in distributed systems literature: quorum-based fencing. Before promoting a replica, the failover system must obtain confirmation from a majority of nodes (or an external arbiter) that the primary is truly unreachable — and, ideally, must execute a fencing mechanism to ensure the old primary cannot accept writes even if it comes back online.

In the MySQL context, this can be implemented in several ways:

• STONITH via IPMI/iDRAC: physically power off the node via management interface before promotion
• VIP (Virtual IP) revocation: revoke the primary's virtual IP before assigning it to the new one
• Semi-synchronous replication with rpl_semi_sync_master_wait_for_slave_count: ensure writes are only acknowledged when at least N replicas have received them
• MySQL Group Replication or InnoDB Cluster: topologies with native consensus based on Paxos/Raft

GitHub, after the incident, implemented several of these improvements. The post-mortem specifically mentions adding fencing checks and revising Orchestrator timeouts to be more conservative in maintenance environments.

There is also a lesson about multi-datacenter topology: in an active-active or active-passive configuration with replicas in multiple regions, the failover system needs to be topology-aware — it should not promote a replica in a different region without considering replication lag and the risk of divergence. A replica in US West is, by definition, a few milliseconds behind the primary in US East. Promoting it as primary during a transient partition guarantees that writes made to the original primary during those milliseconds will be lost or conflicting.