Design Doc: Multi-Region Active-Active Payments API

Most systems that self-declare as 'highly available' are, in practice, active-passive with manual or semi-automatic failover. For a payments API, this is insufficient for reasons that go beyond technical SLAs.

First, the regulatory context. Central banks and card schemes (Visa, Mastercard) require RTO measurable in seconds, not minutes. In Brazil, the Central Bank monitors PIX availability with minute-level granularity and fines participants that exceed downtime thresholds. This transforms resilience from an engineering decision into a compliance obligation.

Second, the nature of the data. Payments are not simple idempotent operations. A transfer involves debiting one account, crediting another, recording in a ledger, sending notifications, and frequently integrating with external systems (clearing houses, correspondent banks). If a region fails mid-saga, state can become inconsistent in ways that are hard to detect and dangerous to correct automatically.

Third, the split-brain problem. In a genuine active-active system, two regions can receive concurrent requests for the same resource — for example, two simultaneous withdrawals from the same account processed in different regions before replication propagates the updated balance. Without a robust conflict resolution mechanism, the system may authorize transactions that should be denied, resulting in fraud or overdraft.

This document does not propose 'high availability'. It proposes a system that maintains transactional correctness even under region failure, with minimal added latency on the happy path and predictable degraded behavior on the failure path.

The design starts from a premise I consider non-negotiable for financial systems: correctness precedes availability. This means I prefer to reject a transaction with a clear error rather than process it incorrectly. With that established, the design is organized into three layers.

Layer 1: Routing and Failover (Route 53 ARC + Global Accelerator)

Amazon Route 53 Application Recovery Controller (ARC) is the central control plane for failover. It maintains continuous readiness checks for each recovery cell (one per region) and allows programmatic failover with a single API call. Global Accelerator routes traffic to the nearest region based on network latency, with endpoint-level health checks every 10 seconds.

The critical decision here is not to use pure DNS-based failover. DNS TTLs, even with low values (60s), combined with intermediate resolver caches, make failover timing unpredictable. Global Accelerator operates at the network layer (Anycast), which eliminates this problem and enables failover in seconds.

Layer 2: Transaction Processing (EKS + Saga Pattern)

Each region runs an EKS cluster with the payment microservices. The Choreographed Saga pattern is used for transactions involving multiple services, with events published to Amazon EventBridge. Each saga step is idempotent and records its state in DynamoDB Global Tables with conditional writes.

Idempotency is implemented via an idempotency key in each request header (UUID v4 generated by the client). The authorization service checks for the key's existence in DynamoDB before processing — if it already exists, it returns the previous result. This guarantees exactly-once even when the client retries after a timeout.

Layer 3: Persistence and Replication (Aurora Global + DynamoDB Global Tables)

The financial ledger (balances, confirmed transactions) lives in Amazon Aurora Global Database with a writer instance in us-east-1 and readers in sa-east-1 and us-west-2. Aurora Global's typical replication lag is < 1 second, meeting the defined RPO.

Short-term operational state (idempotency keys, distributed locks, sessions) lives in DynamoDB Global Tables, which uses asynchronous multi-master replication. Here is the central trade-off: DynamoDB Global Tables uses last-writer-wins (LWW) based on timestamp as the default conflict resolution policy. For idempotency keys this is acceptable — the first write wins in practice because the key TTL is long enough. For balances, it is not acceptable — which is why balances live in Aurora, not DynamoDB.

For the Aurora failover case (promoting reader to writer), Route 53 ARC coordinates the sequence: (1) stop writes in the primary region, (2) wait for zero-lag replication confirmation, (3) promote reader, (4) update endpoints. This process typically takes 60–90 seconds, which only meets the 30-second RTO if step 2 is eliminated — which requires accepting potential sub-second data loss. My recommendation is to keep the real RTO at 90 seconds for database failover and be transparent about this with stakeholders.

Conflicts in active-active payment systems are not theoretical — they happen every time there is replication latency and concurrent traffic. The question is not whether they will occur, but how the system behaves when they do.

Typical conflict scenario: User has a balance of $1,000. Makes two withdrawals of $800 almost simultaneously — one processed in us-east-1, another in sa-east-1, before replication propagates the updated balance. Without protection, both are authorized, resulting in a negative balance of -$600.

Our approach: Balance Reservation with Distributed Lease

For transactions involving balance (withdrawals, debit transfers), we implement a distributed lease mechanism using DynamoDB with conditional writes. Before processing, the authorization service attempts to acquire an exclusive lease for the account with a 5-second TTL. The conditional write guarantees that only one region acquires the lease at a time.

ConditionExpression: attribute_not_exists(lease_owner) OR lease_expires < :now
UpdateExpression: SET lease_owner = :region, lease_expires = :now + 5s, version = version + 1

If the lease cannot be acquired (another region is processing), the request returns HTTP 409 with Retry-After: 1. The client retries, and on the second attempt the previous lease has already expired or been released.

Why not use Two-Phase Commit (2PC)?

2PC guarantees distributed atomicity, but introduces a transaction coordinator that becomes a single point of failure and adds latency proportional to the number of participants. For a payments API with a 200ms P99 SLA, cross-region 2PC is not viable — the round-trip between us-east-1 and sa-east-1 alone is ~120ms.

Why not use Paxos/Raft?

Consensus protocols like Raft guarantee strong consistency, but require majority quorum for each write. With three regions, this means each transaction needs confirmation from at least 2 regions — again, unacceptable latency for the happy path.

The pragmatic solution: Separate the data domain by access pattern. Data requiring strong consistency (balances, ledger) lives in Aurora with a single writer at a time. Data tolerating eventual consistency (idempotency keys, audit events) lives in DynamoDB. The distributed lease via DynamoDB is the coordination mechanism — it is eventual, but the short TTL (5s) limits the conflict window to an interval acceptable for the business.