ADR: Migrating Financial Workloads to EC2 R8i in Tokyo and Frankfurt

Institutional-grade financial workloads — order matching platforms, intraday risk engines, real-time price caches, settlement pipelines — share a characteristic that distinguishes them from conventional web systems: they are simultaneously memory-bound, network-bound, and latency-sensitive, and must operate within regulatory data residency windows that make region selection non-negotiable.

Before this expansion, R8i instances were only available in us-east-1, us-east-2, us-west-2, and ap-southeast-1 (via earlier announcements). This created a painful architectural asymmetry: the best compute nodes in the R family were geographically inaccessible to systems that must operate in Tokyo under Japanese FSA requirements or in Frankfurt/Ireland under GDPR and EBA guidelines.

The forces making this decision non-trivial are multiple. First, migration cost: moving an MSK Kafka cluster or a relational database primary node to a new instance type in production requires maintenance windows, performance regression testing, and SLO validation. Second, regression risk: the previous R6in/R6idn generation is still available and is a stable platform — the "if it isn't broken, don't fix it" argument carries real weight in environments where a P1 incident at 2 PM in Tokyo has regulatory consequences. Third, absolute cost: 48xlarge and 96xlarge instances carry price tags that justify rigorous ROI analysis, not reflexive adoption.

The R8i family is not monolithic — it has four variants with radically different I/O profiles, and the decision of which to use is as important as the decision to migrate.

R8in (network-optimized, no local NVMe): The most direct use case for Redis or Memcached-based real-time price cache systems. With 600 Gbps of network bandwidth and abundant memory, an R8in.48xlarge node can serve as the primary node of a self-managed ElastiCache cluster with synchronous replication to multiple AZs in Frankfurt without network being the bottleneck. For MSK (Kafka) pipelines that need brokers with high message ingestion capacity — think high-frequency market feeds with 10M+ events/second — the R8in eliminates the network contention that frequently appears as NetworkIn throttling on R6in instances under peak load.

R8idn (network + local NVMe): The natural candidate for self-managed Kafka brokers where you need extreme network throughput AND fast local storage for partition logs. In Tokyo, where exchange operators need Kafka brokers with P99 production latency < 2ms, the combination of 600 Gbps network with low-latency local NVMe eliminates two bottlenecks simultaneously.

R8ib (EBS-optimized, no NVMe): The canonical case here is Aurora PostgreSQL or RDS for position and reconciliation databases. With 300 Gbps of EBS bandwidth — the highest among non-accelerated instances — you can sustain heavy OLTP workloads with io2 Block Express volumes without EBS throughput being the limiting factor. This is relevant for T+1 settlement systems in Frankfurt that need to process batch reconciliations overnight within 2-3 hour windows.

R8idb (EBS + local NVMe): Institutional-scale data lakes that use S3 as the storage tier but need local scratch space for intermediate transformations — think Glue or Spark jobs that materialize 500GB+ risk datasets before writing aggregated results back to S3. The local NVMe here is not persistence; it is shuffle acceleration.

The decision is to adopt Option C — phased migration by workload criticality — with explicit SLO validation gates between each phase. The central reasoning is that in regulated financial environments, the speed of new infrastructure adoption is secondary to operational predictability. A P1 incident caused by a poorly executed migration in Tokyo at 9 AM JST (market open) has consequences beyond the technical SLA — there are regulatory notifications, incident reports to the FSA, and potential reputational impact.

The phase sequencing is deliberate. Phase 1 starts with real-time price caches (R8in) — these are the workloads with the highest migration ROI because they are stateless from a persistent data perspective, have trivial rollback (simply point the Auto Scaling Group back to the previous type), and the 600 Gbps network benefit is immediately measurable via NetworkIn/NetworkOut metrics in CloudWatch. Phase 2 addresses Kafka brokers — MSK or self-managed (R8idn) — where risk is higher because Kafka has state (partition logs), but the rolling broker restart process is well-documented and tested. Phase 3 covers database nodes (R8ib for Aurora/RDS, R8idb for data lake processing nodes) — these carry the highest risk and the highest EBS throughput benefit.

Each phase has a validation gate: 72 hours of production observation with defined SLOs (network latency P99, message throughput, EBS IOPS) before proceeding. If any SLO regresses more than 5% relative to the R6i baseline, the phase is rolled back and investigated before continuing. This is not excessive conservatism — it is the minimum operational standard for systems that process real financial transactions.

The instance decision is only the beginning. What differentiates a successful migration from a P1 incident are the specific configurations that frequently don't appear in product announcements.

Placement Groups and EFA: For 48xlarge, 96xlarge, metal-48xl, and metal-96xl sizes, EFA (Elastic Fabric Adapter) support is a performance multiplier for tightly coupled clusters — but it only works correctly when instances are in a Cluster Placement Group in the same AZ. In financial environments with multi-AZ requirements, this creates an architectural tension: you cannot have EFA and multi-AZ simultaneously in the same cluster. The solution is segmentation: use Cluster Placement Groups with EFA for the high-frequency processing tier (matching engine, risk calculation) and Spread Placement Groups for the persistence tier (Kafka brokers, database nodes).

IAM and KMS for instances with local NVMe (R8idn, R8idb): Local NVMe is NOT encrypted by default by AWS — encryption is the responsibility of the operating system or application. In financial environments under GDPR (Frankfurt) or FSA (Tokyo), data at rest on local NVMe must be encrypted. The correct approach is to use dm-crypt with LUKS on Linux, with the key derived from a secret stored in AWS Secrets Manager and retrieved via the instance IAM Role with an aws:SourceVpc condition. Never hardcode the key directly in user-data.

Auto Scaling and Instance Refresh: When using Launch Templates for migration, configure InstanceRefresh with MinHealthyPercentage: 90 and CheckpointPercentages: [25, 50, 75, 100] — this ensures the migration pauses at each checkpoint for validation before continuing. For Kafka clusters, configure MaxBatchSize: 1 to ensure only one broker is replaced at a time, preserving availability of all partitions during migration.

CloudWatch Alarms for SLO Gates: Validation alarms must monitor NetworkPacketsIn, EBSWriteBytes, EBSWriteOps, and — critical for Kafka — the custom metric UnderReplicatedPartitions via JMX exported to CloudWatch via OTEL agent. A value of UnderReplicatedPartitions > 0 for more than 60 seconds during a broker migration is an immediate rollback signal.

The availability of R8i instances in Tokyo and Frankfurt via Savings Plans, On-Demand, and Spot (as announced) opens three purchasing strategies with distinct trade-offs for financial environments.

Compute Savings Plans are the right choice for the capacity baseline of critical workloads — Kafka brokers, database nodes, price caches. The flexibility of Compute Savings Plans (vs. EC2 Instance Savings Plans) is crucial here: it automatically applies to R8in even if you purchased the plan while on R6in, without renegotiation. In terms of numbers: an R8in.48xlarge in Frankfurt with a 1-year Compute Savings Plan offers approximately 38-42% discount over On-Demand — on a 10-node cluster, this represents six-figure annual savings in USD.

On-Demand should be the entry position during migration phases — you don't want to be locked into a Savings Plan on an instance type that may need to be rolled back. Use On-Demand for the first 30 days post-migration of each phase, validating the performance baseline before converting to Savings Plan.

Spot has limited but real applicability in financial environments: batch risk processing jobs (overnight VaR calculation, stress testing), backtesting pipelines, and R8idb data lake transformations are legitimate Spot candidates. The key is using Spot with instance type diversification via capacity-optimized allocation strategy in the ASG — include R8idb, R8idn, and R6idn in the same pool to maximize Spot availability. For critical financial batch jobs, use the checkpointing pattern with Step Functions: every 15 minutes of processing, persist intermediate state to S3 with versioning enabled, so a Spot interruption doesn't discard more than 15 minutes of work.

An anti-pattern I see frequently: using Reserved Instances (RI) instead of Savings Plans for new instance generations. RIs are tied to a specific instance type and region — in a generation migration cycle, you get stuck. Compute Savings Plans are the correct choice for any workload that will go through a generation migration in the next 1-3 years.