ADR: EventBridge vs Kafka/MSK for Order Processing

The order system is the operational core of the platform. Every state transition of an order — created, paid, picked, shipped, delivered, cancelled — must be reliably propagated to multiple downstream consumers. A failure in that propagation can result in double billing, incorrect inventory, missed customer notification, or inconsistency in the fraud system.

The engineering team has solid experience with AWS managed services and Lambda, but no operational history with Kafka. The platform team has two senior engineers dedicated to infrastructure. Business pressure favors fast time-to-market, but the CTO has signaled that any order-loss incident carries high political cost.

The forces shaping this decision are:

• Ordering: individual orders must be processed in sequence. An order.cancelled event must not be processed before order.paid for the same orderId.
• Throughput: current volume is not extreme, but Black Friday and seasonal spikes require real elasticity.
• Replay: the analytics team and fraud team need to reprocess historical events when models change or bugs are fixed.
• Operational burden: with a small team, every hour spent tuning brokers is an hour away from features.
• Cost: MSK has fixed infrastructure cost regardless of usage; EventBridge charges per published event.
• AWS ecosystem integration: the existing stack is 100% AWS; native integrations reduce glue code.

The common temptation is to treat EventBridge as "simpler managed Kafka". That analogy is dangerous. EventBridge is a rules-based event router — it routes events from sources to targets based on content patterns. Kafka (and MSK) is a distributed commit log — it stores events durably and in order, allowing multiple independent consumers to read at their own pace.

These differences have concrete implications:

Ordering: EventBridge does not guarantee ordering between events. For an order system where the sequence created → paid → shipped is semantically critical, this requires consumers to implement their own ordering logic or the application to tolerate out-of-order events. MSK with partitioning by orderId guarantees ordering within a partition — which is exactly what we need.

Replay: EventBridge Archive allows event replay, but with limitations — replay targets the same destination and does not support fine-grained per-consumer filtering. MSK retains the log for a configurable period (days to weeks) and any consumer group can reposition its offset to any point in time, independently of others. For the analytics team that needs to reprocess 30 days of events without affecting the payments consumer, MSK is structurally superior.

Fan-out and coupling: EventBridge excels at decoupled fan-out — you add a new rule and a new target without touching the producer. MSK requires new consumers to create their own consumer groups, which is equally decoupled but requires more initial configuration.

Throughput and latency: EventBridge has a default throughput of 10,000 events/second per region (with soft limits adjustable via support), with latency typically under 1 second but no published latency SLA. MSK supports throughput of hundreds of MB/s with millisecond latency in appropriate configurations. For the current system volume, both are sufficient — but MSK has far greater headroom.

The core point is: EventBridge is the right choice when the problem is routing and service integration; MSK is the right choice when the problem is data streaming with ordering, replay, and multiple independent consumers. Order processing is the second case.

Partitioning strategy: The partition key must be orderId. This guarantees that all events for a specific order are written to the same partition and therefore consumed in order. The initial number of partitions for the orders topic should be calculated based on expected peak throughput divided by sustainable per-partition throughput (typically 1-10 MB/s depending on message size and broker configuration). For the estimated volume, 12 partitions is a reasonable starting point — it allows up to 12 parallel instances of a consumer group without rebalancing.

Retention and replay: Configure retention.ms to 604800000 (7 days) on the main topic. For the orders topic, consider 30 days given the analytics team's reprocessing requirement. Replay is operationalized via consumer group offset reset — the team must have documented runbooks for this operation, including how to pause the production consumer before replay to avoid unintentional duplicate processing.

Consumer criticality tiers: Not all consumers have the same SLA. Payments and inventory are critical — they should run on ECS/Fargate with auto-scaling based on consumer lag (EstimatedMaxTimeLag metric in CloudWatch MSK). Notifications and analytics can use Lambda via MSK Event Source Mapping, accepting higher latency in exchange for lower operational cost.

Dead Letter Queue: Each consumer must have a DLQ (SQS) for messages that fail after N retries. The retry schema should be exponential backoff with jitter. Messages in the DLQ should generate CloudWatch alerts and have a documented manual reprocessing process.

Rebalancing risk: In ECS consumers, Kafka rebalancing can cause processing pauses of seconds to tens of seconds depending on group size and rebalancing strategy. Configure partition.assignment.strategy=CooperativeStickyAssignor to minimize impact. Monitor RebalanceLatency and alert if it exceeds acceptable thresholds.

MSK Serverless vs Provisioned: MSK Serverless simplifies the start, but has limitations: maximum throughput of 200 MB/s write and 400 MB/s read per cluster, no control over partition count (automatically managed), and cost per capacity unit that may be higher than Provisioned at high volumes. Migration to Provisioned should be planned when monthly data volume makes the cost per capacity unit unfavorable — typically above a few TB/month (estimate).