Postmortem: When AI Meets Resilience — AWS Resilience Hub and SRE

The original AWS Resilience Hub was, essentially, an audit tool: you mapped your application, defined a target RTO/RPO, and the service compared your architecture against predefined resilience policies, generating a score and a list of recommendations. Useful, but passive. The new generation adds three capabilities that change the nature of the product.

First, AI-assisted failure mode analysis: instead of just checking whether you have Multi-AZ enabled on RDS, the system can now reason about failure chains — for example, identifying that a Lambda function with reserved concurrency of 100 can become a bottleneck when upstream an MSK consumer group suffers a rebalance and triggers a processing spike. This is causal analysis, not just configuration compliance.

Second, contextual runbook generation: Resilience Hub can now generate incident response runbooks based on the actual topology of your application, not generic templates. A runbook generated for an EKS cluster with Karpenter is different from one for ECS Fargate — and that difference matters at 2 AM during an incident.

Third, integration with AWS Systems Manager and FIS (Fault Injection Simulator): the cycle now closes — you analyze, inject failure, observe, and the model updates its risk assessment. This transforms Resilience Hub from a point-in-time audit tool into a continuous learning loop. The question is: is that loop reliable enough for critical systems?

Let me be concrete with a scenario I saw in production on a payments platform. A Kafka (MSK) consumer group processing card transactions started accumulating lag at 2:32 PM on a Friday. The cause was not obvious: a deploy of an upstream Lambda function had increased the average message size from 2KB to 18KB, without changing throughput in messages per second. The result was that the MSK broker started experiencing memory pressure, which caused frequent consumer group rebalances, which in turn increased lag, which triggered a processing latency SLO alarm.

An AI-powered failure mode analysis system would potentially have identified the correlation between the Lambda deploy and the message size increase — if the system had access to deploy metadata and MSK message size metrics (kafka.consumer.fetch-size-avg). That is the critical conditional: the value of AI analysis is directly proportional to the richness of available telemetry.

The generated runbook would likely have suggested increasing consumer concurrency or scaling brokers — both reasonable actions but ones that would not resolve the root cause. The correct remediation was to roll back the Lambda deploy and add message size validation on the producer. This requires business context and knowledge of the deploy chain that Resilience Hub, by itself, does not have access to.

The lesson is not that the tool is useless — it is that it is a first triage layer, not a substitute for resilience engineering. The real value is in reducing diagnosis time from 40 minutes to 8 minutes, not in eliminating the engineer from the loop.

Integrating Resilience Hub with generative AI into a financial system requires a layered approach, not wholesale adoption. Here is how I would structure it.

Layer 1 — Context Enrichment: The value of AI analysis is a direct function of application metadata quality. This means instrumenting every service with business tags (cost-center, criticality, settlement-window), propagating trace IDs across all service boundaries (including MSK messages via Kafka headers), and publishing custom business metrics to CloudWatch — not just infrastructure metrics. A model that does not know your settlement window is 10 PM to 11 PM will suggest remediations that violate operational contracts.

Layer 2 — Human Review Gate with SLA: Every AI-generated runbook must go through a review process with a defined SLA. For P1 incidents, the review SLA is 5 minutes — which means the runbook needs to be readable, specific, and actionable in 5 minutes. Generic or ambiguous runbooks should be automatically rejected by a quality validation process before reaching the engineer.

Layer 3 — Execution with Controlled Blast Radius: Approved automations must be executed with IAM conditions that limit blast radius. For example, a Lambda rollback automation should have permission only for functions tagged auto-remediation: enabled and only within a specific account — never with cross-account permissions without additional approval. Use aws:ResourceTag conditions in IAM and ssm:AutomationAssumeRole with minimum-scope roles.

Layer 4 — Feedback Loop with Audit Trail: Every automation execution must be recorded in CloudTrail with business context, and the result must be fed back to Resilience Hub for score update. This creates an auditable history that is essential for compliance in regulated financial systems.

There is a problem that most analyses of AI-powered Resilience Hub ignore: hallucination in high-pressure incident context. Large language models have a documented tendency to generate plausible but incorrect outputs when context is ambiguous or incomplete. In a development environment, this is an inconvenience. In a P1 incident at 3 AM with a payments SLO at risk, this can be catastrophic.

The specific risk mechanism is as follows: the model receives a partially stale dependency graph (because the last Resilience Hub scan was 6 hours ago and there have been 3 deploys since then), CloudWatch metrics showing degradation but without causal context, and a history of previous incidents that may or may not be relevant. With this input, the model generates a failure mode analysis that appears authoritative but is based on an outdated view of reality.

The mitigation is not to disable AI — it is to implement freshness gates: the system should refuse to generate analysis if the last topology scan is older than N minutes (I would use 30 minutes for critical systems), if there are unreconciled deploys since the last scan, or if telemetry coverage is below a defined threshold (for example, less than 80% of critical services with active traces in X-Ray).

Furthermore, the generated runbook must explicitly include the assumptions under which it was generated — service versions, topology state at the time of analysis, metrics considered. This allows the on-call engineer to quickly assess whether the assumptions are still valid before executing any step. This reasoning transparency is what separates a usable AI system from a dangerous AI system in production.