Lambda recursive loops: four guardrails compared in the Sovereign Cloud
Listen to article
Fernando's voiceFernando · 14:45
Powered by Amazon Polly + OmniVoice
Lambda recursive loop detection reached the European Sovereign Cloud on September 10, 2026. It cuts the chain at ~16 invocations, but only across Lambda, SQS, SNS and S3, and the alert can take 3.5 hours. I compare four guardrails — native detection, structural separation, a concurrency ceiling and an application-level hop counter — and say when each one is enough.
In 16 years operating financial platforms, the Lambda incident that cost me the most was not a logic bug — it was an environment variable holding the same value for source bucket and target bucket. The function wrote to the bucket that triggered it, S3 emitted another event, and within minutes the whole account's concurrency was busy doing useless work. Recursive loop detection reaching the AWS European Sovereign Cloud on September 10, 2026 is good news for anyone who moved regulated workloads to eusc-de-east-1. But the useful question isn't 'does Lambda protect me now?' — it's 'from which loop, with what delay, and what is still on me?'
What the Sovereign Cloud changes in the equation
Isolated partition: the European Sovereign Cloud is its own partition (aws-eusc), with a single region, eusc-de-east-1, in Germany, operated by EU residents. It is not 'one more region': it is a logically separate environment, with two Availability Zones instead of three and roughly 90 services at the January 2026 launch. Anyone arriving from eu-central-1 quickly finds everyday tools missing — IAM Identity Center, Security Hub, Inspector, CloudFront — and that quota increases usually go through a support ticket.
Why this matters for recursive loops: in a partition with fewer security services and less quota automation, the safety net you had in the commercial partition shrinks. Cost Anomaly Detection and billing alarms remain your last-resort defense — but each one has to be verified in the partition itself, not assumed. Native Lambda detection reached the EUSC almost three years after it debuted in commercial regions (July 2023 for SQS, SNS and Lambda; October 2024 for S3). That gap is the lesson: feature parity in a sovereign partition is not automatic. A workload that relied on a free, invisible guardrail in the commercial partition ran for months without it in the EUSC and nobody noticed. That is why I compare four mechanisms rather than recommending only the one that just arrived.
Same mechanics, same limit: the docs.aws.eu documentation carries the same minimum-SDK table and the same ceiling of approximately 16 invocations per chain. What differs is the surroundings.
How native detection works — and where it stops
Mechanism: Lambda piggybacks on the X-Ray trace header. When SQS, SNS or S3 deliver an event, it arrives annotated with a Lineage primitive — a resource hash plus a counter, something like Lineage=43e12f0f:5. When the function writes that event back to a supported service using a supported SDK, the counter increments. At approximately 16 invocations in the same chain, Lambda drops the next one and returns RecursiveInvocationException. No X-Ray active tracing is required and there is no charge.
Three limits the announcement page does not emphasize:
- Coverage by service. Only Lambda, SQS, SNS and S3. If the loop runs through DynamoDB Streams, EventBridge, Kinesis or Step Functions, the counter does not travel and nothing gets cut. The documentation says this in one sentence — it deserved a paragraph.
- Coverage by SDK. The counter only propagates if the function uses a minimum SDK version: Node.js 3.105.0 (v3), boto3 1.24.46, Java 2.20.81 on the 17 runtime, Go v2 1.57.0. A Python function that bundles an old boto3 in its deployment package silently leaves the protection.
- Depth, not breadth. The counter measures how many times that event invoked the function. A fan-out loop — one invocation producing three objects, each producing another invocation — is 16 levels deep, but every level multiplies the width. Sixteen hops at factor 3 is tens of millions of invocations before the cut. Detection bounds the chain; it does not bound the tree.
The alert delay: the Health Dashboard notification and the email can take up to 3.5 hours, and the email is at most one per function every 24 hours. The RecursiveInvocationsDropped metric is emitted immediately — that is where the alarm has to sit.
Anatomy of the loop and the four layers that cut it
The Lineage counter rides the event S3 → Lambda → S3; each guardrail acts at a different point of the chain.
- S3 bucket · ObjectCreated → Lineage=hash:n
- Lambda · SDK ≥ mínimo incrementa n
- SQS fila fonte · maxReceiveCount + DLQ
- 1. Detecção nativa · corta em ~16 saltos
- 2. Separação estrutural · prefixo/sufixo, bucket distinto
- 3. Reserved concurrency · limita a largura
- 4. Contador próprio · cobre DynamoDB/EventBridge
- CloudWatch · RecursiveInvocationsDropped (imediato)
- Health Dashboard + e-mail · até 3,5 h; 1 e-mail/24 h
- DLQ / on-failure destination · evento descartado
Four guardrails side by side
| Native detection (Terminate) | Structural separation | Concurrency ceiling + alarms | Application hop counter | |
|---|---|---|---|---|
| What it cuts | The chain, at ~16 invocations | The loop is never born | The breadth, not the chain | The chain, at the ceiling you set |
| Services covered | Lambda, SQS, SNS, S3 | Any, if the design allows | Any | Any that carries metadata |
| Prerequisite | SDK at minimum version; supported partition | Distinct buckets/queues/prefixes per role | Reserved concurrency per function + alarm on ConcurrentExecutions | Code: read/write hop in an attribute or idempotency table |
| Time until you know | Metric immediate; Health/email up to 3.5 h | You never need to know | Alarm in 1-5 min, depending on period | Own log/metric, immediate |
| Cost to maintain | Zero — but audit SDK version on every deploy | IaC discipline and PR review | Revisit ceilings as traffic grows | Own code: test, version, fix for years |
Decision matrix
Native detection (default `Terminate`)
- Free, on by default, no code
RecursiveInvocationsDroppedmetric is immediate- Stopped event goes to DLQ or on-failure destination
- Does not cover DynamoDB, EventBridge, Kinesis, Step Functions
- 16 fan-out hops is already millions of invocations
- Silently disappears with SDK below minimum
Mandatory floor, never the ceiling
Structural separation
- Prevents instead of detecting
- Independent of SDK, partition or service
- S3 prefix/suffix filter is configuration, not code
- Requires IaC discipline and review
- Does not protect against a bug that ignores the convention
The first design decision
Concurrency ceiling + alarms
- Bounds the financial blast radius of any loop
- Protects the rest of the account from the faulty function
- Does not cut the loop; only slows it
- Too low a ceiling becomes throttling on a legitimate peak
Mandatory on every triggered function
Application hop counter
- The only one covering DynamoDB Streams and EventBridge
- Configurable ceiling — 3, not 16
- Own code you maintain for years
- If metadata is lost mid-chain, the counter resets
Only when the loop crosses an uncovered service
The math that justifies the layers
Scenario: a 512 MB function, 200 ms per execution, triggered by ObjectCreated on the same bucket it writes to. Without reserved concurrency, Lambda scales to the account quota — 1,000 default concurrency in a commercial region; in the EUSC, confirm the value on your account, because quotas there usually go through a ticket. I use commercial-region list prices as a proxy; the EUSC has its own price table and you should check it before closing the number.
With no guardrail at all: 1,000 concurrent executions of 200 ms are 5,000 invocations per second. Each costs roughly US$ 0.0000019 in compute plus US$ 0.000005 for the S3 PutObject — the PUT costs more than the function. That is about US$ 0.035 per second, or US$ 125 per hour. In the 3.5 hours the Health Dashboard may take to warn you, the bill reaches US$ 440. On a Friday night it passes US$ 1,500 before anyone opens the email. Not catastrophic for a bank; enough to become an audit item and a CFO question.
With native detection only: the simple loop dies at 16 invocations — cents. The fan-out loop does not: 16 levels at factor 3 is 3^16 ≈ 43 million invocations, and detection cuts each individual chain only when it reaches level 16. The bill goes back to the previous scenario, now bounded by how long the tree takes to exhaust itself.
With reserved concurrency of 50: the worst case drops to 250 invocations per second — US$ 6 per hour. The 3.5-hour delay costs US$ 22. That is the number that lets me sleep: native detection cuts the chain; reserved concurrency bounds the breadth; neither alone bounds the cost.
The lesson: each guardrail acts on one dimension — depth, breadth or the loop's existence. Stacking the three is not redundancy; it is covering three different axes of the same problem.
Configuration I require before enabling a trigger
Detection state as code: PutFunctionRecursionConfig accepts Allow or Terminate, and the default is Terminate. Declare it explicitly in SAM or CloudFormation and add an SCP or pipeline policy denying lambda:PutFunctionRecursionConfig outside the platform role. Whoever needs Allow — a crawler that deliberately re-enqueues itself — justifies it in an ADR and gets low reserved concurrency in return. Remember that with Allow the RecursiveInvocationsDropped metric is not emitted: you lose the signal along with the protection.
Alarm on the right metric: RecursiveInvocationsDropped with Statistic: Sum, a 60-second period, threshold >= 1, one evaluation. Do not wait for the Health Dashboard. Next to it, ConcurrentExecutions per function with a threshold at 80% of reserved concurrency — that is the alarm that catches the fan-out and the DynamoDB loop that detection cannot see.
SQS as source: configure the DLQ on the source queue, not on the function. The function DLQ only applies to asynchronous invocation. And understand the interaction: after detection, the RecursiveInvocationException increments receiveCount on every retry, so the message only lands in the DLQ when the redrive policy's maxReceiveCount is exceeded. A maxReceiveCount of 1,000 keeps the message knocking for hours — 5 is reasonable for most cases.
S3 as source: distinct input and output prefixes, with the notification filter pointing only at the input one. A separate bucket when compliance allows; a prefix when it doesn't. Never the same environment variable for both — that is exactly what took me down.
SDK: pin the minimum version in package.json/requirements.txt and make the pipeline fail below it. It is the only way the protection does not silently vanish.
The loop detection cannot see is the most common one on data platforms
In data architecture, the dominant pattern is Lambda reading DynamoDB Streams and writing to the same table — enrichment, materialization, counters. That loop does not pass through SQS, SNS or S3, so the Lineage counter does not travel and Lambda cuts nothing. Here the real protection is a hop attribute on the item plus an event filter on the event source mapping (FilterCriteria dropping items where hop >= 3). It costs 20 lines of code and one filter condition; not having it costs a table growing exponentially until it blows through the WCU quota.
Anti-patterns I see in review
- Trusting detection as the only guardrail: it covers four services and cuts at 16 — the fan-out and the DynamoDB loop pass through untouched.
Allowin the base template: someone copied it from an intentional-recursion example and every new function is born without protection and without the metric.- Function DLQ instead of queue DLQ: the poisoned message never leaves the source queue and
receiveCountclimbs to amaxReceiveCountnobody reviewed. - Alarming on the Health Dashboard: 3.5 hours of delay is loop time, not response time. The alarm goes on
RecursiveInvocationsDroppedandConcurrentExecutions. - Assuming parity in the EUSC: the guardrail that existed in the commercial partition took almost three years to arrive. Verify every implicit feature in the target partition before moving the workload.
Every triggered function in my environment is born with three things in the template: an explicit RecursiveLoop: Terminate, reserved concurrency sized for the legitimate peak plus 30%, and an alarm on RecursiveInvocationsDropped >= 1 over 60 seconds. S3 prefix separation and a DLQ on the source queue are PR checks, not trust in anyone. The hard-won lesson came from that duplicated environment variable: the cost was not the night's money — it was the week explaining to compliance why a reconciliation function ran 4 million times without anyone knowing. A guardrail that warns in 3.5 hours serves the post-mortem; what serves the on-call shift is the one that cuts in 60 seconds.
Recommendation
Use native detection as the floor on every function, in every partition, and never as the argument for dropping the other layers. Stack structural separation (prevents), reserved concurrency with an alarm on ConcurrentExecutions (bounds breadth) and native detection with an alarm on RecursiveInvocationsDropped (cuts depth) when: the function is triggered by S3, SQS or SNS and writes to a service of the same kind. Add an application-level hop counter when: the loop crosses DynamoDB Streams, EventBridge, Kinesis or Step Functions — native detection sees none of them. Accept Allow only with an ADR, low reserved concurrency and a quarterly review. In eusc-de-east-1, add an explicit check that Cost Anomaly Detection and a billing alarm exist in the partition — the safety net you had in eu-central-1 does not travel with the migration.
References
Architecture, AWS, AI and market deep dives — straight to your inbox. Free.
No spam · unsubscribe anytime
Ask Fernando about this
Get a focused answer about this article from my AI assistant, grounded in my work.
Join the conversation
Sign in to comment
Verify your email to join in — you'll also get the newsletter. No password.
Keep reading
Architecture intelligence, in your inbox
Curated signals and original analysis on AWS, AI, distributed systems and the market — the way a solutions architect reads them.
- Curated AWS · AI · architecture · market signals
- New architecture studies & deep-dives when they ship
- Sharp summaries — depth without the noise
- No spam · double opt-in · unsubscribe anytime