Secure Multi-Tenant RAG: ADR for Two-Layer Authorization

In conventional single-tenant RAG systems, access control is solved at the application layer: you authenticate the user, filter documents by ownership, and pass context to the model. The problem begins when you scale this to dozens of tenants with distinct SLAs, heterogeneous permission schemas, and regulatory obligations such as LGPD, SOC 2, and PCI-DSS.

The central force here is context contamination risk. In a RAG architecture, retrieval determines what the model sees. If the vector search mechanism returns document chunks from another tenant — due to metadata error, filter failure, or prompt injection — the model processes and potentially reproduces confidential information. This is not hypothetical: in penetration tests I have conducted on enterprise RAG platforms, I was able to extract context from other tenants by manipulating the embedding query before any application-layer filter was applied.

The second force is intra-tenant permission heterogeneity. Within a single financial tenant, a junior analyst should not see the same documents as a portfolio manager. This requires attribute-based access control (ABAC), not just role-based. Most RAG implementations I see in production ignore this dimension and treat the tenant as the atomic authorization unit — which is insufficient for any regulated environment.

The third force is the model's own attack surface. LLMs are susceptible to prompt injection. If a malicious document in the corpus instructs the model to ignore restrictions or reveal system context, authorization that lives only in the prompt is not sufficient. The defense must exist before the model sees any token.

Beyond security forces, there are three operational forces that shaped this decision. The first is regulatory auditability. In financial environments, every access decision must be recorded with enough context to respond to an audit: who accessed, which document, under which policy justification, at which timestamp. This rules out approaches where authorization is implicit in application code without a structured trail.

The second force is inference latency. A RAG cycle already carries embedding latency (typically 50-150ms on amazon.titan-embed-text-v2), vector search (20-80ms on OpenSearch Serverless with k-NN), and model inference (500ms-3s depending on the model and context size). Adding an authorization layer that introduces more than 30-50ms of overhead is unacceptable for product SLAs. This rules out solutions requiring synchronous calls to slow external systems on the critical path.

The third force is policy management at scale. With dozens of tenants and hundreds of distinct roles, maintaining authorization policies as application code becomes unmanageable. You need a centralized, versioned, and auditable policy store with native ABAC support — not a DynamoDB table with authorization logic scattered across Lambda functions.

These three operational forces, combined with the security forces, eliminated the simpler approaches and forced me to consider a defense-in-depth pattern with specialized components for each layer.

I adopted the defense-in-depth pattern with two independent authorization layers. Layer 1 is Amazon Verified Permissions (AVP) with Cedar policies, operating before retrieval. Layer 2 is the native metadata filter of Bedrock Knowledge Base, operating at vector search time. The independence between layers is the critical point: a Cedar misconfiguration does not disable the metadata filter, and vice versa.

The choice of Cedar as the policy language is deliberate. Cedar is a policy language with formal semantics and a deterministic evaluation model — unlike Rego, which allows constructs that can produce unexpected results in edge cases. For financial environments, the predictability of the evaluation engine is as important as the expressiveness of the language. Additionally, AVP offers static policy analysis via IsAuthorizedWithToken, allowing policies to be validated before deployment — something OPA does not offer natively.

Entity modeling in AVP follows a hierarchical schema: Tenant > Group > User, with document attributes such as classification_level, tenant_id, and document_type. A typical policy for a financial analyst would be: permit(principal in Group::"analysts", action == Action::"ReadDocument", resource) when { resource.classification_level <= principal.clearance_level && resource.tenant_id == principal.tenant_id };. This policy evaluates in ~12ms p50 and ~18ms p99 in my measurement with medium-sized entity payloads.

The metadata filter on the Knowledge Base is configured as { "equals": { "key": "tenant_id", "value": "<tenant-from-jwt>" } } combined with { "equals": { "key": "clearance_level_max", "value": "<user-clearance>" } }. These metadata fields are indexed in OpenSearch Serverless and the filter is applied before similarity ranking — ensuring that chunks from other tenants never enter the model's context.

There are five implementation details that most reference architectures omit and that make a real difference in financial production.

1. AVP entity synchronization. AVP requires an entity store consistent with the IdP. I use an async pipeline: Cognito User Pool triggers → EventBridge → Lambda → AVP BatchCreateOrUpdateEntities. The consistency SLA is eventual (~2-5s), which is acceptable for role changes, but requires the system to handle the case of a newly promoted user whose new role has not yet propagated. The solution is a 60s cache TTL in the authorization Lambda with fallback to re-fetch if AVP returns an unexpected DENY.

2. Document metadata in the ingestion pipeline. During Knowledge Base ingestion, each chunk must carry tenant_id, clearance_level_max, document_type, and created_at as structured metadata. This is done via a custom Lambda in the Glue pipeline that processes documents before sending them to Bedrock KB via StartIngestionJob. Missing or incorrect metadata is the primary failure vector in this pattern — I implement schema validation with AWS Glue Data Quality before ingestion.

3. Authorization retry idempotency. When AVP returns a 5xx error (rare, but it happens), the correct behavior is fail closed — deny access and log the error, not bypass. I implement this with a circuit breaker in the authorization Lambda: after 3 consecutive AVP failures in 30s, the circuit opens and all requests are denied with HTTP 503 until the circuit closes. This is counter-intuitive for product teams, but it is the correct behavior for financial systems.

4. Authorization layer observability. I instrument the authorization Lambda with OpenTelemetry, emitting spans for each AVP call with attributes avp.decision, avp.tenant_id, avp.policy_id, and avp.latency_ms. These spans feed a Datadog dashboard with SLOs: auth_latency_p99 < 30ms and auth_deny_rate_anomaly (alert if DENY rate increases more than 3σ in 5 minutes — indicator of attack or policy bug).

The most underestimated part of this pattern is not the technical integration with AVP — it is the governance of the Cedar policy lifecycle. In an environment with 50 tenants and 200 distinct roles, you quickly accumulate hundreds of policies. Without engineering discipline, this becomes an authorization system that no one fully understands.

My approach is to treat Cedar policies as first-class Infrastructure as Code. Each policy lives in a Git repository with mandatory PR review by two security engineers. The CI pipeline uses cedar-policy-cli to validate syntax and run policy unit tests (yes, you can write unit tests for Cedar policies) before any merge. Deployment is done via CodePipeline with manual approval for changes affecting more than 10 entities.

A pattern I found to be critical is the explicit deny policy for unclassified documents. By default, Cedar denies everything not explicitly permitted — but documents without a defined clearance_level_max metadata field (due to ingestion failure) are in an ambiguous state. I add an explicit forbid policy: forbid(principal, action, resource) when { !resource.hasAttribute("clearance_level_max") };. This ensures that documents with corrupted metadata are never accessible, regardless of any other policy.

For the policy drift problem — when policies in AVP diverge from what is in Git — I implement a reconciliation Lambda that runs every 15 minutes, compares the hash of active policies in AVP with the expected state in S3 (pipeline artifact), and emits a CloudWatch alarm if there is divergence. This detects manual changes outside the pipeline, which is the primary misconfiguration vector in regulated environments.