RAG on ServiceNow: the connector that erases user criteria

What happened

I stood the connector up the day it shipped, September 4, 2026, against a staging instance loaded with a real export: roughly 12,000 knowledge articles spread across six separate knowledge bases, and 40,000 active catalog items. The goal was to measure sync time and retrieval quality for an internal IT assistant — the exact use case AWS names in the announcement.

I followed the documentation line by line. The glide.oauth.inbound.client.credential.grant_type.enabled property set to true. A service account flagged Web service access only, no interactive sign-in. The OAuth application registered through the interceptor page — not by inserting straight into oauth_entity, as the guide itself warns. An Inbound Authentication Profile wired to a REST API Access Policy over now/table, because without it the token is valid and the Table API still returns 401. Step 3 asks for exactly two roles: knowledge_admin and catalog_admin. I assigned both, and ServiceNow inherited around fourteen contained roles.

The sync finished. Then I asked the question I ask of every enterprise RAG before anything ships: I queried the assistant about a subject I knew was restricted to one group on the source instance. The answer came back whole, citing the article, for an identity that held no role there at all. It wasn't a hallucination — it was correct retrieval of a document ServiceNow would never have rendered for that user.

The pilot stopped there. What follows is the retro, blameless: the configuration was right per the documentation, and that is precisely the point.

The fix: scope at ingestion, because there is none at query time

When the access boundary doesn't exist at retrieval time, it has to exist earlier — in what enters the index. Four concrete changes.

One knowledge base per audience, not one per system. Retrieve takes a knowledgeBaseId; that is the only unit IAM can see. I split into two bases: one fed with crawlPublicKnowledgeArticlesOnly set to true, another with inclusionKnowledgeBaseSysIds listing only the sys IDs whose audience matches an IdP group. The 10,000-knowledge-bases-per-account-per-Region quota isn't the constraint here; the constraint is operational, which is why granularity stopped at audience rather than group.

Authorization moves back into the application. The bedrock:Retrieve permission is now granted through a distinct role per audience, with a Condition on the knowledge base's resource tag. The service answering the user resolves the group from the IdP token and assumes the matching role. That is more code than I would like — and it is the price of using a source with no ACL.

sys ID filtering stopped being an optimization. inclusionServiceCatalogSysIds and inclusionKnowledgeArticleCategorySysIds are scope control and sync-time control at once: the troubleshooting page attributes hours-long syncs precisely to their absence.

Protected deletion. I enabled deletionProtectionConfiguration with deletionProtectionThreshold at 15. If a sync tries to remove more than 15% of the index — which a user criteria mistake at the source triggers easily — the delete phase is skipped instead of emptying the assistant in production.

Identity: the detail that decides whether the mitigation works

Even on connectors that do carry ACLs, Managed Knowledge Base's identity model is narrower than most enterprise designs assume — worth knowing before you promise fine-grained segregation to the risk team.

The universal identifier is email. userContext.userId on Retrieve is always the user's email, and it must match exactly the email associated with that user in each connected data source. There is no alias resolution and no cross-identity-provider mapping. If the company carries two domains from an acquisition, half the people get zero results — silently, which is the worst failure mode available.

Groups come from the last sync. User-to-group membership is crawled at ingestion and resolved at query time from that snapshot. Real-time verification covers the window between syncs on the four connectors that support it; third-party IdP credentials are cached for up to one hour, and permission changes are eventually consistent, typically within minutes. For a termination, that is exposure time — treat revocation as an IdP and application event, not a knowledge base one.

Missing ACL means inaccessible, not public. A document with no extracted permissions in an ACL-enabled source is returned to nobody, and evaluation fails closed: a group-resolution error yields fewer results, never more. That is the correct behavior, and it is also the most common explanation for "the assistant got dumber after the deploy".

And the sentence the documentation insists on repeating: ACL-aware is filtering, not authorization. Authentication is yours.

What now gets measured before any release

The retro doesn't end at a configuration fix; it ends at a signal. A RAG leak throws no error, shows no latency anomaly and never lands on an availability dashboard — it shows up as a useful answer for the wrong person. Three things joined the release checklist.

A canary query suite. A small set of questions whose correct answer is silence, one per audience, run against each knowledge base after every sync. It is access regression testing, it runs in seconds and it costs US$ 1.00 per thousand calls. The alternative is learning about it from the ombudsman.

Document counts per sync, with an expected band. The statistics object from GetIngestionJob reports how many documents were added, failed and removed. A sharp swing usually means a scope change at the source — a category moved, a catalog reactivated — before it means a platform problem. With deletionProtectionThreshold enabled, the sync that would skip its delete phase is exactly the event you want alarming.

The origin of cited chunks. Because the assistant returns citations, the dataSourceId of each retrieved chunk is free telemetry. If a restricted-audience knowledge base shows up in an open-channel answer, the alarm fires without anyone having to read the answer.

The connector saves weeks of hand-rolled pipeline — what it does not save is the design of who may see what. That part was never outsourceable.