The AI Only Sees What You're Allowed to See: How We Mirror Permissions into Retrieval

In our article on Microsoft Copilot's sovereignty gap, we described a scenario: an intern asks the assistant about the salary structure in marketing. The answer comes back friendly and precise. The source is a forgotten Excel file in an archive folder nobody has cleaned up in years. The file was never locked down. It was just hard to find. For AI-powered search, "hard to find" is no longer an obstacle.

Pointing a finger at Copilot is easy. Then we built our own retrieval for TheroAI, spanning 19 applications connected in read-only mode, and faced exactly the same question: what may the assistant see when it answers on behalf of a specific person?

This article tells the story of how we answered that question. It also covers the obvious solution we rejected and where the honest limit of our approach lies.

The Problem Starts with Copying

TheroAI answers questions from company knowledge. To do that, we connect sources like Google Workspace, Microsoft 365, and GitHub through read-only connectors. To make search fast, we index content. An index is a copy.

This is exactly where the problem starts. The moment a document from Google Drive is copied into a search index, Drive's permission model no longer protects it. Drive knows who may open the file. The index initially does not. Without countermeasures, search would answer every question from the entire corpus. We would have built the intern scenario right into our own product.

On top of that, every source brings its own permission model.

• Google Drive has personal shares, folder inheritance, and shared drives.
• Mailboxes in Gmail and Outlook are personal by default.
• In GitHub, organization, team, and repository role decide who gets access.
• Cutting across all of this are groups that change whenever someone switches departments.

So the task is: these models have to apply in retrieval exactly as they do in the source itself. Internally, we call this permission mirroring.

The Obvious Approach We Rejected

The obvious architecture is quick to build. Search the entire index, take the best hits, then check each hit afterwards to see whether the requesting person may read the document. Forbidden hits get dropped from the list. This is called post-hoc filtering. In a demo, it looks correct.

We rejected it for two reasons.

First: filtered-out documents still influence the result. Retrieval never returns everything. It returns a limited number of best hits. If forbidden documents compete, they occupy slots in that ranking. A document the person is allowed to read gets pushed out of the top hits and is never loaded. The answer gets worse, and it gets worse as a function of content the person is not allowed to see. Snippets make it more delicate. If text excerpts or intermediate summaries are produced before the filter runs, forbidden content is already inside the language model's context. A filter that trims the hit list afterwards does not pull it back out.

Second: a stale filter quotes documents that no longer exist for the person. Checking after the search requires current permission data at the moment of the query. In practice, those checks get cached, because otherwise every question triggers dozens of API calls against the source. Caches go stale. The result: the assistant quotes verbatim from a document whose share was revoked yesterday. The source link under the answer leads to an error page. The content is in the answer anyway. The damage happens on reading, not on clicking.

Our Decision: The Filter Is Part of the Query

We chose the opposite path. Permissions are indexed together with the content, and the permission filter is part of the retrieval query itself.

Concretely, every connector reads two things from the source during sync: a document's content and its access list. That means which users and groups may read it, with folder inheritance resolved. Both land in the index together. When someone asks a question, the retrieval query carries a fixed condition alongside the search terms: only consider documents whose access list contains this person.

A document a person cannot open must not influence that person's answer.

The difference sounds small and is architecturally large:

• Ranking is computed from the start only over documents the person may read. Forbidden content occupies no slots in the hit list.
• Snippets and answers are produced exclusively from permitted content. There is no pipeline stage where forbidden text was briefly visible.
• A document without a matching permission entry is simply not found. Missing access information leads to a missing hit, never to one that slips through. The system fails in the safe direction.

Here is the comparison at a glance:

One Path Through the Product

What does this look like in practice? We follow the path every admin takes during setup and stay with Google Drive as the example.

The connection starts in the "App hinzufügen" dialog (add app). It lists the 19 applications TheroAI can connect in read-only mode, grouped by vendor.

Enlarge

When the admin selects Google Drive, the Workspace connector setup opens. It mattered to us that permission mirroring is visible at this point instead of living in fine print. The setup lists the indexing of shares and permissions as its own ticked items, right next to the step-by-step guide. Whoever connects the source sees in plain sight that content and permissions are ingested together.

Enlarge

After connecting, the first full sync runs. It ingests documents and access lists. Then the source flips to the status "Synchronisiert" (synced) in the connector overview. From that moment, the assistant answers questions from Drive, for each person strictly from their own view.

Enlarge

The interesting moment comes when a permission changes. Take a concrete case: in Drive, you revoke a person's share for the folder "Compensation 2026". What happens then, step by step?

1. 1.In Drive, the change takes effect immediately. The person can no longer open the files.
2. 2.The connector learns about it on the next delta sync. Delta means we only ingest what has changed since the last run. These runs start several times per hour. Where the source supports change notifications, we process them in real time instead of on an interval.
3. 3.The sync updates the access lists of the affected documents in the index.
4. 4.From now on, this person's retrieval query no longer matches those documents. The assistant does not find them, does not quote them, and does not list them as sources. There is no separate deletion mechanism in the answering logic. For this person, the documents are simply no longer part of the search space.

The Honest Limit: As Fast as the Next Sync

Step 2 is the limit of our approach, and we would rather name it ourselves: mirroring is exactly as current as the last delta sync. Between revoking a share in the source and the index being updated, there is a time window. Inside that window, a person can still see content in answers that they can no longer open in the source. For sources with real-time notifications, the window is short. In interval mode, it equals the time until the next sync run.

This works in both directions. A newly granted share also takes effect only after the next sync. Someone who just received access to a folder starts getting answers from it once the connector has mirrored the change.

The alternative would be a live check against the source for every single question. It would slow down every answer, strain the sources' API quotas, and still leave the ranking problem described above unsolved, because candidate selection would already be over by then. We consider the sync window the more honest trade-off. And we would rather state it here than have you discover it in production.

What You Can Take Away

If you put an AI tool on top of your data sources, regardless of vendor, four questions are worth asking:

• Are the source's permissions indexed together with the content, or does the system only check at display time?
• Does the permission filter run before ranking or after? That answer decides whether forbidden documents can influence results.
• How fast does a revoked share reach the index? Ask for the concrete sync interval. The word "synced" alone does not answer that.
• Can you test it yourself? Revoke a share in the source, ask the same question again, and watch when the answer changes.

The fourth question is the most important one. Permission mirroring you cannot verify is a promise. Permission mirroring you can verify is a property of the product.