Christophe MontagneInterim CTO · Advisory

Building It/2 min read

06Security and compliance

Where the data goes, who can see it once it is there, instructions hidden in documents, and the records an audit needs.

Many pilots that work well never go live because of questions nobody asked early enough: where does the data go, and who can see it once it is there?

Where the data goes

Using a third-party model means sending it data: every question, and the passages retrieved to answer it. If a third-party service also produces the vectors, the entire knowledge base passes through it at ingestion, which is easy to overlook. What matters is less the vendor's reputation than the contract: whether data is retained or used for training, where it is processed, by whom, and under which laws. The decision is made per class of data. Public manuals may go to a third-party service, confidential sources only to a model hosted in house, and documents under a supplier's non-disclosure agreement may never enter, although the system can still point to them. Users' questions, which can contain sensitive data themselves, follow the same rules.

Who can see it once it is there

Source systems know who may read what. An HR system shows salaries to payroll staff only; a maintenance system restricts incident reports. Once text is extracted, cut into passages and indexed, that knowledge disappears unless it is carried along on purpose (Figure 6). This is how these systems most often leak.

Rights lost on the way Source system knows who may read Extract and index text only Index no rights attached Anyone who asks may receive it Rights carried to the answer Source system knows who may read Extract and index rights attached Index every passage carries its rights Filter first before the model reads anything Only what this user may see rules that depend on context are checked live at the source
Figure 6. Access rights lost during extraction, versus carried to the answer
  • Rights travel with the data: every passage records who may see it, at the granularity of the source.
  • Rights change, and the copy must follow. Any delay between source and index is a period of exposure.
  • Some rights depend on context, such as a manager seeing only their own team's salaries. These are checked against the source system at answer time, or the source is queried live instead of indexed.
  • Derived knowledge, such as a summary or an extracted table, inherits the strictest rights of its inputs.
  • Rights are applied before anything reaches the model, not after. And because harmless passages can be revealing together, which sources may be combined, and for whom, is a decision in itself.

Access is tested like safety

The test set includes questions from users who must not receive certain answers. Any leak counts as a failure that blocks a release, exactly like an unsourced value.

Documents can contain instructions

The model reads retrieved text as input, and text can be written to steer it, hidden in a supplier document or a pasted note. This is known as prompt injection. The defences are architectural: treat retrieved text as data, never as instructions; limit what the answering side is able to do; and test for it like any other failure.

Compliance rests on the same foundations

In Europe, for instance, data protection law gives people the right to have their personal data erased, so deletions must reach the index and everything derived from it. After an incident or during an audit, the company must be able to reconstruct what a user was told: the question, who asked, the passages and their versions, the model and its version, the rules in force, and the response. Those records contain confidential content and show what employees did, so they are protected and kept only as long as explicit rules allow. Other regions have other laws, and the architecture must support them in the same way.