Christophe MontagneInterim CTO · Advisory

Beyond the Demo/3 min read

01Anatomy of a RAG

What RAG is, the example followed throughout, and why most of the work sits where the demo does not look.

What the AI does not know

A large language model, the engine behind tools like ChatGPT, writes fluently on almost any subject. But it has never read your maintenance manuals, your procedures, or the notes your best engineers have written over thirty years. Ask it about them and it will either admit it doesn't know or, worse, give a confident answer that sounds right and isn't.

RAG (retrieval-augmented generation) is the most common way to close that gap. Before the model answers, the system finds the passages in your documents most likely to hold the answer and asks the model to answer from those passages only, citing its sources. The model supplies the language. Your documents supply the facts.

Our example

Consider a company that maintains rail equipment: locomotives and trains that stay in service for thirty or forty years. The company is a composite, but readers in the sector will recognise it. It has accumulated tens of thousands of documents: manuals for every model and revision, service bulletins, retrofit records, procedures, test reports, technicians' notes. Some are well structured. Many are scans, spreadsheets, or files in formats nobody opens anymore.

And the people who really know the older engines are retiring. Much of what they know was never written down. When they leave, the company loses the ability to maintain its own fleet with confidence. That is a business problem before it is a technology question, and it is the right place to start. The idea that follows is appealing: give technicians in the depot an assistant that answers their questions from the company's accumulated knowledge, with references.

A few years ago, this would have been out of reach for most companies. Hand-written rules and carefully built classifications proved slow and costly; search tools returned lists of documents and left the reading to the user. Generative AI can read messy documents and answer in natural language. The technology is now within reach. The rest of the work is not made any easier.

Two halves, very unequal

1. PREPARING THE KNOWLEDGE built once, then maintained for as long as the system lives Company documents manuals, procedures, bulletins, notes Collect and pick the right version of each Clean and convert scans, tables, old formats Split and index and attach who is allowed to see what Knowledge index 2. ANSWERING every time someone asks a question A question from a technician on site Find the passages most likely to hold the answer The AI writes an answer using only those passages Answer with references to its sources searches
Figure 1. Anatomy of a RAG: preparing the knowledge, then answering questions from it.

The first half prepares the knowledge. Someone decides which documents may enter the system at all (chapter 03 returns to this) and which version of each is authoritative. Documents are cleaned and converted, which is far harder than it sounds with scans and complex tables, then split into passages and indexed, each carrying the rules about who may see it. This half is never finished: documents change, engines are modified, and the knowledge must follow.

The second half answers questions. A technician asks; the system retrieves the relevant passages; the model writes an answer from them, with its sources. This is what every demonstration shows, and it is where expectations and budgets tend to be set.

The demo a question in, a fluent answer out The rest of the project Finding the documents, and the right versions Cleaning scans, tables and old formats Deciding who may see what Keeping the knowledge up to date Measuring whether answers are right Naming owners for content and answers Legal, data protection, staff representatives Deciding what may leave the building What gets shown and what sets expectations What gets paid for most of the effort, cost and risk
Figure 2. What the demo shows, and what the project actually involves.

As Figure 2 shows, most of the effort, cost and risk sit in the first half and around it. Analysts see the same pattern: Gartner expects organisations to abandon 60% of AI projects that lack data fit for AI, and found that nearly two thirds of organisations either lack the data practices AI needs or don't know whether they have them.

There is also a limit no technology removes: the system can only find what has been written. The knowledge that matters most in our example, in the heads of retiring experts, is precisely what the documents lack. Capturing it is the organisation's job, though the project can be designed to help.

None of this is a technical detail. Each point is a question about how the organisation manages its knowledge, its risks and its people. A project that looks like an IT initiative in the demo turns out, very quickly, to be a management one.