Christophe MontagneInterim CTO · Advisory

Beyond the Demo/3 min read

04What does “better” mean?

Three questions and a baseline, errors weighed by their consequences, and measuring as a way to capture what experts know.

Every presentation of a project like ours promises better answers. Few say what "better" means, or how anyone would know. Without that, nobody can say whether the project works, whether it is worth its cost, or whether it is safe.

Three questions, and a baseline

A technician's answer must pass three tests. Did the system find the right document, in the right version? Did the answer say what that source says, and nothing more? Did it actually help with the job? Only people who know the work can judge these. Relevance is defined by experts and users, not by the IT department.

"Better" also needs a comparison. Not the AI with and without documents, but the new system against how technicians find answers today: searching a shared drive, leafing through a manual, calling the colleague who knows. How long does it take? How often does it fail? Few projects measure this baseline before they start, and without it, both improvement and return on investment are matters of opinion.

Not all errors are equal

Take a simple question: at what torque must a given bolt be tightened? The answer depends on the exact engine model, and getting it wrong can be catastrophic.

The model is found, and so is the torque. The system gives the value with its source. This is the goal.

The model is found, but no torque is documented. The right response is "I don't know." The technician calls an expert, and the project learns where its knowledge has a gap.

The model is not found, but torques exist for other models. Everything hinges on behaviour. The system can say it has no value for this model and send the technician to an expert. Or it can take the nearest model's value and state it as confidently as a correct one. Same documents, same gap: one response is safe, the other can destroy an engine, or worse.

A technician asks: "At what torque must this bolt be tightened on engine model X?" WHAT THE DOCUMENTS HOLD WHAT THE SYSTEM DOES CONSEQUENCE Model found, torque found Gives the value with its source Time saved, job done right Model found, torque missing Says "I don't know" and points to an expert Expert called; a gap in the knowledge is revealed and filled Model not found, torques exist for other models Says it has no value for model X, mentions similar models, points to an expert Safe: the technician checks before acting Picks the nearest model and gives its value, as confidently as a correct one Wrong torque applied: potentially catastrophic Same documents, same gap: the outcome depends on a behaviour someone has to decide in advance.
Figure 5. One question, several outcomes. The dangerous one depends on a behaviour someone must decide in advance.

"Right nine times out of ten" means little on its own. What matters is the tenth time, and what the system does when it doesn't know. Errors must be weighed by their consequences, and the behaviour in the face of missing knowledge decided explicitly. Both are judgements about risk, and they belong to the business.

Measuring, and what it captures

Before anything is built, experts write down real questions with the answers they consider correct. Every cycle, the system is tested against this set, so that "better" becomes visible and a change that quietly makes things worse gets caught. In use, other signals follow: time saved, questions passed to an expert, answers flagged as wrong.

This takes expert time, the scarcest resource in our example, and it belongs in the budget from day one. It is not overhead; it is the instrument panel. It also has a hidden benefit. When an expert corrects an answer ("on the older units, you must also replace the bolts"), they write down something that was never written before. Measuring quality becomes a way of capturing what retiring experts know, one question at a time.