04Building the knowledge base

The knowledge base is a production line, not a one-time load (Figure 4). Documents change every week and figures every month, and each change must go through the same steps: read, check, describe, repair or record, index, and measure that nothing broke.

Start from the data profile

The Crawl audit examines the whole collection, not only the easy part: how much is scanned, how much is in tables, how much is in formats nobody has opened for years, how much contradicts something else. That profile decides the pipelines. A born-digital manual, a scanned bulletin from the 1990s and a spreadsheet of retrofits need different readers and different checks. One generic pipeline for everything is a demo habit.

Buy the parts, own the assembly

Tools that read scans and tables, cut text into passages, or turn text into vectors already exist, improve quickly, and should generally be adopted rather than built. They are chosen by measuring them on the company's own documents and test questions, not by reputation. What cannot be bought is what makes the result trustworthy: how each piece of knowledge is described, the checks, the decisions about what to repair, and the test set. That is where the company's own effort goes.

Describe knowledge with the people who own it

The contract's facts are only as good as their definitions. "Applies to" sounds simple until an expert explains that a locomotive's configuration depends on the retrofits it has received, so a value from its original manual may no longer apply. Engineers working alone will create the field and miss the meaning. Experts also decide what counts as validated: the pipeline runs checks, but the business grants trust. And trust has a date. When a bulletin supersedes a page of a manual, its status must change, and the knowledge base must notice.

Repair, fill, or record

Every hole found follows the same decision (Figure 2). Defects are repaired: missing revision numbers, unreadable scans, duplicates, mislabeled versions, contradictions settled by an expert. Missing knowledge is filled only from real sources, such as an expert interview, recorded with its own provenance. The rest is recorded as a known gap. Nothing is estimated to look complete.

Index text with care

For text, retrieval works on passages, called chunks, each turned into a vector and stored in an index searched by similarity; chapter 02 of What Claude, Gemini and ChatGPT actually are explains what vectors are. Three decisions matter more than the tools. The chunk is the unit of truth: one that splits a procedure step, or separates a table from its header, cannot be trusted. Every chunk carries the contract's facts, or traceability and access rights break at retrieval. And part numbers or model codes are matched poorly by vectors alone, so keyword search works alongside them. Changing how documents are cut, or the model that produces the vectors, means reprocessing the whole collection, so those choices deserve the most care.

Treat the index as a copy

The index holds the company's knowledge in another form, so it must be protected at the level of its most sensitive document. Deleting a source does not delete what was extracted from it: the production line must carry deletions and withdrawals as reliably as additions.