08Hardware, cost and capacity: measure, then decide

Capacity planning is hard for any new system. It depends on three things that are unknown at the start: the resources one unit of activity consumes, how much activity there will be at its quietest, busiest and on average, and how much redundancy is needed when something fails. Here, all three are less known than usual. There is little past experience to estimate from, growth depends on adoption nobody can predict, and the hardware involved may be expensive.

Know where the computing power goes

If the model is a third-party service, capacity is bought per use, though rate limits, quotas and variable response times still need planning. If it is hosted in house, generating answers takes most of the computing power, usually on GPUs. The knowledge base has more ordinary needs. Three loads must be planned separately: ingestion, which comes in bursts; questions, which come in peaks; and evaluation, which runs on every change and is easy to forget.

Treat cost per question as a design choice

How many passages are retrieved, whether a second ranking step is used, how large the model is: these choices can change the cost of a question tenfold. Every design change is therefore judged on two numbers, its quality on the test set and its cost per question.

Don't buy hardware up front

Before measuring, nobody knows what to buy, let alone how much. Cloud platforms let a team provision, test and decommission at will, so on-demand use comes first, with costs watched closely. Test platforms sit idle most of the time, so they must shut down automatically; savings that depend on someone remembering to switch things off rarely happen. GPUs on demand are not always available, and some tests may need reserved capacity.

Let the stages produce the figures

Capacity cannot be planned up front, and it does not need to be: each stage provides the measurement the next decision needs (Figure 8). Crawl gives the cost of preparing each kind of document. Walk gives cost and latency per question, and stress tests built from realistic questions show how the system scales. Run gives what no test can: real usage, peaks and growth. Projections over one to three years are then made as scenarios (low, expected and high adoption) and revised at each stage.

Then decide where to run

With reliable figures, the choice between cloud and owned hardware rests on evidence. It also depends on contracts that may require in-house hosting, on the skills needed to operate GPU hardware, on how fast that hardware ages, and on utilisation, since owned hardware pays off only under high, steady load. A mix is common, such as ingestion bursts in the cloud and everyday questions in house. Because the model can be replaced, the decision remains reversible.

Plan for failure

If the model becomes unavailable, the system can fall back to pointing to sources, which needs no generation at all. And because rebuilding the index can take days, the index is backed up like any other production data.

One capacity is often missed: experts' time. Validating knowledge, correcting drafts and growing the test set depend on a few people, and in many organisations they are the real bottleneck.