Designing RAG Systems That Work in Production

A working RAG demo is one of the easiest impressive things to build in modern AI. Point a language model at some documents, add a vector database, wire up a retrieval step, and the first questions work well enough that stakeholders start planning the rollout. Six months later, the same system is answering internal questions inconsistently, the corpus has drifted, no one is quite sure why some answers are wrong, and the team is quietly rebuilding.

The gap between the demo and the production system is not one big thing. It is a dozen small things that only surface at scale, and every serious RAG deployment we have worked on has had to solve them explicitly.

Start with the corpus. The demo probably used a clean set of a few hundred documents. Production has tens or hundreds of thousands of documents, of varying quality, in inconsistent formats, with unclear ownership, some of them out of date, some of them contradictory, some of them containing information that should not be surfaced to certain audiences. The corpus is a product now, not an input. It needs an owner, a lifecycle, and a definition of what belongs in it.

The first practical decision is chunking. Naive fixed-size chunking is what every tutorial teaches and what almost no production system uses for long. The problems are predictable: chunks split mid-sentence, mid-table, or mid-thought; chunks lose their surrounding context; short chunks match too readily on surface features while long chunks dilute the signal. Structure-aware chunking — respecting section boundaries, table boundaries, list boundaries, and preserving a small amount of neighbouring context — outperforms fixed sizing on almost every evaluation we have run.

Retrieval quality is the next place demos and production diverge. Pure vector similarity is easy and often insufficient. Hybrid retrieval — dense vectors combined with a lexical signal like BM25, reranked by a smaller model — closes most of the accuracy gap. The additional latency is modest, the additional cost is small, and the reduction in "the answer was in the corpus but the retriever missed it" failures is often the difference between a system users trust and one they route around.

Metadata matters more than most teams expect. If your documents have structure — a department, a document type, an effective date, an audience — pushing that structure into the retrieval layer as filters, not just as text, dramatically improves precision. A user asking about parental leave should not get answers from a document that was superseded three years ago, and no amount of embedding cleverness will reliably prevent that. A date filter will.

Freshness is where most production RAG systems quietly rot. Documents change. Policies get updated. Product specs move. Unless there is a documented ingestion pipeline with a schedule, a monitored queue, and an alerting layer for failures, the corpus drifts out of sync with the underlying source of truth. Six months in, the RAG system is answering yesterday's questions correctly and today's questions confidently wrong. Build the freshness pipeline before you launch; retrofitting it into a system already answering questions is painful.

Evaluation is the discipline that separates serious RAG programmes from hobby ones. The evaluation set is a collection of real questions — sourced from actual users, not made up by the build team — paired with acceptable answers, or in some cases acceptable answer criteria. It is refreshed regularly. Every retrieval change, every model change, every prompt change is scored against it before promotion. Regressions block deploys. The scores are visible.

There are a few evaluation patterns that repay the setup effort. Retrieval-only evaluation: given a question, did the retriever return the passages needed to answer it, regardless of what the model then did? End-to-end evaluation: given a question, was the final answer factually correct and appropriately caveated? Hallucination detection: did the answer make claims that were not supported by the retrieved passages? These are three different failure modes and they need three different measurements.

For anything customer-facing or decisioning, add a groundedness check as a hard requirement. Every answer cites its sources, and every claim in the answer maps to a specific passage in the corpus. If it does not, the system refuses to answer, or hands off. This is more work than "just let the model write freely," and it is the difference between a system a regulator will accept and one they will not.

The interface matters as much as the retrieval. Users trust answers with citations they can click. They lose trust rapidly when they follow a citation and the passage does not obviously say what the answer claimed. Design the citation experience deliberately: which sources to show, how to highlight the relevant passage, what to do when confidence is low. A well-designed "I could not find a reliable answer for this — here are the closest things" response is far more valuable than a confidently wrong answer.

Latency has its own economics. Users tolerate a two-second answer for a question that would have taken them ten minutes to research. They do not tolerate a fifteen-second answer for the same question, because at that point they open a browser tab instead. Design for the latency budget users actually have, which is usually shorter than the one engineers assume, and instrument it so you know when it slips.

Cost management, similarly, is a design problem, not just a monitoring one. Retrieval that reranks with a large model on every query is expensive. Caching common queries, using smaller models for reranking, and batching where possible all matter at scale. The cost per query at pilot volume tells you almost nothing about the cost per query at production volume, so run a load test before the go-live meeting, not after.

Access control is the one thing you cannot bolt on afterwards. If different users are allowed to see different documents, the retrieval layer must enforce that at query time, not at response time, and not by relying on the model to be discreet. This means the vector store needs to understand identity, and the ingestion pipeline needs to preserve the source system's permissions. Retrofitting permissions into a RAG system that was built assuming everyone sees everything is a rebuild, not a patch.

Finally, the operating model. A serious production RAG system has three ongoing workstreams: corpus curation (owned by the business function whose knowledge it is), retrieval quality (owned by the platform team), and evaluation (owned by someone independent of the build team). Cut any of these and quality degrades within a quarter. Fund all three, or accept the drift.

None of this makes RAG less useful. It makes it useful for longer. The organisations that treat RAG as a demo scale to a couple of use cases and stop. The organisations that treat it as a production system with a corpus, an evaluation harness, and a freshness pipeline build a knowledge capability the whole enterprise can compose against.