Building Production-Grade RAG Systems

Why the demo works and production does not

A basic RAG pipeline can be assembled in an afternoon and it will demo beautifully, because the person demonstrating it asks questions they already know the documents answer well. Real users do not do that. They ask questions using different vocabulary, they reference things obliquely, and they ask about content that spans several documents.

Retrieval quality is the entire game. A strong model given the wrong passage produces a fluent, confident, wrong answer — which is worse than no answer, because it is harder to detect and more likely to be believed.

Chunking: the decision everyone gets wrong first

The default advice is to split documents into fixed-size chunks with some overlap. It is easy to implement and it is why so many RAG systems underperform, because it splits arguments in half and separates claims from their qualifying conditions.

Chunk on document structure instead. Contracts chunk on clauses. Documentation chunks on sections. Support tickets chunk as whole conversations. Where a chunk needs surrounding context to make sense, attach that context as metadata rather than hoping the retriever picks up the adjacent chunk too.

Split on semantic boundaries — sections, clauses, conversations
Attach document title and section path to every chunk as metadata
Keep tables intact rather than splitting them across chunks
Store a parent-document reference so full context can be pulled when needed

Hybrid retrieval beats pure vector search

Vector search understands meaning and is poor at exact matching. Ask about part number "XR-4400B" and semantic search will helpfully return passages about similar products while missing the one document that names it.

Combining vector search with keyword search and metadata filtering solves this. Semantic search handles paraphrase and intent; keyword search handles identifiers, names and quoted phrases; metadata filtering narrows by date, document type or permission scope before ranking even begins.

Re-ranking is the cheapest quality win available

Retrieval returns candidates ranked by embedding similarity, which is a rough proxy for relevance. Very often the genuinely correct passage sits at rank eight or eleven — retrieved, but never passed to the model because only the top five are used.

A cross-encoder re-ranker scores each candidate against the query directly rather than comparing pre-computed vectors. It is slower per document, which is why it runs on twenty candidates rather than twenty thousand, and in our experience it is consistently the single highest-return improvement in a RAG pipeline.

Grounding, citation and the ability to refuse

A production system must be able to decline. Systems instructed to always answer will always answer, including when retrieval returned nothing relevant — and that is precisely the situation where a confident answer causes damage.

Constrain answers to retrieved context, require a citation for every substantive claim, and tune a relevance threshold below which the system says it cannot find the information. Users trust a system that admits uncertainty far more than one that is confidently wrong twice.

Evaluation is not optional

If you cannot measure retrieval quality, you cannot improve it, and you will not notice when it degrades. Every RAG system we ship carries a golden set of questions with known-correct source passages, drawn from real user queries wherever possible.

Every configuration change is scored against that set before release. This is what makes iteration safe: you can try an aggressive chunking change knowing that a regression will show up as a number rather than as a support ticket three weeks later.

Retrieval recall — is the correct passage in the candidate set at all?
Ranking precision — does it make it into the passages actually used?
Groundedness — is every claim in the answer supported by a cited passage?
Refusal accuracy — does it decline when it genuinely should?

Frequently asked questions

How accurate can a RAG system realistically be?

Well-built systems over a well-maintained corpus routinely reach 90 to 96% answer accuracy on questions the documents genuinely cover. The remaining errors are usually retrieval failures on ambiguous phrasing rather than model failures, which is why retrieval work delivers more improvement than model upgrades.

Which vector database should we use?

For most workloads, pgvector inside your existing PostgreSQL is sufficient and avoids introducing another system. Pinecone, Weaviate and Qdrant become worthwhile at very large scale or when you need specific filtering behaviour. The database is rarely the constraint — chunking and re-ranking are.

How often does the index need rebuilding?

Incremental updates as documents change, which should be automated as part of the ingestion pipeline. Full rebuilds are only needed when you change the embedding model or the chunking strategy — both of which should be validated against the golden set first.