RAG Is Easy. Production RAG Is a Nightmare.

Every tutorial makes Retrieval-Augmented Generation look simple. Embed your documents, store them in a vector database, retrieve the top-k chunks, stuff them into a prompt. Demo works in an afternoon.

Then you try to ship it. Answers hallucinate. Retrieval misses obvious documents. Latency blows past your SLA. Costs triple when real users hit the system. The "afternoon project" becomes a six-month engineering effort.

The gap between demo RAG and production RAG is enormous, and almost nobody talks about it honestly.

Chunking Is the First Bottleneck

Your chunking strategy determines retrieval quality more than your embedding model does. Get it wrong and your retriever returns fragments that lack context or chunks so large they dilute the signal.

There is no universal best strategy. Start with fixed-size (512 tokens, 20% overlap) as a baseline. Measure retrieval quality. Then iterate.

Retrieval Evaluation Is Non-Negotiable

Most teams skip retrieval evaluation entirely. They eyeball a few queries, see reasonable-looking results, and move on. Then production users ask questions the team never tested, and the system returns irrelevant garbage.

Build an eval harness before you optimise anything else:

• Curate a test set. 50-100 representative queries with known relevant documents. This is tedious. It is also the single most valuable asset in your RAG pipeline.
• Measure recall@k. Of the documents that should be retrieved, how many actually appear in the top-k results? If recall@5 is below 0.8, your generation quality has a hard ceiling.
• Measure MRR (Mean Reciprocal Rank). Where does the first relevant result appear? If it's consistently at position 4 or 5, your reranker needs work.
• Track retrieval latency. P50 and P99 separately. A slow retrieval step compounds with generation latency.

Run this eval on every change: new embedding model, different chunking, updated documents. Without it, you're tuning blind.

The Hidden Data Engineering Problem

RAG tutorials assume your documents are static. In production, they aren't. Policies update. Product docs change. Knowledge bases grow. If your RAG system serves stale content, it's worse than useless because users trust it.

Production RAG requires a proper ingestion pipeline:

• Change detection. Know when source documents update. Polling, webhooks, or CDC depending on the source.
• Incremental re-indexing. Re-embed only changed documents, not the entire corpus. Full re-indexing doesn't scale past a few thousand documents.
• Deduplication. The same content appears across multiple sources (wiki mirrors, copied PDFs, email forwards). Duplicate chunks inflate retrieval noise and cost.
• Freshness SLOs. Define how stale is acceptable. Real-time for support docs? Daily for policy manuals? Match your pipeline cadence to the SLO.
• Deletion handling. When a document is retracted or superseded, its chunks must be removed from the index. Orphaned chunks serve outdated or incorrect answers.

This is data engineering, not AI engineering. Treat it accordingly.

Cost and Latency at Scale

Demo-scale RAG costs almost nothing. Production-scale RAG with real traffic gets expensive fast.

Embedding compute. Re-embedding a 100K document corpus on every update is wasteful. Cache embeddings, re-compute incrementally, and batch embedding calls.

Vector database sizing. Millions of vectors need careful index tuning (HNSW parameters, quantisation settings). Over-provisioned indexes waste memory. Under-provisioned indexes degrade recall.

LLM inference. Every retrieval-augmented call sends retrieved chunks as context. More chunks means more input tokens means higher cost and latency. Find the minimum context window that maintains answer quality.

Caching. Identical or near-identical queries hit production systems constantly. Semantic caching (embedding-based similarity on queries) can eliminate redundant LLM calls and cut costs dramatically.

Guardrails for Production

Production RAG needs guardrails that demos don't:

• Confidence thresholds. If retrieval similarity scores are below a threshold, return "I don't know" instead of hallucinating an answer from weak context.
• Source attribution. Every answer should link to the chunks that informed it. Users need to verify. Auditors need to trace.
• Fallback behaviour. When the retriever returns nothing relevant, degrade gracefully. Route to a human, return a canned response, or escalate. Never guess.
• Content filtering. If your corpus contains sensitive or restricted content, enforce access controls at retrieval time, not just at the document level.

The Bottom Line

Production RAG is three systems pretending to be one: a data pipeline, a search engine, and a language model orchestrator. Treating it as "just an LLM feature" is why most RAG projects stall after the demo.

Build the eval harness first. Invest in the ingestion pipeline. Monitor retrieval quality in production. The LLM is the easy part.