Retrieval-Augmented Generation (RAG) has become the dominant pattern for building LLM applications that need access to custom knowledge. But most tutorials show you the happy path. Here's what actually breaks when you go to production.
What is RAG and Why Does It Break?
RAG works by embedding your documents into a vector store, then at query time, finding the most relevant chunks and stuffing them into the LLM context. Simple in theory. In practice, you have five distinct failure modes.
- Retrieval returns irrelevant chunks — the wrong text gets sent to the LLM
- Chunk boundaries split critical context — an answer spans two chunks, neither is complete
- Embedding model mismatch — you embed with one model and query with another
- Context window overflow — too many chunks, the LLM ignores the later ones
- Stale embeddings — your vector store diverges from your source of truth
Chunking Strategy: What Actually Works
Most tutorials use fixed-size chunking with 512 tokens and 50-token overlap. This is fine for demos. For production legal or medical documents, you need semantic chunking.
from langchain.text_splitter import RecursiveCharacterTextSplitter
# Bad: Fixed chunking ignores semantic boundaries
bad_splitter = RecursiveCharacterTextSplitter(
chunk_size=512,
chunk_overlap=50,
)
# Better: Semantic-aware chunking
from langchain_experimental.text_splitter import SemanticChunker
from langchain_openai import OpenAIEmbeddings
good_splitter = SemanticChunker(
OpenAIEmbeddings(),
breakpoint_threshold_type="percentile", # splits at semantic breaks
breakpoint_threshold_amount=95,
)For documents with headers (PDFs, markdown), always split at heading boundaries first, then apply semantic chunking within each section.
The Retrieval Problem: Hybrid Search
Pure vector search (cosine similarity) fails when users ask precise factual questions — exact keyword matches beat embeddings for specific terms, product codes, or names.
from langchain.retrievers import EnsembleRetriever
from langchain_community.retrievers import BM25Retriever
from langchain_pinecone import PineconeVectorStore
# Vector retriever (semantic)
vector_retriever = PineconeVectorStore(...).as_retriever(
search_kwargs={"k": 6}
)
# BM25 retriever (keyword)
bm25_retriever = BM25Retriever.from_documents(docs)
bm25_retriever.k = 4
# Hybrid: 60% vector, 40% keyword
ensemble = EnsembleRetriever(
retrievers=[vector_retriever, bm25_retriever],
weights=[0.6, 0.4]
)Preventing Hallucinations with Source Grounding
The most common production complaint: 'the AI made something up.' This happens when the retrieval returns nothing relevant but the LLM answers anyway. Always instruct the model explicitly:
SYSTEM_PROMPT = """You are a helpful assistant. Answer ONLY based on the
provided context. If the context does not contain the answer, say
"I don't have enough information to answer that."
Do NOT make up information or use your training knowledge.
Context:
{context}"""Add a confidence check: after generating the answer, run a second LLM call that scores whether the answer is grounded in the retrieved context. If score < 0.7, return a 'not enough information' response.
Production Checklist
- ✅ Semantic chunking with heading-aware splitting
- ✅ Hybrid search (vector + BM25) for all retrievers
- ✅ Source citation in every response
- ✅ Confidence scoring to prevent hallucinations
- ✅ Embedding model pinned to a specific version
- ✅ Nightly re-indexing job for document freshness
- ✅ Retrieval evaluation with RAGAS metrics (faithfulness, answer relevancy)
Building RAG right takes more than an afternoon. But if you implement these patterns from day one, you'll avoid the painful rewrites we've seen clients go through after launching with naive implementations.