The first RAG system I built worked perfectly in the demo and embarrassed me in the first real meeting. Someone asked a question whose answer sat in a table halfway through a policy PDF. The system retrieved three chunks about a vaguely similar topic, the model wrote a fluent and completely wrong answer, and I learned the lesson that governs everything I have built since.
Retrieval-augmented generation is a search problem wearing an AI costume. When RAG fails, the model is almost never at fault. The right text was not in the context, so the model did what models do with insufficient information: it produced something plausible.
Here is what the architecture actually looks like, where it breaks, and how to decide between retrieval and fine-tuning.
The Pipeline, Honestly Described
RAG has two halves that people tend to conflate.
Ingestion, which runs offline: take your documents, split them into pieces, convert each piece into a vector using an embedding model, and store the vectors along with the original text and metadata.
Query time: embed the user's question with the same model, find the nearest stored vectors, take the corresponding text, put it into the prompt with the question, and ask the model to answer using only that.
That is the whole idea. The difficulty is not in the concept, it is in the fact that "find the nearest vectors" is a much weaker operation than it sounds, and every choice you made during ingestion constrains what it can possibly find.
Chunking Is the Decision You Will Regret
Most bad RAG systems are bad because of how the documents were split, and it is the step people spend the least time on.
Splitting every 1,000 characters is the default in every tutorial and it is close to the worst option. It cuts sentences in half, separates a table from its heading, and detaches a clause from the section that gives it meaning. When that chunk comes back at query time, it is missing exactly the context needed to interpret it.
What works better, in order of how much difference it has made for me:
Split on document structure. Headings, sections, list items, table boundaries. A chunk should be a thing a human would recognise as a unit. This alone fixes more retrieval problems than any amount of tuning downstream.
Keep a breadcrumb in every chunk. Prefix the text with its document title and heading path. A chunk that reads "must be submitted within 30 days" is useless; "Refund Policy > Consumer Returns > must be submitted within 30 days" is answerable.
Overlap a little. A couple of sentences of overlap between adjacent chunks stops an answer that straddles a boundary from being lost by both.
Store the neighbours. Retrieve the matching chunk, but pass the surrounding chunk on each side into the prompt. Cheap, and it rescues a lot of near-misses.
Handle tables separately. Tables embed terribly as raw text. Convert each row to a sentence, or store a text description of the table alongside it. My policy PDF failure was a table.
Vector Databases: Less Important Than the Choice Above
People agonise over this and it is rarely the constraint. Pinecone, Qdrant, Weaviate, Milvus, Chroma, or the vector extensions in Postgres and MongoDB — they will all retrieve approximately the same things given the same embeddings and chunks.
My actual advice: if you already run Postgres, start with pgvector. One less system to operate, transactional consistency between your documents and your vectors, and the ability to filter on ordinary SQL columns in the same query. For a corpus under a few million chunks this is entirely sufficient, and the operational simplicity is worth more than a benchmark difference you will not notice.
Move to a dedicated vector database when you genuinely outgrow it — very large corpora, demanding latency requirements at high concurrency, or a need for features like multi-tenant namespaces that would be awkward to build yourself.
Whatever you use, store metadata alongside every vector and filter on it. Document type, department, date, access permissions. Filtering before the similarity search is both faster and more accurate than retrieving broadly and hoping. It is also how you stop one customer's documents from surfacing in another customer's answers, which is a data breach rather than a quality problem.
Pure Vector Search Is Not Enough
Embeddings capture meaning, which is exactly why they are bad at things where the literal string matters. Product codes. Error numbers. Names. Version identifiers. Ask about "error E4021" and semantic search cheerfully returns chunks about other errors, because they are all semantically about errors.
The fix is hybrid search: run a keyword search and a vector search, then combine the rankings. Every serious system I have built ended up here, usually after being embarrassed by an exact-match query in a demo.
The second addition worth its cost is a reranker. Retrieve twenty candidates cheaply, then use a cross-encoder model to score each one against the question properly and keep the best five. It adds latency in the low hundreds of milliseconds and it consistently improves what reaches the model, because the initial retrieval is optimised for speed rather than precision.
async function retrieve(question: string, tenantId: string) {
const [semantic, keyword] = await Promise.all([
vectorSearch(question, { tenantId, limit: 20 }),
keywordSearch(question, { tenantId, limit: 20 }),
]);
const merged = reciprocalRankFusion(semantic, keyword); // combine rankings
const ranked = await rerank(question, merged); // cross-encoder
return withNeighbours(ranked.slice(0, 5)); // add context
}
The Generation Half, Which Is the Easy Half
Once the right text is in the context, the prompt does not need to be clever. It needs three things: an instruction to answer only from the provided material, an explicit instruction to say when the material does not contain the answer, and a requirement to cite which chunk each claim came from.
The citation requirement is not just a user-facing nicety. It changes the model's behaviour, because a claim that must be attributed to a source is harder to invent. It also gives you a debugging tool — when an answer is wrong you can immediately see whether retrieval failed or generation did, which is otherwise guesswork.
And build the "I do not know" path deliberately. A system that admits it cannot find something is trusted; a system that improvises once loses users permanently. If the top retrieval scores are all below a threshold, do not call the model at all.
RAG or Fine-Tuning?
This comes up in almost every AI project meeting, and the answer is nearly always the same.
Use retrieval when the problem is knowledge. The model needs to know things — your documentation, your policies, your product catalogue, this week's data. Retrieval handles changing information natively: update the document, re-embed one chunk, done. It gives you citations. It lets you enforce permissions at query time. It is dramatically cheaper to keep current.
Use fine-tuning when the problem is behaviour. A consistent tone, a rigid output format, a specialised task where you want a smaller model to perform above its weight, or shorter prompts because the instructions are baked in.
Fine-tuning to teach facts is the mistake, and it fails in the worst possible way: the model produces confident, fluent, well-styled answers that are invented, with no source to check. You have made the hallucinations harder to detect rather than less frequent.
The two combine well. Fine-tune for the format and the voice, retrieve for the facts.
Evaluate Retrieval Separately From Answers
The last thing, and the one most teams skip. Build a set of thirty real questions with the chunk that should be retrieved for each. Then measure one number: how often the correct chunk appears in the top five.
That number tells you where to spend your time. If retrieval is at 60%, no amount of prompt tuning will save you and you should go back to chunking and hybrid search. If retrieval is at 95% and answers are still poor, the problem is in the generation prompt.
Without that split, teams tune the prompt for weeks against a retrieval problem. I have watched it happen more than once, and I have done it myself. RAG is search first, generation second — and it stays that way no matter how good the models get.
