ON THIS PAGE (TAP TO EXPAND)
How RAG actually works, step by step
Technical guide and implementation breakdown for How RAG actually works, step by step.
RAG (retrieval-augmented generation) lets an AI model look things up before it answers. Your documents are split into passages and indexed. For each question, the closest passages are found and pasted into the prompt, so the model answers from your content instead of from memory, and can show where the answer came from.
Why models need it
A language model only knows what was in its training data. It has never read your refund policy, last week’s price list or your internal wiki. Ask it about them and it may still give a confident answer, just not a correct one.
You could retrain the model on your documents, but that is slow, expensive, and out of date the moment a document changes. RAG takes the other route: keep the model as it is, and change what it reads.
The five steps, one at a time
Click a step, or use the arrows. The first two happen once, when you add documents. The last three happen every time someone asks a question.
Chunk
Split each document into small passages, a few sentences to a few paragraphs each, so every passage is about one thing.
Try a search
Here is a tiny help center with five passages. Pick a question and watch which passages come out on top. Only the top three are sent to the model.
Toy example: the scores are illustrative similarity values, not output from a real model.
Where RAG breaks
Most RAG failures are retrieval failures. The model can only be as good as the passages it is given.
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
The method explained, an interactive model of it, and what it changes for a business.
Questions people ask
Is RAG the same as fine-tuning?›
No. Fine-tuning adjusts the weights inside the neural network during an expensive offline training run. RAG keeps the model weights frozen and changes what context is passed in the prompt at query time. Fine-tuning teaches tone and syntax; RAG provides live, citable facts.
Does RAG stop hallucinations?›
It dramatically cuts hallucinations when relevant passages are retrieved and strict system prompts enforce citation. However, if the retrieval engine misses the right passage or retrieves contradictory policies, the model can still hallucinate.
How many passages should be retrieved?›
There is no universal number. Too few passages can miss the one that holds the answer; too many give the model more to wade through and cost more per question. Liu et al. (2023) found that models use information in the middle of a long input less reliably than information at its start or end (Lost in the Middle). Start small, measure on your own questions, and adjust.