What is RAG? Retrieval-augmented generation, explained simply

RAG in one sentence

RAG, short for retrieval-augmented generation, is a way of building an AI system so that it first searches a set of documents for passages related to your question, then writes its answer using those passages. Retrieval is the looking up. Generation is the writing. The looked-up text augments, or adds to, what the model already knows.

The term comes from a 2020 research paper by a team at Facebook AI Research, University College London and New York University. They described models that combine what a language model learned in training with an external index of documents, in their case Wikipedia, accessed by a retriever (Lewis et al., 2020). Most tools that let you "chat with your documents" use some version of this idea.

Why AI models need it

A language model on its own answers from patterns it absorbed in training. That has two problems. It can't know anything that wasn't in its training data, such as your lecture notes from last Tuesday. And when it doesn't know, it can still produce a fluent, confident answer that is wrong. The original RAG paper names both issues: models struggle to access and precisely use knowledge, and providing provenance for their decisions and updating their world knowledge remain open research problems (Lewis et al., 2020).

A later survey of the field lists the same motivations: hallucination, outdated knowledge, and reasoning that is hard to trace (Gao et al., 2023). Retrieval helps with all three, because the answer can be tied to specific text that you can read yourself. For more on why models make things up in the first place, see why AI answers make things up.

How RAG works, step by step

Imagine you've saved a semester of physiology notes and you ask: "What does the mitral valve do?" A RAG system goes through roughly these steps:

1. 1Prepare the documents. Before any question is asked, your notes are split into smaller passages, often a paragraph or so each, and stored in an index that can be searched by meaning rather than only by exact words.
2. 2Search. Your question is compared with every passage, and the system picks the few that look most relevant. Here it might find a paragraph from week 4 about the left side of the heart and a line from a lab write-up about heart sounds.
3. 3Add the passages to the prompt. The selected passages are placed alongside your question, with an instruction along the lines of "answer using these sources".
4. 4Generate. The model writes an answer, such as "The mitral valve sits between the left atrium and left ventricle and stops blood flowing back into the atrium when the ventricle contracts."
5. 5Cite. Many systems also show which passages they used, so you can open them and check.

That's the whole idea. Real systems add refinements, like rewriting the question before searching or reranking the results, and the survey groups these into stages it calls Naive RAG, Advanced RAG and Modular RAG (Gao et al., 2023). The core loop stays the same: search, then answer.

What RAG fixes

• Answers can use your material. The model can draw on documents it never saw in training, including your own notes.
• Answers can be checked. When a system shows its sources, you can open the passage and compare.
• Knowledge can be updated without retraining. Add a new note and it can be found in the next search.

In the original paper, the RAG models produced more specific, diverse and factual language than a comparable model without retrieval (Lewis et al., 2020). That's a real improvement, but it's a comparison between two research models, not a promise about any product.

Where RAG still goes wrong

Retrieval narrows the problem. It doesn't remove it. Three failures are worth knowing.

The search misses the right passage. If your note says "bicuspid valve" and the search is looking for "mitral", a weak retriever might not connect them. The model then answers from whatever it did find, or from its training, and the result can look just as confident.

The model doesn't use what it was given. Researchers tested how well language models use long inputs and found that performance is often highest when the relevant information is at the beginning or end of the input, and significantly degrades when the model has to use information in the middle of a long context (Liu et al., 2023). So even a correct passage can be overlooked.

The citation doesn't support the sentence. A 2023 audit of four popular generative search engines found that, on average, only 51.5% of generated sentences were fully supported by their citations, and only 74.5% of citations supported their associated sentence (Liu, Zhang & Liang, 2023). Those were web search tools, not notes apps, but the lesson carries over: a citation tells you where to look, not that the claim is right.

How to check a RAG answer

1. 1Open the cited passage. Read the sentence it supports, not just the title of the note.
2. 2Check that the passage says what the answer says. Watch for small shifts, like "usually" becoming "always".
3. 3Notice what's missing. If the answer covers only one of three lectures on a topic, the search may have missed the others.
4. 4Rephrase if needed. Try the term your notes actually use.

This is the approach we took with Lumyo. You choose which notes a chat reads, each answer cites its sources, and tapping a source opens the exact line. When the chosen notes don't cover a question, it says it doesn't know instead of guessing. You still need to check, but checking takes one tap.