What is retrieval-augmented generation (RAG)?

How RAG works

A RAG system runs in three stages: retrieve, augment, generate. When a user asks a question, the system converts that question into a numeric vector called an embedding, which captures its meaning in a high-dimensional space. It then searches a vector database for the stored text passages whose embeddings are closest to the query, typically ranked by cosine similarity. This semantic search returns the top-k most relevant chunks rather than exact keyword matches.

In the augment step, those retrieved passages are inserted into the model's prompt alongside the original question, usually with an instruction such as answer using only the context below. The large language model then generates its response conditioned on that supplied context. Because the evidence sits in the prompt, the model reasons over fresh, specific material instead of guessing from parameters. AWS describes RAG as optimizing an LLM's output so it references an authoritative knowledge base outside its training data before responding.

Preparing the knowledge base is a separate, offline step called ingestion or indexing. Source documents are split into chunks, each chunk is passed through an embedding model, and the resulting vectors are stored in the vector database with their original text and metadata. This index is what the retriever searches at query time.

Why RAG matters

Standalone language models store knowledge in their weights, which creates three well-known limits: they can hallucinate confident but false statements, their knowledge is frozen at their training cutoff, and they cannot see private or proprietary data. RAG addresses all three by moving the authoritative facts out of the weights and into a searchable store the model consults on demand.

By grounding answers in retrieved evidence, RAG reduces hallucination and, critically, lets the system show its sources. IBM notes that RAG lets LLMs include references in their responses so users can verify the cited material, adding transparency that a pure generative model cannot offer. This provenance is why RAG is now the default architecture for enterprise question-answering, customer support, and internal knowledge assistants.

The core components

A production RAG stack has four moving parts. The embedding model turns text into vectors; common choices include OpenAI text-embedding models, Cohere Embed, and open-source options such as sentence-transformers on Hugging Face. The vector database stores and searches those vectors at scale; widely used systems include Pinecone, Weaviate, Milvus, Qdrant, and the pgvector extension for PostgreSQL.

The retriever executes the similarity search and often adds a re-ranking step, in which a second model rescores the top candidates for relevance before they reach the prompt. Finally the generator, the LLM itself, composes the answer. Orchestration frameworks such as LangChain and LlamaIndex wire these components together and handle chunking, prompt assembly, and citation formatting.

Common use cases

RAG is used wherever an organization needs a model to answer over a specific corpus it does not own the training rights to memorize. Typical deployments include customer-support bots grounded in help-center articles, internal assistants over company wikis and policy documents, legal and financial research tools that must cite exact clauses, and developer assistants that query product documentation.

The pattern also underpins consumer AI search. Perplexity, Google's AI Overviews, and ChatGPT search all retrieve live web results and generate an answer over them, with links back to the sources, which is RAG applied to the open web rather than a private index.

Limitations

RAG is only as good as its retrieval. If the search returns irrelevant, outdated, or contradictory passages, the model will faithfully generate a wrong answer, a failure mode summarized as garbage in, garbage out. Retrieval quality depends heavily on chunking strategy, embedding quality, and index freshness, so poorly tuned pipelines can underperform a plain model.

RAG also does not guarantee the model will stay within the retrieved context; models can still ignore or misread the evidence, so citations must be checked rather than trusted blindly. Long retrieved contexts add latency and token cost, and sensitive corpora raise access-control and data-governance requirements. RAG reduces hallucination but does not eliminate it.

RAG vs fine-tuning

RAG and fine-tuning are complementary, not competing. RAG adds knowledge at inference time by supplying facts in the prompt, which makes it ideal for information that changes often or must be cited, and it requires no model training. Fine-tuning updates the model's weights on curated examples to change its behavior, format, tone, or task competence, which is better for teaching a durable skill or style than for injecting a large body of facts.

A common rule of thumb: use RAG to change what the model knows and fine-tuning to change how the model behaves. Many production systems combine both, fine-tuning a model to follow a house style or output schema while using RAG to feed it current, source-backed facts. See our companion explainer on fine-tuning for how weight updates and techniques like LoRA and RLHF work.

Frequently asked questions

RAG vs fine-tuning, which is better?

Neither is universally better; they solve different problems. RAG is better when the model needs current, proprietary, or citable facts, because it retrieves them at query time without retraining. Fine-tuning is better when you need to change the model's behavior, tone, or task format durably. Many systems use both together.

Does RAG stop hallucinations?

RAG substantially reduces hallucination by grounding answers in retrieved evidence, and it lets the system cite sources so answers can be verified. It does not eliminate hallucination entirely: if retrieval returns poor context or the model misreads it, the answer can still be wrong. Citations should be checked, not trusted blindly.

What is a vector database in RAG?

A vector database stores text as embeddings, numeric vectors that capture meaning, and finds the passages most similar to a query using distance metrics such as cosine similarity. In RAG it is the search layer that retrieves relevant context for the model. Common examples include Pinecone, Weaviate, Milvus, Qdrant, and PostgreSQL with pgvector.

Who invented RAG?

The technique was introduced in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Patrick Lewis and colleagues at Meta AI (then Facebook AI Research), presented at NeurIPS 2020. It combined a pre-trained seq2seq model with a dense vector index of Wikipedia and set state-of-the-art results on three open-domain question-answering tasks.

What is an embedding in RAG?

An embedding is a list of numbers that represents the meaning of a piece of text, so that texts with similar meaning sit close together in vector space. RAG embeds both the stored documents and the incoming query, then retrieves the documents whose embeddings are nearest to the query embedding.

Do I need a vector database to do RAG?

Not always. For small corpora you can compute similarity in memory or use keyword search, and some RAG systems retrieve directly from the web or a SQL store. A dedicated vector database becomes valuable at scale, when you need fast semantic search over millions of chunks with metadata filtering.

Related

Explainers: LLM, Fine-tuning

Glossary: Embedding (vector), Vector database, Large language model (LLM), Hallucination

Sources

• arXiv — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al., 2020)
• AWS — What is RAG (Retrieval-Augmented Generation)?
• IBM Think — What is retrieval-augmented generation (RAG)?
• Google for Developers — Machine Learning Glossary: Generative AI