RAG (Retrieval-Augmented Generation)

RAG is a technique that improves LLM outputs by retrieving relevant information from a knowledge base before generating a response. It grounds the model in facts instead of relying solely on its training data.

Category: Ai (also: Architecture)

Author: Atomic Glue Editorial Team

## Definition

[Retrieval-Augmented Generation (RAG)](/glossary/rag-ai) is an [AI](/glossary/artificial-intelligence) architecture that combines a retrieval system with a generative language model. When a user asks a question, RAG first searches a vector database or search index for relevant documents, then feeds those documents into the LLM as context for generation. This approach solves LLMs' fundamental problem: they only know what they were trained on, and they hallucinate when asked about anything outside that training data. RAG gives the model access to your private data, current information, and domain-specific knowledge without fine-tuning. It is the most widely adopted pattern for production LLM applications because it is practical, effective, and does not require retraining the model.

## How RAG Works

The RAG pipeline has two main phases: **Indexing (offline):** Documents are split into chunks, each chunk is converted to a vector embedding, and the embeddings are stored in a vector database (e.g., Pinecone, Weaviate, pgvector). **Inference (online):** When a query arrives, it is embedded using the same model. The vector database performs a similarity search to find the most relevant chunks. These chunks are injected into the LLM prompt as context. The LLM generates a response grounded in that context. This pattern is deceptively simple but requires careful tuning: chunk size and overlap, embedding model selection, retrieval strategy (hybrid search combining vector and keyword), reranking of results, and prompt template design all significantly impact quality.

## RAG vs Fine-Tuning

RAG and fine-tuning serve different purposes and are often combined: - **RAG** is for injecting new information (company documents, recent news, user-specific data). It is dynamic: update the knowledge base, and the model has access immediately. No retraining needed. - **Fine-tuning** is for changing the model's behavior, tone, or style. It teaches the model how to respond rather than what to know. Use RAG when you need factual accuracy and up-to-date information. Use fine-tuning when you need the model to follow a specific format, tone, or reasoning pattern. Use both for maximum effect.

## Advanced RAG Patterns

**Agentic RAG:** The LLM decides when to retrieve, which tool to use, and whether to refine the query before retrieval. **Graph RAG:** Uses a knowledge graph instead of flat vector search to capture entity relationships. Better for multi-hop questions. **Multi-hop RAG:** The system performs multiple retrievals, using results from one to inform the next. **Self-RAG:** The model retrieves multiple passages, generates answers from each, and then evaluates and selects the best one. **Corrective RAG:** After retrieval, the system evaluates relevance and re-queries if results are insufficient.

## In code

# Simplified RAG pipeline (pseudocode)
from openai import OpenAI
from my_vector_db import VectorDB

db = VectorDB()
client = OpenAI()

def rag_answer(query):
    # Retrieve
    query_embedding = client.embeddings.create(input=query, model="text-embedding-3-small")
    relevant_chunks = db.similarity_search(query_embedding, top_k=5)
    context = "\n\n".join(relevant_chunks)
    
    # Generate
    prompt = f"Context:\n{context}\n\nQuestion: {query}\n\nAnswer:"
    response = client.chat.completions.create(
        model="gpt-4o",
        messages=[{"role": "user", "content": prompt}]
    )
    return response.choices[0].message.content

## Common questions

Q: Does RAG eliminate hallucination?

A: No, but it dramatically reduces it. If the retrieved context contains the answer, the model will almost always use it. But if retrieval fails (wrong chunks, incomplete coverage), the model may still hallucinate. RAG requires robust retrieval evaluation and fallback strategies.

Q: What vector database should I use?

A: For small-scale (under 1M vectors), pgvector on PostgreSQL is simple and effective. For larger scale, Pinecone, Weaviate, or Qdrant. For production, consider managed services to avoid operational overhead. Start simple, scale up when needed.

Q: How do you evaluate RAG quality?

A: Evaluate both retrieval metrics (recall, precision, MRR) and generation metrics (faithfulness, relevance, answer correctness). Tools like RAGAS, TruLens, and LangSmith provide evaluation frameworks. Always test on real user queries, not synthetic ones.

## Key takeaways

• RAG grounds LLM responses in retrieved facts, dramatically reducing hallucination.
• It is the most practical production pattern: no retraining needed, just update the knowledge base.
• Quality depends on chunking strategy, embedding choice, retrieval method, and prompt design.
• RAG and fine-tuning are complementary, not competing, approaches.

## Related entries

• [Large Language Model](atomicglue.co/glossary/large-language-model)
• [Vector Embeddings](atomicglue.co/glossary/vector-embeddings)
• [AI Agent / Autonomous Agent](atomicglue.co/glossary/ai-agent)
• [Prompt Engineering](atomicglue.co/glossary/prompt-engineering)

Last updated July 2026. Permalink: atomicglue.co/glossary/rag-ai