Chapter 16: Retrieval Augmented Generation (RAG) — Mastering LangGraph: From State Machines to Production AI Agents

In Chapter 8, we gave our Agent a search_web() tool. This allowed the LLM to access real-time public information.

But what if you are building an Internal HR Agent? The LLM cannot search Google for your company's private employee handbook, vacation policies, or proprietary source code.

To give an Agent access to private knowledge, we use Retrieval Augmented Generation (RAG).

16.1 The Mechanics of RAG

RAG is a pipeline that bridges the gap between your private documents and the LLM's prompt.

It consists of two phases:

Ingestion (Offline): You chop your massive PDF documents into small chunks, convert them into numbers using an Embedding Model (like OpenAI's text-embedding-3-small), and save them in a Vector Store.
Retrieval (Runtime): When a user asks a question, you convert their question into an embedding, find the most mathematically similar chunks in the Vector Store, and inject those chunks into the LLM's prompt as context.

The Chunking Problem

A common mistake beginners make is dumping entire 50-page PDFs into the Vector Database as a single chunk. Embeddings represent the average semantic meaning of the text. If you average 50 pages of text, the resulting vector becomes a blurry, useless mess.

You must use a RecursiveCharacterTextSplitter or semantic chunking tools to break your documents into 500-token paragraphs before embedding them.

16.2 Building a Retrieval Node in LangGraph

In a standard RAG pipeline, the retrieval happens sequentially before the LLM generates an answer.

def retrieval_node(state: GraphState):
    query = state["user_query"]
    
    # Search the Vector Store for the top 5 most relevant document chunks
    retrieved_docs = vector_store.similarity_search(query, k=5)
    
    # Combine the chunks into a single text string
    context_text = "\n\n".join([doc.page_content for doc in retrieved_docs])
    
    # Save the context to the state so the LLM node can read it!
    return {"retrieved_context": context_text}

def generation_node(state: GraphState):
    prompt = f"""Answer the user based ONLY on the following context:
    {state['retrieved_context']}
    
    Question: {state['user_query']}
    """
    response = llm.invoke(prompt)
    return {"final_answer": response.content}

This is naive RAG. It works, but it is rigid. What if the user just says "Hello"? The system will still blindly search the vector store for "Hello", wasting compute, generating a bad vector, and retrieving garbage documents.

16.3 Agentic RAG

Because LangGraph gives us Control Flow and Tools, we can upgrade from Naive RAG to Agentic RAG.

Instead of forcing a retrieval on every turn, we turn the Vector Database into a Tool (@tool). We give it to the LLM and let the Agent decide if it needs to search the private database.

@tool
def search_company_policies(query: str) -> str:
    """Use this tool to search the private HR database for company policies."""
    docs = vector_store.similarity_search(query)
    return "\n".join([d.page_content for d in docs])

llm_with_tools = llm.bind_tools([search_company_policies])

Now, if the user says "Hello", the Agent responds conversationally. If the user says, "What is the maternity leave policy?", the Agent actively requests the search_company_policies tool, reads the result in a ToolMessage, and formulates the answer.

16.4 Query Transformations

Users are notoriously terrible at writing search queries.

If a user asks: "What is the policy?", and you pass that directly to the Vector DB, it will fail. The vector representation of "What is the policy?" is meaningless.

In advanced LangGraph architectures, we insert a Query Transformation Node before the Retrieval Node.

def query_transform_node(state: GraphState):
    # Look at the whole conversation history
    history = state["messages"]
    
    # Ask an LLM to rewrite the query into a perfect search string
    rewritten_query = llm.invoke(f"""
    Given the chat history: {history}
    Rewrite the user's latest message into a standalone search query optimized for a Vector Database.
    """)
    
    # Update the state with the optimized query!
    return {"optimized_query": rewritten_query.content}

Now, "What is the policy?" gets transformed into "Maternity leave policy 2024 Acme Corp", resulting in dramatically better retrieval.

16.5 Corrective RAG (CRAG)

One of the most powerful architectures you can build in LangGraph is Corrective RAG.

What happens if the Vector DB returns documents that don't actually answer the user's question? A standard agent might hallucinate an answer anyway based on irrelevant context.

In CRAG, we introduce a Grader Node.

[Retrieval Node] ➔ [Grader Node]
                       ↙       ↘
              [Relevant]      [Irrelevant]
                  ↓                ↓
            [Generation]      [Rewrite Query & Retry]

Using LangGraph's conditional routing, the Grader Node uses with_structured_output to score the retrieval.

class GradeDocuments(BaseModel):
    binary_score: str = Field(description="Documents are relevant to the question, 'yes' or 'no'")

def grader_node(state: GraphState):
    grader_llm = llm.with_structured_output(GradeDocuments)
    
    prompt = f"Does this document: {state['retrieved_context']} answer the query: {state['user_query']}?"
    score = grader_llm.invoke(prompt)
    
    return {"is_relevant": score.binary_score}

def route_crag(state: GraphState):
    if state["is_relevant"] == "yes":
        return "generation_node"
    else:
        return "query_transform_node" # Loop back and try again!

Self-RAG (Web Fallbacks)

If the Grader Node loops 3 times and still cannot find relevant documents in the private Vector Store, you can implement Self-RAG. The graph gracefully accepts that the private knowledge base lacks the answer, and routes execution to a public Web Search node as a last resort fallback.

16.6 Multi-Hop RAG

Sometimes, answering a question requires connecting dots across multiple documents. "Who is the manager of the person who wrote the Q3 Financial Report?"

A single vector search cannot answer this. An Agentic RAG loop handles this gracefully via multi-hop reasoning (Chapter 9 ReAct):

Agent searches: "Author of Q3 Financial Report"
Observes: "Author is Sarah Jenkins"
Agent searches again: "Manager of Sarah Jenkins"
Observes: "Manager is David Smith"
Generates Final Answer.

16.7 Summary

By integrating RAG into LangGraph, you transform static language models into brilliant domain experts equipped with enterprise knowledge. Agentic RAG, Query Transformations, and CRAG topologies ensure that your agents research dynamically and rigidly verify their sources before answering.