In the previous chapters, we have delved deep into the inner world of large language models (LLMs). We have learned to guide them through prompt engineering and customize them through efficient fine-tuning. At this point, the LLM in our hands is like a knowledgeable, well-trained "superbrain." However, in its default state, this brain is cut off from the world. Its knowledge is frozen at the moment of the training cutoff. It cannot access the latest information, nor can it interact with external tools to execute tasks. It can "speak," but it cannot "do." It can "recall," but it cannot "query."
To truly upgrade an LLM from a powerful "language model" to an "intelligent assistant" capable of solving real-world problems, we must break through this barrier and connect it to the outside world. This chapter focuses on two core technical paradigms for achieving this goal: Retrieval-Augmented Generation (RAG) and Agent.
RAG: Equipping the LLM with an "External Knowledge Base"
We know that LLMs suffer from "hallucination" and "outdated knowledge." The RAG architecture was born to address this pain point. Its core idea is that before asking the LLM to answer a question, we first retrieve the most relevant information fragments from an external, reliable, and updatable knowledge base (such as company documents, product manuals, or databases). We then augment the prompt with this information as context, and finally have the LLM generate an answer based on this reliable information. RAG is like equipping the LLM with a powerful "search engine" and an "open bookshelf," enabling it to answer questions based on private or real-time knowledge, greatly improving the accuracy and timeliness of answers.
Agent: Giving the LLM the Ability to "Think" and "Act"
If RAG is about letting the LLM "read ten thousand books," then Agent is about letting the LLM "travel ten thousand miles." An Agent system elevates the LLM from a passive text generator to an active, goal-oriented task executor. It uses the LLM as its core "brain" and, through a "Think-Act-Observe" loop, decides what to do next. It can be given a set of tools, such as invoking a calculator, querying a weather API, executing code, or searching the web. When faced with a complex task, the Agent autonomously decomposes the task, selects and uses the appropriate tools, observes the results, and proceeds with further thinking and action based on those results until the task is completed.
In this chapter, we will delve into these two exciting technologies:
- Detailed RAG Architecture: We will dissect every aspect of RAG, from text chunking and vectorization to the application of vector databases and the complete process from retrieval to generation.
- Introduction to Agent Development: We will learn the core conceptual framework of Agents (such as ReAct) and, with the help of powerful open-source frameworks like LangChain or LlamaIndex, quickly get started with developing our own Agent and equipping it with practical tools.
- Dual Hands-On Projects: Through two practical projects -- building an intelligent Q&A bot based on company documents (RAG) and developing a simple Agent that can query the weather and use a calculator -- we will turn theoretical knowledge into tangible engineering practice.
Mastering RAG and Agent means mastering the two "killer applications" of current LLM application development. You will be able to build truly useful, reliable, and intelligent applications that interact with the real world. Now, let us equip our large models with "eyes" and "hands" and unlock their true potential.
9.1 Detailed RAG Architecture
RAG is an architecture that combines information retrieval with language model generation, aiming to enhance the quality of LLM answers by introducing external knowledge.
Core Advantages of RAG
- Mitigating Hallucination: The LLM is forced to answer based on the provided context, rather than fabricating information.
- Real-Time Knowledge Updates: You do not need to retrain the expensive LLM. Simply update the external knowledge base, and the model can access the latest information.
- Traceability and Interpretability: You can show users which source documents the answer is based on, increasing the answer's credibility.
- Data Security: Private data is stored in your own knowledge base without being used to train the model, reducing the risk of data leakage.
A typical RAG process consists of two phases: Data Indexing and Retrieval & Generation.
9.1.1 Data Indexing Phase: Building Your Knowledge Base
This phase is performed offline. Its purpose is to process your raw documents (such as PDF, TXT, Markdown, HTML, etc.) into a format suitable for fast retrieval.
Step 1: Load and Split
Raw documents are often too long to fit into the LLM's context window. Therefore, the first step is to split long documents into smaller, meaningful text chunks.
- Loaders: Using document loaders from libraries like LlamaIndex or LangChain, you can easily read files in various formats.
- Splitters:
- Fixed-size Chunking: The simplest method, splitting by a fixed character count (e.g., 1000 characters) with a certain overlap (e.g., 100 characters) to maintain semantic continuity.
- Recursive Character Text Splitter: A smarter method that attempts to split along a sequence of delimiters (such as
\n\n,\n,), prioritizing the preservation of paragraph and sentence integrity. - Semantic Chunking: A more advanced method that analyzes the semantic similarity between text blocks to determine split points, aiming for each chunk to be a semantically complete unit.
The Art of Chunking:
- Chunk size is a critical hyperparameter.
- Too small: May lose important contextual information, resulting in retrieved segments that are too fragmented.
- Too large: May contain too much noise unrelated to the query, increasing the processing burden on the LLM.
- A common starting point is 512 to 1024 tokens.
Step 2: Embedding
Once the chunks are created, we need to convert each text chunk into a vector. This process is called embedding. This vector represents the coordinates of the text chunk in a multidimensional semantic space.
- Embedding Model: We use a pretrained sentence transformer model to accomplish this task. These models are specifically designed to map text into dense vector spaces that capture their semantics.
How to Choose an Embedding Model?
- MTEB (Massive Text Embedding Benchmark): The gold standard for evaluating embedding model performance.
- Mainstream Choices:
- English:
BAAI/bge-large-en-v1.5(among the top performers on MTEB at the time of writing),sentence-transformers/all-MiniLM-L6-v2(lightweight and efficient). Model rankings change quickly; consult the latest MTEB leaderboard when selecting. - Chinese/Multilingual:
BAAI/bge-m3(powerful multilingual model),infgrad/stella-base-zh-v2(excellent Chinese model).
- English:
- Implementation: The
sentence-transformerslibrary or Hugging Face'stransformerslibrary can be used to easily load and use these models.
from sentence_transformers import SentenceTransformer
# Load the embedding model
model = SentenceTransformer('BAAI/bge-base-en-v1.5')
# Prepare text chunks
chunks = ["RAG stands for Retrieval-Augmented Generation.",
"It enhances LLMs with external knowledge."]
# Perform vectorization
embeddings = model.encode(chunks)
print(embeddings.shape) # (2, 768) -> 2 text chunks, each a 768-dimensional vector
9.1.2 Vector Database Selection and Application
Now we have a large collection of text chunks and their corresponding vectors. When a user asks a question, we need to find the text chunk most "similar" to the question. In a knowledge base with millions of text chunks, computing similarity one by one is not feasible. This is where vector databases come in.
Vector databases are specifically designed for efficiently storing and retrieving high-dimensional vectors. Their core technology is Approximate Nearest Neighbor (ANN) search.
How It Works (Brief):
ANN algorithms build special index structures (such as IVF, HNSW) to avoid exhaustive search. While they cannot guarantee finding 100% of the most similar vectors, they sacrifice a tiny amount of precision for several orders of magnitude improvement in search speed, which is perfectly acceptable for real-time applications.
Mainstream Vector Database Options:
In-Memory/Local Libraries:
- FAISS (Facebook AI Similarity Search): A high-performance vector similarity search library developed by Facebook AI. It is a C++ library with a Python interface. It is powerful and extremely fast, but does not provide database management capabilities itself. It is more like a "search engine library."
- ChromaDB: An open-source vector database designed for AI applications. It is very easy to use, with a simple Python API that supports local persistent storage, making it ideal for rapid prototyping and small-to-medium projects.
Server-Side/Distributed Databases:
- Pinecone, Weaviate, Milvus: These are more feature-rich databases that can be deployed as independent services. They support distributed scaling, metadata filtering, real-time index updates, and other advanced features, making them suitable for large-scale production environments.
Using ChromaDB Example:
import chromadb
# 1. Initialize the ChromaDB client (can persist to disk)
client = chromadb.PersistentClient(path="/path/to/db")
# 2. Create or get a collection
collection = client.get_or_create_collection(name="my_knowledge_base")
# 3. Add data (Indexing)
# Assume we already have chunks and embeddings
collection.add(
embeddings=embeddings.tolist(), # Embedding vectors
documents=chunks, # Raw text chunks
metadatas=[{"source": "doc1.pdf"}, {"source": "doc2.txt"}], # Metadata
ids=[f"chunk_{i}" for i in range(len(chunks))] # Unique IDs
)
# --- Retrieval Phase ---
# 4. Query
query_text = "What is RAG?"
query_embedding = model.encode([query_text])[0].tolist()
# Retrieve the top-k most similar results
results = collection.query(
query_embeddings=[query_embedding],
n_results=2 # Return the 2 most similar
)
print(results['documents'])
# [['RAG stands for Retrieval-Augmented Generation.', 'It enhances LLMs with external knowledge.']]
9.1.3 The Complete Process from Retrieval to Generation
Now that we have covered indexing and retrieval, we can connect the second phase of the complete RAG pipeline.
Step 3: Retrieve
- Receive the user's query.
- Use the same embedding model as during indexing to convert the query into a query_embedding.
- In the vector database, use the query_embedding to perform a similarity search and retrieve the top-K most relevant text chunks (retrieved_chunks).
Step 4: Augment and Generate
Build the Prompt: Combine the retrieved text chunks and the user's original query into a carefully designed prompt.
Context information is below. --------------------- {context_str} <-- Concatenate retrieved_chunks into a single string --------------------- Given the context information and not prior knowledge, answer the query. Query: {query_str} Answer:Call the LLM: Send this augmented prompt to the LLM.
Get the Answer: The LLM generates the final answer based on the provided context.
This process ensures that the LLM's answers are grounded in evidence, greatly improving the quality and reliability of the answers.
9.2 Introduction to Agent Development
If RAG gives the LLM a "read-only" external brain, then an Agent gives the LLM the ability to "think" and "act," allowing it to interact with the outside world in a "read-write" manner.
9.2.1 The Core Idea of Agents: The ReAct Framework
ReAct (Reasoning and Acting) is one of the most core and fundamental conceptual frameworks for current Agent systems. It explicitly decomposes the LLM's thinking process into a "Thought -> Action -> Observation" loop.
- Thought: The LLM analyzes the current task goal and available information, performs reasoning, and decides what action to take next. This thought process is human-readable text generated by the LLM itself.
- Action: Based on the thought, the LLM decides to call a tool and specifies the input required for that tool (Action Input). For example,
Action: Calculator, Action Input: 2+2. - Observation: The Agent system executes this Action (e.g., runs the calculator and gets the result
4), and feeds the tool's output back to the LLM as "observation" information.
After receiving this Observation, the LLM begins a new cycle: based on the new observation, it forms the next Thought, decides on the next Action, and continues until it determines the task is complete and generates the final answer.
ReAct Example (Simplified):
Task: "What is the square of Einstein's age?"
- Thought 1: I need to know Einstein's age first. I do not have this information, so I need to search.
- Action 1:
Search("Einstein's age") - Observation 1: "Albert Einstein (March 14, 1879 -- April 18, 1955), lived to be 76."
- Thought 2: Now I know Einstein's age is 76. The task asks for the square of his age, which is 76 squared. I need a calculator for this.
- Action 2:
Calculator("76^2") - Observation 2: "5776"
- Thought 3: I have the result 5776. I have completed all steps of the task, and I can now give the final answer.
- Final Answer: The square of Einstein's age is 5776.
Through this loop, the LLM breaks down a complex task requiring external information into multiple simple, executable steps and uses tools to compensate for its own limitations.
9.2.2 Getting Started with LangChain/LlamaIndex Frameworks
Implementing an Agent system from scratch is very complex, requiring handling a large number of details such as prompt templates, tool calls, output parsing, and loop control. Fortunately, two powerful open-source frameworks -- LangChain and LlamaIndex -- greatly simplify Agent development.
- LangChain: A comprehensive, highly flexible LLM application development framework. It provides all the components needed to build an Agent (LLM interfaces, prompt templates, output parsers, tools, etc.) and allows you to freely combine them like building blocks. Its learning curve is relatively steep, but its degree of freedom is high.
- LlamaIndex: Initially focused on RAG, it has now also developed powerful Agent capabilities. Its abstraction level is higher, typically allowing you to implement a fully functional RAG or Agent system with less code, making it ideal for getting started quickly.
The Process of Creating a Simple Agent with LangChain:
- Define Tools: Define the list of tools the Agent can use.
- Initialize the LLM: Select and configure an LLM (e.g.,
ChatOpenAIorHuggingFaceHub). - Create a Prompt Template: Design a prompt template that follows the ReAct framework, telling the LLM what tools are available and how it should think and act.
- Build the Agent: Combine the LLM, tools, and prompt to create an Agent.
- Create an Agent Executor: This is a controller responsible for running the Agent loop.
- Run the Agent: Call the executor to complete the task.
9.2.3 Giving the Agent Tools
Tools are the bridge for the Agent to interact with the outside world. Any functionality that can be called programmatically can be encapsulated as a tool.
Common Tool Types:
- Calculator: Performs mathematical operations.
- Search Engine: Performs web searches via APIs (e.g., Google Search API, Tavily).
- Python REPL: Executes Python code. Extremely powerful but also high-risk.
- Database Query: Connects to a database and executes SQL queries.
- API Calls: Calls any third-party API (weather, stocks, maps, etc.).
- RAG Retriever: Encapsulate the RAG retriever we built earlier as a tool. When the Agent determines it needs to look up information from a private knowledge base, it can call this tool.
In LangChain, defining a tool typically requires:
name: The tool's name, which the LLM uses to decide which tool to call.description: Extremely important. A clear description of the tool's functionality. The LLM relies entirely on this description to understand the tool's purpose and when to use it.func: The actual Python function executed behind the tool.
from langchain.tools import tool
@tool
def get_weather(city: str) -> str:
"""Returns the current weather for a given city."""
# Implement the actual logic to call a weather API here
if city == "Beijing":
return "Beijing is sunny today, 25 degrees Celsius."
else:
return f"Sorry, I cannot query the weather for {city}."
# The Agent can now use this tool via the name 'get_weather'.
9.3 Hands-On Project 1: Building an Intelligent Q&A Bot Based on Company Documents (RAG)
Project Objective: Assume we have some Markdown documents about company policies. We will build a RAG system that allows employees to ask questions about these policies.
Tech Stack: transformers (for embeddings), chromadb, langchain
Step 1: Prepare Data and Environment
Create some
.mdfiles, such aspolicy_leave.md,policy_expense.md.policy_leave.md: "The company provides 15 days of paid annual leave per year. Leave requests must be submitted through the HR system two weeks in advance."policy_expense.md: "Business travel transportation expenses for employees are reimbursable. Economy class must be chosen for flights. Receipts are required for taxi fares."Install libraries:
pip install langchain chromadb sentence-transformers
Step 2: Index the Data
# rag_indexing.py
from langchain.document_loaders import DirectoryLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.vectorstores import Chroma
# 1. Load documents
loader = DirectoryLoader('./company_policies/', glob="/*.md")
documents = loader.load()
# 2. Split text
text_splitter = RecursiveCharacterTextSplitter(chunk_size=500, chunk_overlap=50)
texts = text_splitter.split_documents(documents)
# 3. Load the embedding model
embeddings = HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5')
# 4. Create and persist the vector database
# This stores the vector data in the 'db' directory
vectordb = Chroma.from_documents(documents=texts,
embedding=embeddings,
persist_directory="./db")
vectordb.persist()
print("Index created.")
Step 3: Build the QA Chain
# rag_qa.py
from langchain.vectorstores import Chroma
from langchain.embeddings import HuggingFaceEmbeddings
from langchain.chat_models import ChatOllama # Use a local LLM running on Ollama, can also use ChatOpenAI
from langchain.chains import RetrievalQA
# 1. Load the embedding model and vector database
embeddings = HuggingFaceEmbeddings(model_name='BAAI/bge-base-en-v1.5')
vectordb = Chroma(persist_directory="./db", embedding_function=embeddings)
# 2. Initialize the LLM
# Assume you already have Llama 3 running locally via Ollama: ollama run llama3
llm = ChatOllama(model="llama3")
# 3. Create a Retriever
retriever = vectordb.as_retriever(search_kwargs={"k": 2}) # Retrieve the 2 most relevant chunks
# 4. Create a RetrievalQA chain
# chain_type="stuff" is the simplest method, "stuffing" all retrieved documents into a single prompt
qa_chain = RetrievalQA.from_chain_type(
llm=llm,
chain_type="stuff",
retriever=retriever,
return_source_documents=True # Also return source documents for traceability
)
# 5. Ask a question
query = "How many days of annual leave do I get per year?"
result = qa_chain({"query": query})
print("Answer:", result['result'])
print("Sources:", [doc.metadata['source'] for doc in result['source_documents']])
After running rag_qa.py, the system will first retrieve text chunks about the leave policy from the vector database, then send them along with the question to the LLM, ultimately obtaining an accurate answer that also tells you which document the answer came from.
9.4 Hands-On Project 2: Developing a Simple Agent That Can Query Weather and Use a Calculator
Project Objective: Build an Agent that can understand natural language questions and autonomously decide whether to use a weather query tool or a calculator tool to answer.
Tech Stack: langchain, langchain-openai (or langchain-community for Ollama)
Step 1: Define Tools
# agent_tools.py
from langchain.tools import tool
@tool
def get_weather(city: str) -> str:
"""Returns the current weather for a given city."""
print(f"--- Calling weather tool, city: {city} ---")
if "Beijing" in city:
return "Beijing today: cloudy to sunny, 15-28 degrees Celsius."
elif "Shanghai" in city:
return "Shanghai today: light rain, 20-25 degrees Celsius."
else:
return f"Sorry, I cannot query the weather for {city}."
@tool
def calculator(expression: str) -> str:
"""A simple calculator that evaluates a mathematical expression."""
print(f"--- Calling calculator tool, expression: {expression} ---")
try:
# Using eval has security risks; use a safer library in real projects
result = eval(expression)
return str(result)
except Exception as e:
return f"Calculation error: {e}"
tools = [get_weather, calculator]
Step 2: Build and Run the Agent
# agent_run.py
from langchain_openai import ChatOpenAI
from langchain import hub
from langchain.agents import create_react_agent, AgentExecutor
from agent_tools import tools
# 1. Initialize the LLM
# Need to set your OpenAI API Key: os.environ["OPENAI_API_KEY"] = "..."
llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
# 2. Get the ReAct framework's prompt template
# This is an optimized standard ReAct prompt provided by LangChain
prompt = hub.pull("hwchase17/react")
# 3. Create the Agent
# This function binds the LLM, tools, and prompt together
agent = create_react_agent(llm, tools, prompt)
# 4. Create the Agent Executor
agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True) # verbose=True prints the complete thought chain
# 5. Run the Agent
# Test 1: Needs to call the weather tool
response1 = agent_executor.invoke({"input": "What is the weather like in Beijing today?"})
print("Final answer:", response1["output"])
print("\n" + "="*50 + "\n")
# Test 2: Needs to call the calculator tool
response2 = agent_executor.invoke({"input": "What is 3 to the 5th power?"})
print("Final answer:", response2["output"])
When you run agent_run.py, the verbose=True allows you to clearly see each step of the ReAct loop:
For question 1, the LLM will Thought: "I need to check the weather in Beijing," then Action: get_weather("Beijing").
For question 2, the LLM will Thought: "I need to calculate 3 to the 5th power," then Action: calculator("3**5").
This perfectly demonstrates how an Agent autonomously selects and uses the right tools based on task requirements.
Chapter Summary
In this chapter, we took two critical steps toward transforming an LLM from a "closed brain" into an "intelligent agent" capable of interacting with the outside world.
We first deeply analyzed the Retrieval-Augmented Generation (RAG) architecture. We learned its complete process from data indexing (loading, splitting, vectorization) to retrieval and generation, and we mastered how to use vector databases like ChromaDB to build and query knowledge bases. Through RAG, we equipped the LLM with a powerful "external knowledge base," effectively solving its knowledge limitations and hallucination problems.
Then, we explored the more cutting-edge Agent technology. We understood its core ReAct framework, which, through a "Think-Act-Observe" loop, enables the LLM to break down tasks and call external tools. With the help of frameworks like LangChain, we learned how to quickly build an Agent capable of autonomous decision-making and action.
Finally, through two practical hands-on projects, we transformed the theoretical knowledge of RAG and Agents into code that can be run and experienced. We built the prototype of an enterprise-level intelligent Q&A bot and developed a simple Agent that can use tools.
After completing this chapter, you have mastered the two most core and popular paradigms in current LLM application-layer development. You are no longer just a user or fine-tuner of LLMs. You have become an "architect" capable of designing and building complex, practical AI applications. The applications you build will no longer be limited by the model's own knowledge but will be able to connect to unlimited external data and functionality, creating immense value in the real world. In the final chapter of this book, we will look ahead to the future of AI engineering, discussing how to turn the applications we build into truly stable, reliable, production-grade systems through CI/CD, monitoring, and evaluation.