Practical Guide to LangChain & LlamaIndex: Build Powerful Retrieval‑Augmented Generation Apps

Introduction

The AI landscape is shifting fast. Two open‑source frameworks—LangChain and LlamaIndex—have emerged as the de‑facto building blocks for Retrieval‑Augmented Generation (RAG), conversational agents, and data‑aware LLM applications. Whether you’re a data scientist, a product engineer, or a startup founder, mastering these tools lets you connect large language models (LLMs) to your own data sources, create context‑rich chatbots, and automate complex workflows.

In this practical guide we will:

Explain core concepts (RAG, agents, data connectors) in plain language.
Walk through a complete end‑to‑end example that combines LangChain and LlamaIndex.
Show real‑world use cases from companies like Microsoft, IBM, and Ollama.
Compare the strengths of each framework in a concise table.
Provide actionable next steps and resources (including a few helpful books).

By the end of this post you’ll be ready to spin up your own LLM‑powered application, pick the right library for the job, and avoid common pitfalls that trip up beginners.

1. Why LangChain and LlamaIndex Matter

1.1 Retrieval‑Augmented Generation (RAG) in a nutshell

Traditional LLMs generate text based solely on the patterns they learned during pre‑training. RAG adds a retrieval step: the model first pulls relevant documents from an external knowledge base, then generates a response that references that content. This approach:

Improves factual accuracy – the model can quote the exact source.
Keeps responses up‑to‑date – you only need to refresh the index, not retrain the model.
Reduces hallucinations – the model is grounded in real data.

Both LangChain and LlamaIndex provide the plumbing that makes RAG practical.

1.2 What is LangChain?

LangChain is an AI agent engineering platform that offers a modular stack for building chains (sequences) of LLM calls, tool invocations, and memory management. It abstracts away boilerplate code for prompt templating, chat history handling, and function calling, letting you focus on the business logic of your agent — the same library that powers many enterprise‑grade bots today 2.

1.3 What is LlamaIndex?

Formerly called GPT Index, LlamaIndex acts as a data‑centric interface between raw data (PDFs, CSVs, APIs, etc.) and LLMs. It builds indexes (vector, keyword, or tree‑structured) that enable fast similarity search, then formats retrieved chunks into prompts. The library also bundles LlamaHub, a catalog of ready‑made data connectors for services like Google Drive, S3, and SQL databases 1.

1.4 When to use each library

Scenario	Best fit	Reason
Simple RAG over a few PDFs	LlamaIndex	Built‑in loaders & index types make setup trivial
Multi‑step agent that calls external APIs	LangChain	Chains, tools, and memory are first‑class concepts
Hybrid workflow (retrieval + tool use)	Combine both	LlamaIndex handles data, LangChain orchestrates actions
Need for visual UI or low‑code integration	LangChain (LangGraph)	Provides graph‑based UI for complex flows
Custom connector for a proprietary data lake	LlamaIndex (LlamaHub)	Easy to add new loaders and indexers

2. Setting Up the Development Environment

Before diving into code, make sure you have a clean Python environment (≥3.9) and the latest versions of the two libraries.

# Create a virtual environment
python -m venv rag-env
source rag-env/bin/activate

# Install LangChain, LlamaIndex, and a compatible LLM client (e.g., OpenAI)
pip install langchain llama-index openai tiktoken

If you prefer local LLMs such as those from Ollama, install the Ollama CLI and point LangChain to the local endpoint 1.

pip install ollama
export OLLAMA_BASE_URL="http://localhost:11434"

3. End‑to‑End Example: A Customer‑Support Bot for an E‑Commerce Site

We’ll build a bot that:

Retrieves relevant product manuals from a knowledge base (using LlamaIndex).
Answers the user’s question, citing the source.
Calls an external order‑status API if the question is about shipment — handled by LangChain.

3.1 Preparing the Data with LlamaIndex

Assume we have a folder ./manuals/ containing PDF specifications for three products: EchoSmart Speaker, Nimbus Drone, and AquaPure Water Filter.

from llama_index import SimpleDirectoryReader, GPTVectorStoreIndex, ServiceContext
from llama_index.embeddings import OpenAIEmbedding

# Load documents
documents = SimpleDirectoryReader("./manuals").load_data()

# Create an embedding model (OpenAI or Ollama)
embed_model = OpenAIEmbedding(model="text-embedding-3-large")
service_ctx = ServiceContext.from_defaults(embed_model=embed_model)

# Build a vector index
index = GPTVectorStoreIndex.from_documents(documents, service_context=service_ctx)

# Persist the index for later use
index.storage_context.persist("./persisted_index")

The index now stores each chunk’s embedding, enabling fast similarity search.

3.2 Querying the Index

from llama_index import QueryEngine

query_engine = index.as_query_engine()
retrieved = query_engine.query("How do I reset the EchoSmart speaker?")
print(retrieved)

The output contains the most relevant paragraph(s) plus a citation to the source PDF.

3.3 Wrapping Retrieval in a LangChain Tool

LangChain treats external functions as tools that an agent can call.

from langchain.tools import BaseTool
from typing import Any, Dict

class LlamaIndexRetrieverTool(BaseTool):
    name = "LlamaIndexRetriever"
    description = "Retrieves relevant product manual excerpts for a user query."

    def _run(self, query: str) -> str:
        result = query_engine.query(query)
        return str(result)

    async def _arun(self, query: str) -> str:
        raise NotImplementedError("Async not needed for this demo")

3.4 Adding an Order‑Status Tool (mocked)

import requests

class OrderStatusTool(BaseTool):
    name = "OrderStatus"
    description = "Gets the shipping status of an order given an order ID."

    def _run(self, order_id: str) -> str:
        # In a real app, replace with your order API endpoint
        response = {"order_id": order_id, "status": "Shipped", "eta": "2024‑11‑02"}
        return f"Order {order_id} is {response['status']}, ETA {response['eta']}."

    async def _arun(self, order_id: str):
        raise NotImplementedError

3.5 Building the LangChain Agent

from langchain.agents import initialize_agent, AgentType
from langchain.chat_models import ChatOpenAI

# LLM (OpenAI GPT‑4 or local Ollama model)
llm = ChatOpenAI(model_name="gpt-4o-mini", temperature=0)

# Register tools
tools = [LlamaIndexRetrieverTool(), OrderStatusTool()]

agent = initialize_agent(
    tools,
    llm,
    agent=AgentType.OPENAI_FUNCTIONS,  # Enables function calling
    verbose=True
)

3.6 Running the Bot

user_query = "My EchoSmart speaker won't connect to Wi‑Fi. How can I fix it?"
response = agent.run(user_query)
print(response)

What happens under the hood?

LangChain receives the user query and decides (via its internal planner) to call LlamaIndexRetriever.
The retriever returns a manual excerpt about Wi‑Fi setup.
The LLM synthesizes a step‑by‑step answer, citing the source.
If the user later asks “What’s the status of order #12345?”, the agent calls OrderStatusTool and returns the live shipping info.

This hybrid pattern—retrieval + tool usage—is exactly what many enterprise RAG bots require.

4. Real‑World Deployments

4.1 Microsoft Copilot for Business

Microsoft integrates LangChain‑style orchestration inside Copilot for Business, allowing the assistant to fetch internal SharePoint documents, run SQL queries, and call Azure Functions—all while staying within the corporate security perimeter.

4.2 IBM WatsonX Hackathon 2026

The IBM hackathon guide explicitly recommends using LangChain for multi‑step function calls and LlamaIndex for a Text‑to‑SQL task 4. Participants built a chat interface that could retrieve policy documents via LlamaIndex and then invoke WatsonX APIs through LangChain agents.

4.3 Ollama‑Powered Personal Knowledge Base

Developers who run local LLMs with Ollama often pair it with LlamaIndex’s LlamaHub connectors to index markdown notes, then use LangChain’s OpenAIFunctions‑style wrappers (compatible with Ollama) to query the local model without sending data to the cloud 1.

5. Comparison Table: Core Features at a Glance

Feature	LangChain	LlamaIndex
Primary focus	Agent orchestration – chains, memory, tool calling	Data indexing & retrieval – loaders, vector stores
Built‑in data connectors	Limited (via `langchain_community`)	LlamaHub offers >50 connectors (Google Drive, S3, SQL, etc.)
Prompt templating	`PromptTemplate`, `ChatPromptTemplate`	Uses built‑in `PromptHelper` to format retrieved chunks
Memory support	Session, summary, vector memory	Relies on external memory modules (can integrate LangChain memory)
Graph / workflow UI	LangGraph visual editor for complex graphs	No native UI; focus is on programmatic index creation
Compatibility with local LLMs	Yes (via `ChatOllama`, `ChatAnthropic`, etc.)	Yes (any embedding model; works with Ollama embeddings)
Community size (2026)	Large, backed by venture capital and enterprise partners	Growing rapidly; strong ties to the Llama ecosystem
Typical use‑case	Multi‑tool agents, autonomous reasoning	Retrieval‑first RAG, document‑centric apps

6. Deep Dive: Technical Terms Explained

Term	Simple definition	Why it matters
RAG (Retrieval‑Augmented Generation)	A two‑step process: fetch relevant text, then let the LLM generate a response.	Keeps answers factual and up‑to‑date.
Vector Store	A database that stores high‑dimensional embeddings for fast similarity search.	Enables semantic search across unstructured data.
Prompt Template	A reusable string with placeholders (e.g., `{question}`) that the LLM fills in.	Guarantees consistent prompting and reduces prompt‑engineering errors.
Tool / Function Calling	A way for an LLM to request execution of external code (e.g., API call).	Turns a pure text model into an actionable agent.
Memory	A mechanism that retains previous interactions (chat history, summary).	Allows agents to maintain context over long conversations.
LangGraph	A visual, node‑based editor for building complex LangChain workflows.	Helps non‑programmers design multi‑step agents without writing code.
LlamaHub	A curated library of data loaders/connectors for LlamaIndex.	Saves time when pulling data from SaaS platforms or custom sources.

7. Hands‑On Tips & Common Pitfalls

Pitfall	How to avoid it
Embedding mismatch – using a different embedding model for indexing vs. query.	Reuse the same `embed_model` instance for both index creation and query time.
Prompt overflow – sending too many retrieved chunks to the LLM, exceeding token limits.	Use LlamaIndex’s `PromptHelper` to truncate or summarize chunks before passing them to LangChain.
Tool over‑use – calling external APIs for every query, causing latency.	Add a decision layer in LangChain (via a custom `RouterTool`) that only invokes a tool when confidence is low.
Security leak – accidentally exposing proprietary documents in prompts logged to the console.	Redact sensitive fields in the retrieved text, or enable LangChain’s `log_to_stdout=False`.
State loss – forgetting to persist the index after updates.	Call `index.storage_context.persist()` after adding new documents.
Version drift – using mismatched library versions (e.g., LangChain 0.2.x with LlamaIndex 0.9.x).	Pin both libraries to compatible releases in `requirements.txt`.

8. Extending the Stack: From Prototype to Production

Deploy the index to a managed vector DB like Pinecone, Weaviate, or Qdrant for scalability. LlamaIndex supports all of them out of the box.
Wrap the LangChain agent in a FastAPI or Flask server so front‑end apps can call it via HTTP.
Add observability – use LangChain’s built‑in tracing (langchain.callbacks) and LlamaIndex’s logging to capture latency and token usage.
Secure API keys – store OpenAI, Azure, or custom API secrets in a vault (e.g., HashiCorp Vault or AWS Secrets Manager).
CI/CD pipelines – automate index rebuilding whenever source documents change (e.g., on a nightly GitHub Action).

9. Learning Resources (Books & Tutorials)

“Building RAG Applications with LangChain” – a hands‑on guide that walks through real‑world projects. Available on Amazon: Building RAG Applications with LangChain
“LlamaIndex Essentials: From Data to Answers” – covers data ingestion, indexing strategies, and prompt engineering. Find it here: LlamaIndex Essentials
“Practical AI Agents” – explores agent architecture, memory, and tool calling across frameworks. See the book on Amazon: Practical AI Agents

These books complement the official docs and the YouTube crash course that walks through the LlamaIndex vs. LangChain comparison 1.

10. Frequently Asked Questions

Q1: Do I need both libraries?
Answer: Not always. If you only need to retrieve text from a static corpus, LlamaIndex alone suffices. For multi‑tool agents that also need memory, LangChain shines. Combining them gives you the best of both worlds.

Q2: Can I use a closed‑source LLM like Claude or Gemini?
Answer: Yes. Both frameworks expose a generic LLM interface. Just install the provider’s SDK (e.g., anthropic or google-generativeai) and pass the client to LangChain’s ChatAnthropic or ChatGoogleGenerativeAI.

Q3: How do I keep the index up‑to‑date with new documents?
Answer: Use LlamaIndex’s add_documents() method, then call persist() again. Automate this with a file‑watcher or a webhook from your document management system.

Conclusion

LangChain and LlamaIndex have quickly become the twin pillars of modern LLM engineering. By mastering retrieval‑augmented generation, tool calling, and data indexing, you can create AI agents that are both

This article was created using generative AI.