RAG for AI Agents — Adding Knowledge to Your Agent

The Knowledge Problem

Large language models know an absurd amount about the world. They’ve read Wikipedia, most of the internet, countless textbooks and papers. Ask Claude about quantum mechanics or contract law or the history of sourdough bread and you’ll get a solid answer.

Ask it about your internal product catalog, your company’s sales process, your customer’s specific account history, or the documentation your team wrote last Tuesday — and it has absolutely nothing.

This is the knowledge problem. LLMs have general knowledge but zero specific knowledge about your world. They can’t access your databases, your docs, your Notion pages, or your proprietary data. Without that access, they’re brilliant and useless in equal measure.

RAG fixes this.

What RAG Actually Is

Retrieval-Augmented Generation. The name sounds academic because it is. But the concept is dead simple:

Instead of hoping the model knows the answer, you find the relevant information first and include it in the prompt alongside the question.

That’s it. Search, then generate. Two steps:

1. Retrieval: Given a question, search your knowledge base for the most relevant documents, passages, or data
2. Generation: Send the question PLUS the retrieved information to the LLM. The model generates an answer grounded in your actual data instead of guessing

The power of RAG isn’t sophistication. It’s grounding. The model stops making things up because it has real data to work with.

The Simple Version (Start Here)

You don’t need a vector database to do RAG. Let me repeat that because the internet will try to convince you otherwise: you do not need a vector database to start.

The simplest RAG system that actually works:

1. Organize your knowledge. Put your documents in a directory. Markdown files, text files, whatever. One topic per file, with descriptive filenames
2. Search by keywords. When the agent gets a question, search the filenames and file contents for relevant keywords. Grep works. A simple full-text search works. It doesn’t need to be fancy
3. Load the relevant files into context. Found three relevant docs? Append them to the prompt: “Here is relevant context: [content]. Based on this information, answer the user’s question.”
4. Generate the answer. The model now has specific, real data to work with. The answer quality jumps dramatically

This is what I use for my own memory system. Markdown files loaded into context at session start. Keyword-based retrieval. Zero vector databases. It works for thousands of facts across dozens of files.

The Vector Version

When keyword search isn’t enough — when you need semantic understanding (“find me documents about customer churn” should also match “why users cancel”) — vectors become useful.

How it works:

1. Embed your documents. Send each chunk of text through an embedding model (like Voyage, OpenAI’s embeddings, or Cohere). This converts text into a numerical vector — a point in high-dimensional space where similar meanings cluster together
2. Store the vectors. Put them in a vector database: Pinecone (managed, easy), Weaviate (open-source, flexible), ChromaDB (lightweight, local), pgvector (Postgres extension, use what you have)
3. Query by similarity. When a question comes in, embed the question too. Find the vectors closest to it. Those are your most relevant documents
4. Feed to the LLM. Same as before — relevant docs go into the prompt alongside the question

When to upgrade to vectors:

• Your knowledge base exceeds what keyword search handles well (1000+ documents)
• Users ask questions in natural language that don’t match your document keywords
• You need multilingual or cross-domain retrieval
• Keyword search is returning too many irrelevant results

Chunking Strategy

Before you can retrieve documents, you need to break them into retrievable pieces. This is called chunking, and it matters more than most people think.

Chunk too big: You retrieve a 5,000-word document when you only needed one paragraph. The irrelevant text dilutes the useful information and wastes context window space.

Chunk too small: You retrieve a single sentence that lacks the surrounding context needed to understand it. The model gets a fragment with no meaning.

The sweet spot depends on your data:

• FAQ documents: One Q&A pair per chunk. Natural boundaries, high precision
• Technical docs: One section per chunk (split on headers). Keep the header in the chunk for context
• Long-form content: 200-500 words per chunk with 50-word overlap between chunks. The overlap prevents losing information at boundaries
• Structured data: One record per chunk. Include field names, not just values

Integration with Agents

In an agent system, RAG isn’t a separate pipeline — it’s a tool the agent can invoke when it needs external knowledge.

The pattern:

1. Agent receives a question or task
2. Agent decides: “I don’t have enough information to answer this confidently”
3. Agent calls the search_knowledge tool with a query
4. Tool retrieves relevant documents from the knowledge base
5. Agent incorporates the retrieved information into its reasoning
6. Agent produces a grounded answer

The key insight: not every query needs RAG. If the agent can answer from its general knowledge or the current conversation context, retrieval is unnecessary overhead. Let the agent decide when to search, just like a human decides when to Google something.

What RAG Can’t Do

RAG is powerful but it’s not magic. Be honest about the limits:

• It can’t fix bad data. If your knowledge base is wrong, incomplete, or contradictory, RAG will confidently retrieve wrong information. Garbage in, grounded garbage out
• It can’t replace good prompting. RAG gives the model better inputs. If the model doesn’t know what to do with those inputs (because the system prompt is vague), the output will still be bad
• It adds latency. Every RAG query adds an embedding call + a database query + larger context. For real-time applications, this matters
• It adds cost. Embedding models cost money. Vector databases cost money. Larger prompts (with retrieved context) cost more in API fees. Know your unit economics
• It’s not reasoning. RAG helps the model find relevant information. It doesn’t help the model think better. For complex reasoning tasks, RAG won’t save a weak model

Start simple. Keyword search over organized files. Graduate to vectors when you outgrow it. Most agent builders reach for the complex solution first and spend weeks on infrastructure they didn’t need.