RAG & Vector Databases AI & LLMs
RAG and Vector Databases Explained: A Practical 2026 Guide
How RAG and vector databases work: embeddings, chunking, hybrid search and reranking, plus pgvector vs Pinecone vs Qdrant and when long context is enough.
Ask a chatbot about your company's refund policy and it will happily invent one. Retrieval-augmented generation fixes that by fetching the right documents first and handing them to the model, and RAG and vector databases are now the standard way to build AI that answers from your own data.
Here is how the pieces fit together, which vector database to pick, and when a long context window means you can skip RAG entirely.
Key takeaways
- RAG retrieves relevant chunks of your content and adds them to the prompt, so answers stay grounded and current.
- Most quality problems come from chunking and retrieval, not the language model. Hybrid search plus reranking is the biggest easy win.
- If your whole knowledge base fits comfortably in the context window, try pasting it in before building a pipeline.
RAG by the numbers
Menlo Ventures' survey of 600 US IT decision-makers found RAG adoption jumped from 31% to 51% in a year. Postgres (15%) and MongoDB (14%) also remained common homes for vectors.
How RAG works, step by step
A RAG system has two halves: an indexing pipeline that runs whenever your content changes, and a query pipeline that runs every time someone asks a question.
- Chunk your documents into passages, typically a few hundred tokens each.
- Embed each chunk with an embedding model, which turns text into a list of numbers (a vector) that captures its meaning.
- Store the vectors, the original text and useful metadata (source, date, permissions) in a vector database.
- Retrieve: embed the user's question and find the chunks whose vectors are closest to it.
- Rerank the candidates with a more precise model and keep only the best few.
- Generate: pass those chunks to the language model with instructions to answer from them and cite sources.
Embeddings and chunking
Embeddings are why "cancel my subscription" can match a help article titled "Ending your plan". Similar meanings land close together in vector space, even when the words differ.
Chunking matters more than most people expect. Tiny chunks lose context; huge ones dilute the match. Split on headings and paragraphs, add a little overlap, and keep the document title with every chunk. Anthropic's contextual retrieval research goes further, prepending a short, model-written summary of where each chunk sits in its document.
Hybrid search and reranking
Pure vector search is fuzzy. It can miss exact terms such as product codes, error messages and surnames. Hybrid search runs a keyword search (usually BM25) alongside the vector search and merges the results.
A reranker then scores each candidate against the question directly. In Anthropic's tests, contextual embeddings plus contextual BM25 cut the top-20 retrieval failure rate by 49%, and adding reranking took the reduction to 67% (from 5.7% to 1.9%).
Choosing a vector database
A vector database stores embeddings and finds nearest neighbours quickly, usually with an approximate index such as HNSW. The options fall into three camps: an extension to a database you already run, an open-source engine you can host yourself, or a fully managed service.
| Database | Type | Hosting | Best for |
|---|---|---|---|
| pgvector | Postgres extension | Self-hosted or managed Postgres | Teams already on Postgres |
| Pinecone | Proprietary, purpose-built | Fully managed cloud | Production with zero ops |
| Qdrant | Open source (Rust) | Self-hosted or Qdrant Cloud | Fast search with rich filters |
| Weaviate | Open source (Go) | Self-hosted or Weaviate Cloud | Built-in hybrid search |
| Milvus | Open source | Self-hosted or Zilliz Cloud | Very large collections |
| Chroma | Open source | Embedded, self-hosted or Chroma Cloud | Prototypes and local apps |
For most small teams, pgvector is the sensible default. Your vectors live next to your normal data, permissions and backups work as they already do, and version 0.8.0 added iterative index scans to fix filtered queries that returned too few results, as the PostgreSQL project announced. A minimal setup looks like this:
CREATE EXTENSION vector;
CREATE TABLE chunks (
id bigserial PRIMARY KEY,
content text,
embedding vector(1536)
);
CREATE INDEX ON chunks
USING hnsw (embedding vector_cosine_ops);
SELECT content FROM chunks
ORDER BY embedding <=> $1 LIMIT 5;
Move to a dedicated engine when scale, heavy filtering or operations become a burden.
Pick the database last. Get chunking, hybrid search and evaluation right first; they move answer quality far more than the storage engine.
Do long context windows make RAG obsolete?
Not quite, but they shrink the cases where you need it. Google began offering a one-million-token context window with Gemini 1.5 Pro in 2024, and several flagship models at the time of writing accept similar volumes. Anthropic suggests that if a knowledge base is under about 200,000 tokens (roughly 500 pages), you can simply include it all in the prompt, using prompt caching to keep costs down.
Beyond that, RAG still wins on cost, speed and freshness. Long prompts are also less reliable than they look: the "Lost in the Middle" study found models do best when key facts sit at the start or end of the input, and Chroma's context rot research across 18 models found reliability falls as input length grows, even on simple tasks.
How to evaluate a RAG system
Evaluate retrieval and generation separately, or you will never know which half is failing.
- Build a test set of 50 to 100 real questions, each tagged with the document that should answer it.
- Retrieval metrics: does the right chunk appear in the top results (recall at k), and how high does it rank?
- Answer metrics: is the answer faithful to the retrieved text, does it address the question, and are citations correct? Open-source tools such as Ragas automate these checks with an LLM as judge.
- Rerun the suite whenever you change the chunk size, embedding model, prompt or database.
Common RAG failure modes
- Bad parsing: tables, PDFs and scanned pages turn into garbled text before embedding even starts.
- Missing exact matches: vector-only search misses codes and names. Add keyword search.
- Stale index: documents change but embeddings do not. Schedule re-indexing.
- Permission leaks: retrieval ignores who is asking. Filter by access rights at query time.
- Too much context: dozens of chunks bury the answer. Rerank and keep a handful.
- No "I don't know": instruct the model to say when the sources do not contain the answer.
FAQ
Do I need a vector database to use RAG?
No. Small projects can keep embeddings in memory or in Postgres with pgvector. A dedicated vector database earns its place as collections, traffic and filtering needs grow.
Is RAG better than fine-tuning?
They solve different problems. RAG supplies knowledge that changes; fine-tuning shapes behaviour, style and format. Many production systems use both.
Which embedding model should I use?
Shortlist two or three well-supported models and compare them on your own test set. Public leaderboards do not always reflect your documents.
Ready to go further? Learn when to retrain a model instead in our fine-tuning guides, or let our team build a production-ready assistant on your own documents with our AI automation services.
Sources: Menlo Ventures, Anthropic, PostgreSQL, Google, Lost in the Middle (arXiv), Chroma Research