RAG

In depth

RAG became the default architecture for vertical AI products in 2023-2024 because it solved two problems at once: it kept proprietary data out of model weights (safer, more flexible) and it let the product update its knowledge instantly by editing the corpus rather than retraining a model (faster, cheaper).

The typical RAG stack has four layers: an embedding model (converts text to vectors), a vector database (pgvector, Pinecone, Weaviate), a retriever (usually hybrid — semantic + keyword), and the generation model that reads the retrieved context and answers.

Quality depends far more on the retrieval step than on the generation step. Most 'bad AI answer' failures in a RAG system trace back to the retriever returning irrelevant or stale context, not to the model itself.

Formula & example

Rules of thumb

• Invest in retrieval quality before generation quality.
• Use hybrid search (semantic + keyword) — pure semantic misses exact matches.
• Chunk documents at semantic boundaries, not arbitrary lengths.
• Cache hot retrievals; most queries hit the same top 500 documents.

Common mistakes

• Skipping evaluation — you cannot tell if retrieval is good without measurement.
• Using the default vector index without tuning recall or nprobe — hurts accuracy.
• Embedding raw HTML or PDFs; pre-process to clean text first.

Put it into practice

FAQ

Related terms