Retrieval-augmented generation, end to end
Ingest, chunk, embed, retrieve, rerank, generate with citations, and evaluate retrieval separately from generation.
RAG gives a model facts it was not trained on, at answer time. Most RAG quality problems are retrieval problems, so measure retrieval on its own before touching the prompt.
The pipeline
| Stage | Decisions | Typical mistake |
|---|---|---|
| Ingest | Connectors, change detection, permissions captured per document | Forgetting deletes and permission changes |
| Chunk | By structure (headings, sections), with overlap; keep titles and breadcrumbs in each chunk | Fixed 500-token chunks that split tables and lose context |
| Embed | Model choice, dimension, language coverage | Changing embedding model without re-embedding everything |
| Index | Hybrid: keyword (BM25) plus vector; metadata filters | Vector-only search that misses exact codes, names and acronyms |
| Retrieve | Query rewriting, top-k, filters (tenant, ACL, date) | No permission filter at query time |
| Rerank | Cross-encoder or LLM reranker on the top 20 to 50 | Skipping it, then raising top-k until the context is noise |
| Generate | Cite chunk ids; allow "I don't know" | No instruction to refuse when sources do not answer |
Evaluate the two halves separately
- Retrieval: for each question in a golden set, is the right document in the top k? (recall@k, MRR).
- Generation: is every claim supported by a retrieved chunk (faithfulness)? Does it answer the question (relevance)?
- Use an LLM judge for faithfulness only after checking it agrees with human labels on a sample.
When not to use RAG
If the corpus fits comfortably in context and changes rarely, put it in the prompt and cache it. If the task needs the model to learn a style or format, fine-tuning or examples may fit better. If the answer requires computation over structured data, give the model a SQL or API tool instead of retrieved text.
Practice
Written for this site to drill the lesson. Not reported interview questions.
- Your RAG bot answers questions about product codes badly. Which stage do you suspect, and what do you change?
- Design the golden set for an HR policy assistant: where do questions come from, how many, and who labels them?
- Explain hybrid search to an architect who only knows SQL
LIKEqueries.
Check yourself
Why hybrid search?
Vectors capture meaning; keyword search catches exact names, codes and acronyms that embeddings blur.
What two things do RAG evals measure separately?
Retrieval quality (recall@k) and generation quality (faithfulness, relevance).