RAG pipeline: chunking, reranking and evaluation

In short: RAG (retrieval-augmented generation) gives a language model matching passages from your own documents so it answers on that basis. Quality depends more on the search than on the model.

The steps

1. Ingestion: read documents (PDF, web, database) and clean them.
2. Chunking: split into pieces of about 200 to 800 tokens with some overlap. Pieces that are too small lose context, too large ones dilute the search. Better to split at paragraphs and headings than mid-sentence.
3. Embedding: turn each piece into a vector, see Vector databases and similarity search.
4. Retrieval: fetch the most similar pieces for the question (for example the best 20).
5. Reranking: a more precise model reorders these hits and keeps the best 3 to 5.
6. Answer: question and pieces go to the language model, which should answer with sources.

Typical errors

• The right passage is not found (poor chunking or embedding).
• The passage is found but the model ignores it or adds inventions, see Hallucination (AI).
• Outdated documents stay in the index.

Evaluation

Build a list of real questions with known answers and measure separately: is the right passage found (recall@k)? Is the answer correct and supported by the passages? More in AI evaluation.

See also: Hybrid search: BM25 plus vectors