RAG (retrieval-augmented generation)

In short: RAG combines a language model with a search: first matching passages from your own documents are fetched, then the model answers the question on that basis.

In more detail: The model does not have to “know” everything but gets the relevant material in the prompt. That reduces hallucinations, keeps answers current and allows citations.

In Depth

1. Split documents into sections (chunks).
2. Compute and store an embedding of each section.
3. Turn the question into an embedding too and search the most similar sections.
4. Give sections and question to the model, instructing it to answer only from them.

• Quality depends on the search: Poor sections yield poor answers.
• Privacy: With local models the documents stay on your own machine.

See also: Context window, Hallucination (AI)