RAG: why your answers are wrong, and how to fix it

Separate the two halves of the problem

A retrieval-augmented system has two stages: finding the right extracts, then writing from them. A wrong answer comes from one or the other, and the remedies have nothing in common.

The test is simple: display the extracts passed to the model. If they contain the answer and the generated text contradicts it, the problem is in the generation. If they do not contain it, no prompt tuning will save the answer.

The five most frequent causes

1. Chunking that separates the question from its answer: the table is in one extract, its caption in another.
2. Obsolete documents indexed next to current versions, with no date and no priority.
3. Purely semantic search that misses exact references — article numbers, product codes.
4. A context so long that the relevant extract is drowned in the middle.
5. Access rights ignored: the assistant sees documents the user should not see.

Build an evaluation set before optimising

Thirty real questions, with their expected answer and the document that contains it, are enough to turn impressions into measurements. Without that set, every change feels like an improvement.

Measure two distinct rates: the share of questions for which the right document is retrieved, and the share where the produced answer is correct. The first caps the second.

The fixes that actually change the results

Own the refusal to answer

A business assistant must be able to say it does not know. That is a product decision before it is a setting: would you rather have a plausible answer or an honest silence? In a professional context, the second option is almost always the right one.

Requiring a verifiable citation for every claim makes refusal natural: no source, no answer. It is also what lets the user check in one click rather than trust.

A RAG system is not tuned in the prompt: it is tuned in what you give it to read, and in what you allow it not to answer.