5 / 6 RAG Architecture & Retrieval

Buried at Twelve

You concatenate the top 20 retrieved chunks into a 30,000 token prompt. Logging shows the chunk containing the answer was retrieved, ranked 12th, and the model still answered from an unrelated chunk near the top. What is the most effective change to context assembly?

A

Set the sampling temperature to 0 so decoding becomes deterministic and the model weighs every part of the context evenly instead of favouring the text near the start

B

Rerank the candidates and keep only the few strongest, placing the top-scoring chunks at the beginning and end of the context rather than in the middle

C

Raise top-k from 20 to 50 so the relevant chunk is certain to sit somewhere in the assembled context window

D

Order the chunks by document publication date so the newest material is read by the model first and the oldest last