Retrieval that finds the right paragraph

Your policy assistant answers four questions out of five well and gets the fifth badly wrong. You look at the trace. The model reasoned sensibly about three paragraphs that had nothing to do with the question, because those were the three paragraphs it was given.

The generation step was fine. Retrieval handed it the wrong material, and no prompt fixes that. This is the single most common failure in question answering over your own documents, and it is invisible unless you look at what was retrieved rather than at what was said.

So before touching the prompt, answer one question: was the right paragraph in the context at all? If it was not, you have a retrieval bug. If it was and the answer is still wrong, you have a generation bug. These need opposite fixes and people spend weeks on the wrong one.

Chunk on meaning, not on length

The default is to split every document every 500 characters. It is fast, it is one line of code, and it cuts a table in half, separates a heading from the rule underneath it, and leaves you with a chunk that reads "and in that case the thirty day limit does not apply" with no way of knowing what case.

Split on the structure the document already has. Headings, clauses, list items, question and answer pairs. Most documents you care about are structured, and the structure is what a human uses to find things. A chunk should be a thing somebody could read on its own and act on.

When a section is genuinely too long, overlap the splits by a paragraph, and prepend the heading path to each piece:

Refunds > International orders > Exceptions
Where the item was shipped outside the EU, the thirty day limit does not apply.

That prefix costs you a dozen tokens and rescues a chunk that would otherwise be unusable. It also gives the model something to cite.

The question and the answer do not look alike

Embedding search matches text that looks similar. A question and the passage that answers it often do not.

"Can I get my money back on something I bought in France" is close in wording to other questions about France. The passage that answers it says "refunds", "consumer rights" and "EU member states", and may never use the word money.

Two things help. Index a short generated summary or a set of likely questions alongside each chunk, so there is something question-shaped to match. And run a keyword search in parallel with the embedding search, then merge, because exact terms like an order id, a product code or a section number are precisely what embeddings are worst at and plain text search is best at.

Top five is a guess, not a setting

Almost every system retrieves a fixed number of chunks, usually because that is the default. But the number of relevant chunks is a property of the question. A question about one clause needs one. "What are all our termination obligations" needs eleven, and returning five means missing six with no sign that anything is missing.

Retrieve generously, then use a reranker or a cheap model pass to drop what does not help. Sorting twenty candidates down to the four that matter is a much easier job than picking four out of a thousand in one step, and it is where most of the quality is.

Measure retrieval on its own

Write down thirty real questions. For each one, note by hand which chunk contains the answer. Now you can ask: how often is that chunk in what we retrieved?

That single number is the ceiling on the whole system. If the right passage arrives 70% of the time, no prompt work takes you past 70%, and you will spend a month proving it. It also tells you immediately whether a chunking change helped, which is otherwise unknowable, because end to end quality moves for six reasons at once.

Where this doesn't help

Retrieval answers questions whose answer sits in a passage. It does not answer questions about the collection: how many policies mention refunds, which team owns the most documents, what changed since March. Those are queries over metadata, and dressing them up as search produces confident nonsense built from whichever ten chunks came back.

It is also poor at questions requiring two facts from opposite ends of a corpus, where each one is only relevant given the other. Neither looks like the question on its own, so neither gets retrieved.

And retrieval cannot fix a corpus that disagrees with itself. If three documents state different refund windows because two are out of date, a search engine will find all three and the model will pick one. That is a content problem wearing a retrieval costume, and the fix is to delete the old documents.

The move

Before you touch the prompt, check whether the right passage was retrieved at all. Then build the thirty question set and measure that one number, because it is the ceiling on everything downstream.