Loop Engineering for RAG Generation: iterate top-k one at a time

Naive RAG ships batch by default. Retrieval returns top-5, the LLM gets all 5, the answer comes back. It works, and on hard questions (comparison, listing) it is the right choice. But the silent cost is paid on every other question: the easy factual ones where the top-1 already had the answer. This article walks the second regime and the dispatch that picks per question.

Retrieval hands generation an ordered top-K. There are two ways to feed it in, and they cost very differently.

The default pattern across most RAG tutorials looks like this:

top_k = retrieval(question, k=5)
answer = generation(question, top_k)

It runs in two steps for every question. Token cost is roughly generation_cost(question + 5 chunks of context). Latency is roughly retrieval + one LLM call. And the LLM happily processes the 5 chunks even when the top-1 was already sufficient and chunks 2..5 added zero information.

Concretely: the user asks “what is the effective date of this policy?”. Retrieval returns 5 line-windows where the keyword "effective" appears. The first one is the answer (“effective from January 1, 2026”). Chunks 2..5 are signatures, footnotes, and a paragraph about historical effective dates of past policies. The LLM reads all 5 to extract the same date the first one already had. On a corpus of one document, the cost is rounding error. On a corpus of 50k policies, this is real money per month.

The sequential regime treats the K candidates as an ordered list and asks the generation brick to validate sufficiency at each step:

for i, candidate in enumerate(top_k):
    answer = generation(question, [candidate])
    if answer.answer_found and answer.complete_answer_found:
        break

The sufficiency signal lives inside the typed contract Article 8A introduced. The AnswerWithEvidence schema exposes answer_found (was the question’s information present in this candidate?) and complete_answer_found (was the entire answer present, not a fragment?). The loop reads those fields, not a custom heuristic.

In the effective date example above: generation runs once on the top-1 candidate, the LLM returns answer_found = True, complete_answer_found = True, the loop exits. Tokens spent: generation_cost(question + 1 chunk of context). That is 1/5 of the batch cost for this question, on a pipeline that handles thousands of similar lookups per day.

Batch mode keeps the default behaviour: one LLM call sees all K candidates and produces one typed answer. Its case is built on three question types where sequential breaks down:

• Listing questions: “list all exclusions in this contract”. The answer is every matching candidate, not the first. Sequential would stop at top-1 (one exclusion found) and miss the other four. Batch is the only correct mode.
• Comparison questions: “is the premium higher than the previous year’s?”. The answer requires both candidates (this year + last year) in the same call so the LLM can compare them. Sequential would extract each one independently and lose the join.
• Tight-score retrieval: when the top-K candidates’ relevance scores are within 5% of each other, retrieval cannot reliably promote top-1 to the top. Batch lets the LLM arbitrate with the full evidence visible.

Cost analysis: batch always pays the generation_cost(question + K chunks of context) once. Sequential pays generation_cost(question + 1 chunk of context) in the easy case (top-1 sufficient) and generation_cost(question + 1 chunk) × K in the worst case (every candidate insufficient). On a typical enterprise corpus with K = 5, sequential is cheaper on average for factual lookups (~80% of typical traffic) and more expensive for listing / comparison (~20%).

The clean architecture does not pick batch or sequential globally. It picks per question, using the parsed question_df row from brick 2. Question shape, decomposition pattern, and intent drive the choice:

The dispatcher reads question_df.answer_shape and question_df.decomposition and routes. Naive RAG has no way to make this distinction because it has no parsed question to read from.

The same routing table holds across sectors and professions. Different domains carry the same shape patterns and the same sequential / batch decision flows out of them:

In every row, the sequential column is a single typed value (Amount, Date, Boolean) and benefits from the top-1 stop. The batch column is a list or a comparison and needs all K candidates visible at once. The dispatcher’s table covers all five sectors with the same logic.

Sequential mode leans on one thing: the generation brick reporting whether the candidate it just read was enough. That signal, and the rules that stop the loop, live here.

Sequential mode only works if the generation brick can self-report whether the candidate it just saw contained the answer. The typed contract from Article 8A is what makes this possible:

class AnswerWithEvidence(BaseModel):
    value: Any
    evidence: list[Span]
    answer_found: bool
    complete_answer_found: bool
    confidence: float = Field(ge=0, le=1)
    caveats: list[str] = []

The sequential loop reads answer_found and complete_answer_found, not a confidence float or a custom heuristic. The clean separation between the two booleans (from Pattern 4 of Article 8ter) is what makes the loop deterministic. Found + incomplete says “continue to the next candidate”; found + complete says “stop”; not found says “continue or give up at K”.

A confidence float would force a threshold (e.g. “stop at 0.8”), and that threshold drifts model to model. Two booleans do not drift.

The sequential loop has three exits, not one:

• Sufficiency: answer_found and complete_answer_found on a candidate. Stop, ship the answer.
• Exhaustion: every one of the K candidates seen, none sufficient. Stop, return answer_found = False (a first-class answer the validator passes through).
• Budget: a token or time budget set by the dispatcher. Useful when the corpus is large and a runaway sequential loop would blow the cap. Same shape as the bounded iteration M4 (loop engineering) names.

Naive sequential implementations skip the third exit and burn tokens forever on edge cases. The series version sets a budget upfront and the dispatcher logs it on every call.

Two regimes, two cost profiles, and one boundary the series does not cross.

To make the trade-off real, here is a back-of-envelope for an enterprise insurance Q&A workload:

• 100 questions / day, K = 5, average chunk size = 600 tokens.
• Batch baseline: 100 × generation_cost(question + 5×600 tokens) = ~330k input tokens per day for generation.
• Sequential (80% top-1 sufficient): 80 × generation_cost(question + 600) + 20 × generation_cost(question + 5×600) (worst case for the 20% complex questions) = ~115k input tokens per day.

The ratio is 65% saving on input tokens for generation on this workload. The exact ratio depends on the easy-question proportion and the chunk size; the principle is that sequential is the cheap default once the typed contract is in place. Batch is reserved for the question types that need it.

A natural next step is to let the LLM decide between batch and sequential per question (agentic dispatch). The series stops short of this. The dispatcher in Section 2 is deterministic-dispatcher (one of the three approaches catalogued in Article 6C), not LLM-decides. The reason is the same one that holds across the series: audit. The same question on the same day must route the same way; an LLM that re-plans the dispatch per call cannot give that guarantee.

If you need a stronger agentic loop (the LLM picks which candidates to look at, in what order, with what scope), the bigger article on adaptive RAG loops is the right place. This article keeps the scope to the deterministic sequential vs batch decision driven by the parsed question.

The fixed top-K + batch pipeline is the right default for the question types where every candidate matters. It is the wrong default for the factual lookups that make up most enterprise traffic. The series adds two pieces: the typed sufficiency signal (answer_found, complete_answer_found) from Article 8A, and the dispatch table from Section 2. Together they let the generation brick stop after the first candidate when that is enough, and process all K when the question type requires it. The decision is made by the parser, not by the LLM, which keeps the audit trail intact.

The sequential / batch decision sits at the boundary between retrieval and generation. The articles that frame each side: