Loop Engineering for RAG Technology: Iterate top-k One at a Time

Naive RAG ships batch by default. Retrieval returns top-5, the LLM will get all 5, the reply comes again. It really works, and on arduous questions (comparability, itemizing) it’s the proper alternative. However the silent price is paid on each different query: the simple factual ones the place the top-1 already had the reply. This text walks the second regime and the dispatch that picks per query.

Retrieval arms era an ordered top-Ok. There are two methods to feed it in, they usually price very in another way.

The default sample throughout most RAG tutorials appears like this:

top_k = retrieval(query, ok=5)
reply = era(query, top_k)

It runs in two steps for each query. Token price is roughly generation_cost(query + 5 chunks of context). Latency is roughly retrieval + one LLM name. And the LLM fortunately processes the 5 chunks even when the top-1 was already adequate and chunks 2..5 added zero data.

Concretely: the person asks “what’s the efficient date of this coverage?”. Retrieval returns 5 line-windows the place the key phrase "efficient" seems. The primary one is the reply (“efficient from January 1, 2026”). Chunks 2..5 are signatures, footnotes, and a paragraph about historic efficient dates of previous insurance policies. The LLM reads all 5 to extract the identical date the primary one already had. On a corpus of 1 doc, the associated fee is rounding error. On a corpus of 50k insurance policies, that is actual cash monthly.

The sequential regime treats the Ok candidates as an ordered checklist and asks the era brick to validate sufficiency at every step:

for i, candidate in enumerate(top_k):
    reply = era(query, [candidate])
    if reply.answer_found and reply.complete_answer_found:
        break

The sufficiency sign lives contained in the typed contract Article 8A launched. The AnswerWithEvidence schema exposes answer_found (was the query’s data current on this candidate?) and complete_answer_found (was your complete reply current, not a fraction?). The loop reads these fields, not a customized heuristic.

Within the efficient date instance above: era runs as soon as on the top-1 candidate, the LLM returns answer_found = True, complete_answer_found = True, the loop exits. Tokens spent: generation_cost(query + 1 chunk of context). That’s 1/5 of the batch price for this query, on a pipeline that handles hundreds of comparable lookups per day.

Batch mode retains the default behaviour: one LLM name sees all Ok candidates and produces one typed reply. Its case is constructed on three query sorts the place sequential breaks down:

• Itemizing questions: “checklist all exclusions on this contract”. The reply is each matching candidate, not the primary. Sequential would cease at top-1 (one exclusion discovered) and miss the opposite 4. Batch is the one appropriate mode.
• Comparability questions: “is the premium increased than the earlier yr’s?”. The reply requires each candidates (this yr + final yr) in the identical name so the LLM can examine them. Sequential would extract each independently and lose the be a part of.
• Tight-score retrieval: when the top-Ok candidates’ relevance scores are inside 5% of one another, retrieval can not reliably promote top-1 to the highest. Batch lets the LLM arbitrate with the total proof seen.

Price evaluation: batch at all times pays the generation_cost(query + Ok chunks of context) as soon as. Sequential pays generation_cost(query + 1 chunk of context) within the simple case (top-1 adequate) and generation_cost(query + 1 chunk) × Ok within the worst case (each candidate inadequate). On a typical enterprise corpus with Ok = 5, sequential is cheaper on common for factual lookups (~80% of typical site visitors) and costlier for itemizing / comparability (~20%).

The clear structure doesn’t decide batch or sequential globally. It picks per query, utilizing the parsed question_df row from brick 2. Query form, decomposition sample, and intent drive the selection:

The dispatcher reads question_df.answer_shape and question_df.decomposition and routes. Naive RAG has no method to make this distinction as a result of it has no parsed query to learn from.

The identical routing desk holds throughout sectors and professions. Totally different domains carry the identical form patterns and the identical sequential / batch choice flows out of them:

In each row, the sequential column is a single typed worth (Quantity, Date, Boolean) and advantages from the top-1 cease. The batch column is a listing or a comparability and wishes all Ok candidates seen directly. The dispatcher’s desk covers all 5 sectors with the identical logic.

Sequential mode leans on one factor: the era brick reporting whether or not the candidate it simply learn was sufficient. That sign, and the foundations that cease the loop, stay right here.

Sequential mode solely works if the era brick can self-report whether or not the candidate it simply noticed contained the reply. The typed contract from Article 8A is what makes this doable:

class AnswerWithEvidence(BaseModel):
    worth: Any
    proof: checklist[Span]
    answer_found: bool
    complete_answer_found: bool
    confidence: float = Subject(ge=0, le=1)
    caveats: checklist[str] = []

The sequential loop reads answer_found and complete_answer_found, not a confidence float or a customized heuristic. The clear separation between the 2 booleans (from Sample 4 of Article 8ter) is what makes the loop deterministic. Discovered + incomplete says “proceed to the following candidate”; discovered + full says “cease”; not discovered says “proceed or quit at Ok”.

A confidence float would power a threshold (e.g. “cease at 0.8”), and that threshold drifts mannequin to mannequin. Two booleans don’t drift.

The sequential loop has three exits, not one:

• Sufficiency: answer_found and complete_answer_found on a candidate. Cease, ship the reply.
• Exhaustion: each one of many Ok candidates seen, none adequate. Cease, return answer_found = False (a first-class reply the validator passes by means of).
• Finances: a token or time finances set by the dispatcher. Helpful when the corpus is massive and a runaway sequential loop would blow the cap. Identical form because the bounded iteration M4 (loop engineering) names.

Naive sequential implementations skip the third exit and burn tokens endlessly on edge circumstances. The collection model units a finances upfront and the dispatcher logs it on each name.

Two regimes, two price profiles, and one boundary the collection doesn’t cross.

To make the trade-off actual, here’s a back-of-envelope for an enterprise insurance coverage Q&A workload:

• 100 questions / day, Ok = 5, common chunk measurement = 600 tokens.
• Batch baseline: 100 × generation_cost(query + 5×600 tokens) = ~330k enter tokens per day for era.
• Sequential (80% top-1 adequate): 80 × generation_cost(query + 600) + 20 × generation_cost(query + 5×600) (worst case for the 20% complicated questions) = ~115k enter tokens per day.

The ratio is 65% saving on enter tokens for era on this workload. The precise ratio is determined by the easy-question proportion and the chunk measurement; the precept is that sequential is a budget default as soon as the typed contract is in place. Batch is reserved for the query sorts that want it.

A pure subsequent step is to let the LLM determine between batch and sequential per query (agentic dispatch). The collection stops in need of this. The dispatcher in Part 2 is deterministic-dispatcher (one of many three approaches catalogued in Article 6C), not LLM-decides. The reason being the identical one which holds throughout the collection: audit. The identical query on the identical day should route the identical means; an LLM that re-plans the dispatch per name can not give that assure.

In case you want a stronger agentic loop (the LLM picks which candidates to have a look at, in what order, with what scope), the larger article on adaptive RAG loops is the best place. This text retains the scope to the deterministic sequential vs batch choice pushed by the parsed query.

The mounted top-Ok + batch pipeline is the best default for the query sorts the place each candidate issues. It’s the incorrect default for the factual lookups that make up most enterprise site visitors. The collection provides two items: the typed sufficiency sign (answer_found, complete_answer_found) from Article 8A, and the dispatch desk from Part 2. Collectively they let the era brick cease after the primary candidate when that’s sufficient, and course of all Ok when the query kind requires it. The choice is made by the parser, not by the LLM, which retains the audit path intact.

The sequential / batch choice sits on the boundary between retrieval and era. The articles that body both sides: