Loop Engineering for Hierarchical Retrieval: Reading a Long Document by Its Table of Contents

NIST SP 800-53 is 492 pages. It defines the security controls a US federal system has to meet, one control at a time, twenty families of them. A user asks a simple question:

“What does the account management control require?”

The answer is five pages, control AC-2, on pages 46 to 50. The naive RAG move embeds every page and takes the top-k most similar to the question. It fails in a specific way here: the words account, management, control, and access sit on hundreds of pages, because the whole document is about controls. Top-k returns AC-2 mixed with AC-3, AC-17, a couple of audit controls, and the glossary, and the generation model has to guess which one was meant. The bill is paid twice: you embed 492 pages, and the answer is still blurred.

A person opens the table of contents instead. Here is the catch that makes this article necessary: the table of contents is itself 358 entries. You would not hand a colleague all 358 lines any more than the 492 pages. An expert scans the chapter list first, eleven titles, picks The Controls, opens it to the twenty families, picks Access Control, opens that to its controls, and lands on AC-2. Top level first, then down, one small decision at a time. That is the loop this article builds, and it is the series thesis in miniature: amplify the expert. Do what the expert does. Don’t dump the whole document, or the whole table of contents, on the model at once.

Here is the actual contents page of the document, next to the toc_df the parser reads from its native outline.

Retrieval walks the tree top-down. It hands the LLM only the current level, the eleven chapter titles first, each as one compact line: a title, a page range, and when titles are ambiguous a short keyword tally. The model picks the branch that answers the question. If that branch is broad and has finer children, retrieval opens it, its children become the next level, and the step repeats. It stops at a leaf, or at a section small enough to read whole.

Two things fall out of this shape. The long table of contents never enters a prompt whole: the model reads eleven lines, then twenty, then twenty-odd, never 358 at once. And the descent is effectively optional. On a short document the top level has no children to open, so the loop runs once and behaves exactly like flat routing. The model judges at every level whether there is any reason to go deeper.

The step is one function, reason_on_toc, called once per level. It reads the level as text, never a table, so a level of twenty entries is twenty short lines, not a grid that grows a column per keyword.

# Top-down: feed ONE level at a time, never the whole 358-row TOC.
# Each entry is one compact line: title + page range (+ keyword hits).
level = toc_df[toc_df.level == toc_df.level.min()]      # 11 chapter titles

while True:
    pick    = reason_on_toc(question, level, client=client)   # one LLM call
    section = level[level.id.isin(pick.section_ids)]
    kids    = immediate_children(toc_df, section)
    if kids.empty or section.n_pages <= SMALL:               # leaf, or small enough
        break
    level = kids                                             # open it, descend

# 11 chapters -> 20 families -> 25 controls -> AC-2 ACCOUNT MANAGEMENT (pp. 46-50)

Run it on the account-management question against the real document, and it descends one level at a time. At each level the model reads that level’s titles, reasons, and picks the branch to open. The reasoning below is the model’s own, verbatim from the run.

Fifty-six entries read across three small calls, never the 358-row table of contents. Generation then reads five pages, and the other 315 controls never enter the prompt.

When the title alone is not enough to separate two candidates, retrieval leans on that keyword tally, still one line per entry: how many of the question’s keywords fall inside that branch. It is the tie-breaker for the case where a term is used in one section and only defined in another (least privilege is applied throughout Access Control, but defined once in the Glossary). It never becomes a column per keyword.

The loop stops descending in three cases: the picked branch is a leaf, it is already small enough to read whole, or the question is a listing that needs every item under it. The listing case is the NIST CSF from Article 12 (listing): asked for every subcategory of GOVERN, the router picks the whole Appendix A. CSF Core, and because a listing wants all of it, retrieval reads the appendix whole rather than descending into one subcategory.

Here is that second descent, on the same routing step but a listing question. It reaches the section holding the list in a single level and stops, because nothing finer sits under it in the table of contents.

That gives the loop the three control surfaces from Article 13bis (loop engineering): a trigger (a broad branch with children), a termination (a leaf, a small section, or a listing), and recovery that changes something every iteration (it descends a level, never re-reads the one it just judged), with depth bounded by the tree so it cannot spin. It differs from the pipeline loops of Part III in what it reacts to: the document’s structure, not a generation result.

Hierarchical routing wins on both axes at once, which is rare.

Precision. Flat top-k over 492 pages competes AC-2 against every other control that mentions account or access. Routing commits to AC-2 Account Management by name and reads it whole, so the answer’s own five pages arrive together instead of interleaved with five neighbours.

Tokens. The naive pipeline embeds 492 pages once and pays retrieval on all of them at every query. The top-down router never embeds the body. It reads fifty-six short title lines across three calls, then the five pages the answer lives on. Even the table of contents is never sent whole. On a corpus of thousands of such documents, that difference is the line between a system that runs and one that does not.

This article stays on a single document. A folder of many documents is the same map one level higher: the top level is the list of files, each with a title and a one-line summary, and the next level is each file’s own table of contents. The routing step does not change: you pick the relevant files from the top level, then descend into their sections exactly as this article descends into controls. The move is the same, only the number of levels grows. We cover that collection case in detail in Part IV, starting with Article 14 (the corpus problem) and the corpus tables (corpus_toc_df, corpus_index) that carry it.

A long document ships with its own map, and retrieval walks it top-down instead of scoring every page. The router reads one level of the table of contents at a time, one compact line per entry, and descends: eleven chapters to The Controls, twenty families to Access Control, twenty-odd controls to AC-2 Account Management, five pages out of 492, with the other 315 controls never in the prompt. It stops at a leaf, at a section small enough to read whole, or on a listing that wants everything under a heading. That is a bounded loop inside retrieval, optional when the table of contents is shallow, and it saves tokens and lifts precision at the same time, which is why it scales up unchanged: a folder of documents is the same map with one more level.