Loop Engineering for Listing Questions: When the Answer Is Every Passage, Not the Top One

Take a simple listing question on the NIST CSF:

“What are all the categories under the GOVERN function?”

The right answer is a list of six categories: Organizational Context (GV.OC), Risk Management Strategy (GV.RM), Roles, Responsibilities, and Authorities (GV.RR), Policy (GV.PO), Oversight (GV.OV), and Cybersecurity Supply Chain Risk Management (GV.SC).

These six items appear:

• In Table 1 on page 20 (compact list of Functions and Categories).
• In Appendix A on pages 21 to 23 (each category gets its own subsection with subcategories below).

A naive RAG pipeline does this:

1. Embeds the question.
2. Retrieves top-5 chunks by similarity.
3. Sends them to the LLM with a generation prompt.

What comes back from retrieval is often the introduction to GOVERN (page 17, the bulleted list of category descriptions) plus a few paragraphs around it. That’s enough for the LLM to list four or five categories, but it might miss GV.SC if the supply chain mentions are clustered in a different chunk that didn’t make the top-k.

The LLM then produces:

“The categories under GOVERN are: Organizational Context, Risk Management Strategy, Roles and Responsibilities, Policy, and Oversight.”

Five out of six. Looks like a complete answer. The user has no way to know that GV.SC is missing unless they cross-check against the document.

The retrieval task has a different shape for listing questions than for factual ones.

For a factual question, top-k retrieval works because the answer is in one passage. You need to find that one passage, and similarity ranking does a reasonable job.

For a listing question, the answer is in N passages, where N is unknown ahead of time. Top-k with a fixed k either retrieves too few (missing items) or too many (diluting the LLM’s attention with irrelevant chunks).

Worse: if the items in the list are similar to each other (which they usually are, since they’re variations on the same theme), embedding similarity gives them similar scores. Setting k high enough to capture all items also captures their near-duplicates, their summaries, their cross-references, and their introductory paragraphs. The LLM ends up with twenty mentions of “GOVERN” and has to figure out which are actual category definitions and which are passing references.

The structural fix: don’t use top-k for listing questions. Use a retrieval strategy designed to find all items rather than a fixed number of top-ranked ones.

The first thing the pipeline needs is to recognize that the question is a listing question, not a factual one. From Article 6, this is an intent: listing value on the parsed question, set either by a small regex pass on the question wording or by a dedicated LLM classifier when the wording is ambiguous.

Detection happens at the question understanding stage (Article 6). A few signals:

LISTING_MARKERS = [
    r"\b(?:what|which)\s+(?:are\s+)?all\s+(?:the\s+)?",
    r"\blist\s+(?:all\s+)?(?:the\s+)?",
    r"\benumerate\s+(?:all\s+)?",
    r"\bgive\s+me\s+(?:all\s+)?(?:the\s+)?",
    r"\bevery\s+",
    r"\bhow\s+many\s+",  # counting questions are also listing
]

def is_listing_question(question: str) -> bool:
    """Heuristic: does this question want a set of items rather than one fact?"""
    return any(re.search(p, question, re.IGNORECASE) for p in LISTING_MARKERS)

This is a starting point. It misses some cases (“What does GOVERN cover?” is implicitly a listing question if GOVERN has multiple categories) and false-positives others (“What is the list price?” contains “list” but isn’t a listing). The proper version uses an LLM classifier in the question understanding stage, returning an intent and an expected cardinality. Article 6 develops that classifier; here we assume the question has already been tagged as listing.

When the question understanding stage tags an intent as listing, the orchestrator (Article 13) activates the listing pattern.

Every function this article walks through ships in docintel.pipeline.listing. Import them once and the rest of the article is the strategies they implement and when each one fires.

from docintel.pipeline.listing import (
    is_listing_question,        # detection (§1.3)
    list_via_structure,         # strategy 1 (§2.1)
    list_via_pattern,           # strategy 2 (§2.2)
    list_via_semantic,          # strategy 3 (§2.3)
    deduplicate_items,          # merge (§3.1)
    detect_cardinality_cue,     # completeness signal (§3.2)
    assess_completeness,        # completeness verdict (§3.2)
)

The strongest signal for listing on enterprise documents is structure. Section headings, bullet lists, numbered enumerations, table rows, subcategory codes: these are the author’s own enumeration.

For the NIST CSF question “What are all the categories under GOVERN?”, the structure does the work:

def list_via_structure(
    *,
    section_hint: str | None,
    line_df: pd.DataFrame,
    toc_df: pd.DataFrame | None,
    page_range: tuple[int, int] | None = None,
) -> list[dict]:
    """Listing via the document's own structural markers."""
    # Strategy A: TOC children.
    if section_hint is not None and toc_df is not None and not toc_df.empty:
        if "parent_id" in toc_df.columns:
            children = toc_df[toc_df["parent_id"] == section_hint]
            if not children.empty:
                return children.to_dict(orient="records")

    # Strategy B: enumeration regex on a region.
    if page_range is not None:
        lo, hi = page_range
        region = line_df[(line_df["page_num"] >= lo) & (line_df["page_num"] <= hi)]
    else:
        region = line_df
    items = region[region["text"].astype(str).str.match(_ENUM_LINE_PATTERN)]
    return items.to_dict(orient="records")

When the items don’t follow a strict structural pattern but do follow a recognizable shape, you can aggregate them by pattern matching across the document.

Take the question “What are all the subcategory codes under GOVERN?” on the NIST CSF. The codes follow a strict pattern: GV.XX-NN where XX is a category code (OC, RM, RR, PO, OV, SC) and NN is a sequence number.

def list_via_pattern(line_df: pd.DataFrame, pattern: str) -> list[dict]:
    """Find every match of `pattern` across `line_df`, deduplicated by match."""
    rx = re.compile(pattern)
    matches: list[dict] = []
    seen: set[str] = set()
    for _, row in line_df.iterrows():
        text = str(row.get("text", ""))
        if not text:
            continue
        for m in rx.finditer(text):
            code = m.group(0)
            if code in seen:
                continue
            seen.add(code)
            matches.append({
                "code": code,
                "page_num": int(row.get("page_num", 0)),
                "line_num": int(row.get("line_num", 0)),
                "context": text,
            })
    return matches

Run it with the GV pattern:

31 subcategories under GOVERN, all retrieved deterministically with one regex pass over the document. No embedding, no top-k, no LLM call until the very end. The retrieval is exhaustive by construction.

Pattern-based aggregation works whenever the items have a regular shape: codes, identifiers, numbered clauses, normalized references, formula identifiers. Many enterprise documents have these.

When the items don’t follow a structural marker or a fixed pattern (free-form prose lists), the pipeline needs an LLM to identify them. But the standard top-k generation shape is wrong. It returns what’s in the top-k chunks, not what’s in the document.

The fix: a two-pass approach.

Pass 1: discovery: The pipeline does a broad retrieval (more permissive than top-k, often section-level via TOC). It sends the broad context to the LLM and asks for the items found, plus a completeness self-assessment.

class ListingResult(BaseModel):
    items: list[ListingItem]
    is_likely_complete: bool
    reason_if_incomplete: str | None = None
    suggested_additional_keywords: list[str] = Field(default_factory=list)

Pass 2: refinement: If is_likely_complete=False, the pipeline uses the LLM-suggested keywords to expand retrieval, fetches additional regions, and re-runs the listing. Multiple passes until completeness or until a max-iterations bound.

def list_via_semantic(
    *,
    question: str,
    line_df: pd.DataFrame,
    toc_df: pd.DataFrame | None,
    initial_keywords: list[str],
    llm_extract: LLMExtractFn,
    broad_retrieve: Callable | None = None,
    max_iterations: int = 3,
) -> list[ListingItem]:
    """Broad retrieval + LLM extraction in a bounded loop."""
    if broad_retrieve is None:
        broad_retrieve = _default_broad_retrieve
    accumulated: list[ListingItem] = []
    seen_ids: set[str] = set()
    keywords = list(initial_keywords)
    for _ in range(max_iterations):
        candidates = broad_retrieve(line_df, toc_df, keywords)
        new = [c for c in candidates if c.get("id") not in seen_ids]
        if not new:
            break
        seen_ids.update(c.get("id") for c in candidates if c.get("id") is not None)
        result = llm_extract(question, new, accumulated)
        accumulated = _merge_items(accumulated, result.items)
        if result.is_likely_complete:
            break
        keywords = list(set(keywords + result.suggested_additional_keywords))
    return accumulated

This is a feedback loop specific to listing, and its three control surfaces are all visible in the code. The trigger is is_likely_complete=False (or a cardinality mismatch, section 3.2). The termination is threefold: no new passages, a completeness verdict, or max_iterations. The recovery changes one thing before the retry: the keyword set, expanded with the LLM’s suggestions. It is a small loop in the sense of Article 10’s cascade: it runs entirely inside the listing branch, iterating on its own material, and never sends the pipeline back to another brick. Article 13 develops the general iteration mechanics that bound this loop.

For the Transformer paper question “What are all the regularization techniques used?”, the structural strategy already finds them in Section 5.4. But for a question like “What are all the techniques used to improve translation quality?”, the items are scattered: dropout (Section 5.4), label smoothing (Section 5.4), beam search (Section 6.1), checkpoint averaging (Section 6.1), warmup schedule (Section 5.3). Semantic aggregation with multiple passes is the right strategy.

Finding candidate items is half the job. The other half is turning them into an answer the user can trust: merged duplicates, an explicit completeness verdict, and a presentation that shows both.

Whichever strategy you use, you’ll often retrieve the same item multiple times. The same subcategory mentioned in the introduction and in the appendix. The same regularization technique mentioned in the abstract and in Section 5.4. Listing pipelines need to deduplicate.

Deduplication has two levels:

Surface deduplication: Same string, different occurrences. Easy: hash the normalized text.

Semantic deduplication: Different surface forms of the same item. Harder: “Residual Dropout” and “dropout applied to residual connections” are the same technique. “GV.SC” and “Cybersecurity Supply Chain Risk Management” are the same category.

For semantic deduplication, the LLM is the right tool. Send the candidate list of items, ask the LLM to merge synonyms:

def deduplicate_items(
    items: list[ListingItem], llm_dedupe: DedupeFn | None = None,
) -> list[ListingItem]:
    """Surface-form deduplication, with optional LLM semantic merge."""
    # Surface dedup first (exact-string, case-insensitive on canonical_name).
    by_lower: dict[str, ListingItem] = {}
    for it in items:
        k = it.canonical_name.strip().lower()
        if k in by_lower:
            cur = by_lower[k]
            cur.surface_forms = sorted(set(cur.surface_forms + it.surface_forms))
            cur.citations.extend(it.citations)
        else:
            by_lower[k] = it
    surface_unique = list(by_lower.values())
    if llm_dedupe is None or len(surface_unique) <= 1:
        return surface_unique
    return llm_dedupe(surface_unique)

The output is a list where each entry has:

• A canonical name.
• All the surface forms encountered.
• All the page/line citations.

This is what the user wants: a clean enumeration of distinct items, each one fully cited.

The pipeline also needs an explicit signal that the list is complete; nothing in the candidate set says “you have all of them” on its own.

Three sources of completeness:

Source 1: structural completeness: When the items come from a known structural pattern (TOC children, regex matches), completeness is guaranteed by construction. If the document has 31 subcategory codes matching GV\.[A-Z]{2}-\d{2}, the regex finds all 31. There’s nothing to miss.

Source 2: explicit cardinality cues: Sometimes the document tells you how many items there are. “The framework has six Functions”. “The encoder consists of N=6 layers”. “Three types of regularization are used”. The pipeline detects these cues and validates: if the document says six and we found five, something is missing.

If the document explicitly says “six categories under GOVERN” and the listing returns five, the pipeline knows to iterate.

Source 3: LLM self-assessment: When neither structural nor cardinality cues apply, the LLM’s is_likely_complete field is the only signal. It’s a weak signal (LLMs can be wrong about completeness), but it’s better than no signal. Combined with bounded iteration (max 3 passes), it gives reasonable behavior.

The final completeness verdict is the strongest of the three:

def assess_completeness(
    items: list,
    items_seen_last_iteration: list | None,
    document_text: str,
    llm_signal: bool | None,
) -> tuple[bool, Literal["structural", "cardinality", "llm_assessment"]]:
    """Combine the three sources of completeness with a strict precedence."""
    cardinality = detect_cardinality_cue(document_text or "")
    if cardinality is None:
        cardinality = count_explicit_enumeration(document_text or "")
    if cardinality is not None:
        return (len(items) >= cardinality, "cardinality")
    if items_seen_last_iteration is not None and len(items) == len(
        items_seen_last_iteration
    ):
        return (True, "structural")
    if llm_signal is not None:
        return (bool(llm_signal), "llm_assessment")
    return (False, "llm_assessment")

A listing answer needs to be presented differently from a factual answer. The user needs to see:

• The full enumerated list, with each item cited.
• A completeness statement (“this is the complete list” or “this list may be incomplete because…”).
• Optionally, the surface forms encountered for each item.

A schema for listing output:

class ListingItem(BaseModel):
    canonical_name: str
    surface_forms: list[str] = Field(default_factory=list)
    citations: list[Citation] = Field(default_factory=list)
    extraction_confidence: float = 1.0

class ListingAnswer(BaseModel):
    items: list[ListingItem]
    is_complete: bool
    completeness_source: Literal["structural", "cardinality", "llm_assessment"]
    not_found_items: list[str] = Field(default_factory=list)
    notes: str | None = None

For our NIST question, the answer might look like:

The user can immediately see the full list, the citations, and the completeness verification. If a category were missing, the cardinality check would have flagged it during retrieval.

Two runs, one per document, chosen to show the two completeness outcomes: a run where the cardinality check confirms the list on the first pass, and a run where it catches a missing item and the loop fires.

Let’s run a slightly different question to show the cardinality verification:

“What are all the Functions of the NIST Cybersecurity Framework?”

Question understanding. Intent classified as listing. Anchor keywords: ["Function", "GOVERN", "IDENTIFY", "PROTECT", "DETECT", "RESPOND", "RECOVER"].

Strategy selection. The orchestrator (Article 13) selects the structural strategy first. Functions are top-level entries in the TOC.

The CSF Functions don’t appear as top-level TOC entries; they appear as subsections within Appendix A. The orchestrator falls back to pattern-based aggregation.

Pattern-based retrieval. The pipeline scans for the pattern \b(GOVERN|IDENTIFY|PROTECT|DETECT|RESPOND|RECOVER)\s*\([A-Z]{2}\):

Six matches, deduplicated.

Cardinality check. The pipeline scans the document for cardinality cues:

page8_text = "\n".join(line_df.loc[line_df["page_num"] == 8, "text"])
cue = detect_cardinality_cue(page8_text)
# -> 6   (matched: "The Framework Core consists of six Functions:")

The document explicitly lists six on page 8 (Section 2, “Introduction to the CSF Core”). The retrieval returned six. Cardinality verified.

Generation with completeness verdict.

The user gets a clean, complete, cited list with verification that nothing is missing.

A free-form listing question:

“What are all the regularization techniques used to train the Transformer?”

Question understanding. Intent: listing. Anchor keywords: ["regularization", "dropout", "label smoothing", "training"].

Strategy selection. No clear pattern (regularization techniques don’t have a fixed code). Fall back to structural strategy with TOC: find the section titled “Regularization”.

Section retrieval. toc_df has Section 5.4 “Regularization” on page 7. The pipeline retrieves it.

Item extraction. Within Section 5.4, the document uses bold headers as enumeration markers (page 8):

• Residual Dropout, applied to sub-layer outputs and to embeddings + positional encodings sums. Pdrop = 0.1.
• Label Smoothing, value εls = 0.1.

Two items found.

Cardinality check. The first sentence of Section 5.4 says “We employ three types of regularization during training”. The retrieval found two. Cardinality says three. Mismatch, the pipeline iterates.

Iteration. The pipeline reads the LLM’s suggested_additional_keywords. The LLM, seeing two items but knowing three are mentioned, suggests ["attention dropout", "checkpoint averaging"].

Re-running retrieval with expanded keywords finds a third item earlier in the paper: “attention dropout”. Looking back at Section 5.4 carefully, the third regularization is mentioned but as a follow-up to Residual Dropout; the same paragraph says dropout is also applied to attention weights. On closer reading the third type is implicit in the dropout discussion. The LLM extracts it on the second pass.

This iteration mechanic is listing-specific: cardinality mismatch as the trigger, keyword expansion as the recovery (the one thing that changes before the retry), completeness check as the termination. The general iteration mechanism (bounding, drift detection, anti-patterns, audit trail) is what Article 13 develops.

Final answer.

The mismatch between expected count (three) and initial extraction (two) triggered iteration. Without the cardinality check, the pipeline would have returned a confident two-item answer that was missing a third.

Listing questions need three things naive top-k cannot give: detection up front so the pipeline picks the listing branch, a strategy that fits the document’s structure (TOC for sections, regex for markers, semantic iteration for scattered prose), and a completeness signal driven by a cardinality cue when the document declares one. None of the three strategies is universal; the dispatcher (Article 13) picks the right one per question.

The pattern ships as the library’s listing module and composes with TOC retrieval (Article 9), cross-references (Article 11), and the general iteration machinery Article 13 names.