Loop Engineering with Adaptive Parsing in Action: Parsing Flat Tables with Azure and Figures with a Vision LLM

Generation is the last stop in the cascade. Its two checks are check 7 (the LLM’s self-evaluation flag, baked into the typed answer) and check 8 (a separate-LLM or NLI groundedness check after the answer is produced). Check 7 is the most expensive AND least stable check in the panorama, so this part spends real time on it: two walkthroughs that show it firing usefully (Case A in section 2, Case B in section 3), then a stress test that says the LLM signal alone is not enough (section 4). Check 8 is section 5.

Before the two walkthroughs: most questions do not need any escalation at all. Article 9 ran the standard pipeline on the same Attention paper with the question “What are the options for positional encoding?” and got a clean answer at the first generation pass, context_structured=True, no re-parse needed. PyMuPDF alone was enough because the answer lived in prose. Same for “What are the six CSF Functions?” on the NIST document. The two walkthroughs below are the harder cases, where PyMuPDF could not see what the question needed.

Two questions on the Attention paper, two failure shapes, two parser escalations. The walkthroughs mirror each other on purpose: same data model, different parser at the deep step.

The question: “What is the value of h for the base model in Table of Variations on the Transformer architecture?” The correct answer is 8, readable from the h column of the base row of Table 3 on page 9 of the Attention paper. The cell is filled (not blank) so the answer itself is unambiguous; what stresses the pipeline is the flat-parsed shape of the surrounding grid, which is exactly what the check-2 fingerprint already flagged for free in the first part and what the LLM signal at check 7 also flags below.

Article 6 builds a question-parsing brick (parse_question) that handles typos, multi-part questions, language detection, and expert-vocabulary expansion. On a question this short and clean, the brick is overkill. We skip it and feed the question text directly to retrieval and generation. The pipeline does not break; we just choose not to call one of the four bricks because the input is already in the shape downstream wants.

The keywords we pick by eye for the retrieval call below: just the table-caption string, “Table 3: Variations on the Transformer architecture”. One specific phrase is enough on this corpus.

The standard pipeline starts cheap with PyMuPDF. One function, one line:

line_df = fitz_pdf_to_line_df("data/paper/1706.03762v7.pdf")
page_df = build_page_df(line_df)

# line_df.shape ~ (1500, 8)
# columns : page_num, line_num, text, x0, y0, x1, y1, character_count

The first eight lines of the captured line_df (page 1, before we ever look at page 9):

Article 7 covers four retrieval methods: keyword matching, embedding similarity, TOC matching, and an LLM-orchestrator on top. For this question, keyword retrieval is sufficient because the question mentions a specific string (“Table 3”) that the table caption echoes verbatim on page 9. We pass that exact string as the only keyword:

from docintel.retrieval import retrieve_pages

retrieved_pages_df, filtered_line_df = retrieve_pages(
    page_df, line_df,
    keywords=["Table 3: Variations on the Transformer architecture"],
    top_k=5,
)

The captured retrieved_pages_df:

The filtered_line_df returned alongside retrieved_pages_df carries every line of page 9, ready for the generation brick.

The generation brick (llm_answer_with_evidence, Article 8) takes the filtered lines and the question, calls gpt-4.1 with a Pydantic schema as the output contract. The AnswerWithEvidence schema already carries context_structured (added in Article 8), so the orchestrator has a one-bit trigger to read on every answer:

from docintel.generation import llm_answer_with_evidence
from docintel.generation.qa.schemas import AnswerWithEvidence

pass1: AnswerWithEvidence = llm_answer_with_evidence(
    question=("What is the value of h for the base model "
              "in Table of Variations on the Transformer architecture?"),
    filtered_line_df=fitz_p9,    # PyMuPDF lines for page 9
)
# AnswerWithEvidence carries :
#   answer, start/end_page_num, start/end_line_num, confidence,
#   justification, caveats,
#   context_structured  <-- the binary trigger for the cascade
#   complete_answer_found, llm_discovered_keywords

The captured response, copy-pasted from the real run:

{
  "answer": "8",
  "start_page_num": 9,
  "start_line_num": 25,
  "end_page_num": 9,
  "end_line_num": 25,
  "confidence": 0.99,
  "justification": "In the table listing parameters for different Transformer ...",
  "caveats": [],
  "complete_answer_found": true,
  "context_structured": false,
  "llm_discovered_keywords": []
}

The answer is "8", the correct value. The citation points to PyMuPDF L25 of page 9. Look back at the parse: L25 IS the h cell of the base row (base is at L21, then values L22=6 (N), L23=512 (dmodel), L24=2048 (dff), L25=8 (h)). The LLM got both the right value AND the right cell.

But context_structured=False. The LLM flags the structural risk. PyMuPDF returned the 13 base-row cells as 13 separate lines L22-L33, the column headers on lines L5-L20 (some single-line, some split across two lines for compound headers like “train steps”). To map L25 to “h” the LLM had to count column positions in a flattened header. It got it right this time, but it warns: the structure is fragile, do not trust the next similar question without checking.

The binary flag is the trigger. When False, the orchestrator escalates regardless of how confident the answer was.

context_structured=False. The orchestrator picks a structure-aware parser. The project has Azure DI wired up, but Camelot, Docling, or any other table-aware parser would slot in identically:

from docintel.parsing.pdf.azure_layout import azure_layout_pdf_to_line_df

azure_p9 = azure_layout_pdf_to_line_df(
    "data/paper/1706.03762v7.pdf",
    pages=[9],     # only the page retrieval pointed at
)
# azure_p9.shape ~ (52, 9)
# columns : page_num, line_num, text, x0, y0, x1, y1, character_count, parsing_method
# every row carries parsing_method == "azure_layout"

The first eight Azure rows on page 9:

The text column holds Azure’s markdown, one line per table row. Decoded back into a grid, that is Table 3 as Azure recovered it, the structure PyMuPDF had flattened:

Same call as Pass 1, with the Azure rows instead of the PyMuPDF rows. The captured response:

{
  "answer": "8",
  "start_page_num": 9,
  "start_line_num": 7,
  "end_page_num": 9,
  "end_line_num": 7,
  "confidence": 0.98,
  "justification": "In Table 3 (line 7), the 'base' model lists its hyperpara...",
  "caveats": [],
  "context_structured": true
}

Same value, this time the LLM trusts the structure. Reading the Azure row, the model does not need to count columns: h is the 5th cell of the markdown line, the value 8 is right there, the column header h is the 5th cell of L5 above. Citation is the whole row instead of one cell, but the cell-to-column mapping is unambiguous because the pipes anchor everything.

The feedback loop fired and resolved. Cost: one extra Azure DI call (about $0.003 and four seconds). Gain: the answer is now structurally trusted, and line_df carries a permanently structured page-9 record for any future question on this page.

The annotation brick converts the evidence span into a rectangle drawn on the source PDF. We use the Pass 1 citation (PyMuPDF L25, the h cell of the base row) because PyMuPDF returns per-cell lines with tight bboxes; the Azure row would give a row-wide box covering all 13 cells:

from docintel.rendering.pdf.annotation import (
    passage_lines_df_from_answer,
    passage_bbox_by_page,
    draw_passage_rectangles,
)

passage_df = passage_lines_df_from_answer(line_df, pass1)
bboxes_df  = passage_bbox_by_page(passage_df)

draw_passage_rectangles(
    pdf_path="data/paper/1706.03762v7.pdf",
    bboxes_df=bboxes_df,
    out_pdf_path="output/parsing_comparison/h_value_annotated.pdf",
)

The captured passage_df is one row:

And bboxes_df, the same row collapsed per page:

The rendered annotated PDF page (top half of page 9, cropped to Table 3):

End-to-end auditability: the question travelled from text input through four bricks and back to a rectangle on the source PDF. A reviewer can scan the highlighted page and verify in one second that the cited cell matches the LLM’s claim. Even though the feedback loop fired (Pass 1 cs=False -> escalate), the loop did not break the audit trail; it added an Azure-parsed row for the same page, kept the PyMuPDF rows alongside, and the annotation used the PyMuPDF bbox because that is what produced the tightest box.

The walkthrough above used gpt-4.1. To see how parser quality interacts with model strength on this exact question, we ran the same call across three models (gpt-4o-mini, gpt-4o, gpt-4.1) and both parsers (PyMuPDF, Azure DI). The grid below should return "8" in every cell; the interesting variation is the context_structured flag:

Two findings.

First, the answer is the same everywhere. This question is unambiguous on the source: the value 8 sits in a normal filled cell, the LLM picks it up reliably regardless of parser or model.

Second, the context_structured flag varies in a useful way. gpt-4o-mini and gpt-4o accept PyMuPDF’s flat lines without flagging. gpt-4.1 conservatively flags the structural risk, even though it got the answer right. The pipeline’s reliability depends on which model populates the flag: a stronger model is a more honest reporter of upstream parsing risk. With Azure DI, every model trusts the structure; the parser closes the model-strength gap on the flag side.

The takeaway: upgrading the parser is the durable fix, because it removes the structural risk altogether. Upgrading the model raises the chance of detecting parsing risk, but does not remove it. Production pipelines that mix both knobs hedge against either weakness.

The question: “What is the architecture of the Transformer?” The answer is mostly carried by Figure 1 on page 3 (the encoder-decoder architecture diagram). The prose around it describes pieces of the architecture but does not replace the diagram.

Pass 1, PyMuPDF only: Retrieval finds page 3. PyMuPDF returns the surrounding prose plus a type='image' row for Figure 1, with a placeholder text and no actual content. line_df around the figure looks roughly like:

Generation, pass 1: The LLM reads the prose, notices the placeholder and the gap, and produces a partial answer with a structural caveat:

{
  "answer": "The Transformer uses a stack of 6 encoder layers and 6 decoder l...",
  "start_page_num": 3,
  "start_line_num": 12,
  "end_page_num": 3,
  "end_line_num": 14,
  "confidence": 0.55,
  "caveats": [
    "Figure 1 is referenced in the prose but its content is absent fr..."
  ],
  "complete_answer_found": false,
  "context_structured": true
}

Pipeline calls vision LLM on the image. Azure DI would not help here: the diagram is not a table, it is free-form vector geometry. The pipeline reads the caveat “figure referenced but absent”, looks up image_df.image_id=1, sees parsing_method='not_parsed', and routes to a vision-language model:

new_lines = vision_extract(
    "data/paper/1706.03762v7.pdf",
    pages=[3],
    image_ids=[1],
)
# new_lines.parsing_method == "vision_gpt4o"
# image_df : row image_id=1 updated from parsing_method='not_parsed' to 'vision_gpt4o'

The vision model returns a structured description: encoder input embedding plus positional encoding, six identical layers each carrying multi-head self-attention with residual connection and layer norm, then position-wise feed-forward with residual and norm; decoder mirror with masked self-attention plus cross-attention to encoder output; final linear and softmax to vocabulary. This description gets appended to line_df as new text rows with parsing_method='vision_gpt4o'.

Generation, pass 2: The LLM now has both the prose description and the structured visual description. It produces a complete answer with high confidence, naming the residual connections, the layer norm placement, the cross-attention bridge: details the prose alone did not enumerate.

{
  "answer": "Encoder-decoder. Encoder is 6 identical layers ; each layer is m...",
  "start_page_num": 3,
  "start_line_num": 30,
  "end_page_num": 3,
  "end_line_num": 42,
  "confidence": 0.92,
  "complete_answer_found": true,
  "context_structured": true
}

The full cost: 5 seconds PyMuPDF, about 10 to 30 seconds and a few cents for the vision call on one image. The other 14 pages of the paper never see a vision model.

The walkthroughs in sections 2 and 3 show the LLM signal firing usefully on two real questions. That is the best case. To see what the signal looks like outside the best case, we stress-tested it across questions of growing structural difficulty, three models, and both parsers. The picture is less reassuring than a single happy-path walkthrough suggests, and it justifies the cascade order: let the cheap deterministic checks at checks 1 and 2 carry the load; keep the LLM signal as the last line of defence, not the first.

We ran three questions of growing structural difficulty against Table 3 of the Attention paper, on three models (gpt-4o-mini, gpt-4o, gpt-4.1) and two parsers (PyMuPDF, Azure DI). Eighteen runs in total.

The questions:

• Q_easy: the value of h for the base row. Answer is 8, sitting in a directly-labelled cell.
• Q_inheritance: the value of h for the rows labelled (B). Answer is 8 again, inherited from base because the h cell is blank for (B). The flat parse loses the column anchor.
• Q_far: the perplexity of the big model on the dev set. Answer is 4.33, last row of a long table with many similar-looking numbers around.

The starting framing was: a weaker model that cannot answer can still flag the broken parse as a structural canary. The data says the opposite happens. When a weak model gives the wrong answer on Q_inheritance, it does not flag. It fabricates and asserts context_structured=True.

Out of four wrong answers across the matrix, four had cs=True and zero had cs=False. Three different model-parser pairs on Q_inheritance all picked a number from an adjacent column (dk = 16 or 32) and asserted the structure was clean. On Q_far, gpt-4o-mini on PyMuPDF returned 25.7 (the BLEU column, not PPL) with cs=True. The model that mis-aligned columns also failed to notice it had mis-aligned them.

The single cs=False in the matrix came from gpt-4.1 on PyMuPDF for Q_inheritance. Its answer was 8, correct. Read at face value: the only time the LLM signal fired, it was a model that did not need to be rescued.

The natural next move is to replace the binary flag with a 0.0-1.0 score, expecting the LLM to use the middle of the range when uncertain. We re-ran the matrix with a context_structured_score field added.

Scores clumped between 0.85 and 1.00 regardless of correctness. Mean score on correct answers: 0.919. Mean score on wrong answers: 0.920. Identical. At threshold 0.95, the score catches all three wrong answers, but raises nine false alarms on the fifteen correct ones, a 60% false-alarm rate. No usable threshold.

The LLM’s structural verdict is bimodal disguised as continuous. The score buys nothing beyond the binary flag.

Asking gpt-4.1 for a one-sentence rationale alongside the flag changes the picture. The rationale text is concrete and consistent across runs: “headers split across multiple lines”, “each cell on its own line, not aligned”, “blank cells indicate inherited values”. The model accurately describes the parse shape every time.

The verdict on top of the rationale, on the other hand, is unstable. We ran gpt-4.1 three times on Q_inheritance with PyMuPDF, same input, same temperature settings:

The model knows what is wrong with the parse. It does not know reliably whether that is enough to escalate. A feedback loop that reads the rationale and applies a deterministic rule on top of it (“contains ‘split across multiple lines’ or ‘each cell on its own line’ or ‘blank cells’” then escalate) becomes stable. Reading the binary or the score directly does not.

After the answer is produced, a second LLM (or an NLI model) verifies that every claim in the answer is supported by the cited chunks. This catches the fabrications the generating model is too confident to flag itself in check 7. Cost: one extra LLM call. Reliability: higher than the same model self-evaluating its own answer, because the judge is fresh and has no incentive to defend the generation. Cheaper and more reliable than the check-7 LLM-as-judge approach for parsing quality, and the only check in generation that has a chance of catching fabrications check 7 missed.

The cascade as a whole says: checks 1 and 2 fire on every page in milliseconds (document parsing); check 4 fires on the question (question parsing); checks 5 and 6 fire after retrieval (retrieval); check 7 only fires if the upstream checks gave the page a clean bill of health (this section); check 8 fires after generation as a final safety net (this section). The walkthroughs above cover what check 7 looks like when it is the last line of defence; the panorama says that should not be the only line of defence.

Two practical refinements before closing: when the lazy default is wrong, and how the data model gives caching for free.

Lazy parsing is the right default. It is not the right call everywhere:

• Permanent, heavily-queried documents. A standard insurance policy template is queried thousands of times across a portfolio’s lifetime. Parse it once with Docling or Azure at ingestion, store the result, amortize over all future queries.
• Audit-grade documents: Due diligence, regulatory filings, litigation. When you need a defensible record that no information was missed, full parsing upfront is the baseline. The cost is acceptable because the alternative (missing a relevant clause) is far worse.
• Small documents: Under 20 pages, the marginal cost of running Docling or Azure on the whole thing is small. The 15-page Attention paper might fall in this bucket: the cost-benefit shifts.
• Pre-computed search indexes: Full-text retrieval over a corpus parses once at ingestion. Lazy parsing applies within a document at query time, not at corpus ingestion time.

The right question to ask: how many times will this document be queried, on average, across its lifetime? Many times, go eager. A few times or unknown, go lazy. For a typical enterprise corpus where most documents are queried zero or one times, lazy parsing wins by a wide margin.

When the pipeline re-parses page 6 of the Attention paper with Azure, the result lands in line_df with parsing_method='azure_layout' and in page_df with the corresponding audit row. The next question that touches page 6 hits these rows directly. No re-parsing needed. The cache is not a separate layer; it is the same line_df and page_df that retrieval already reads.

def get_or_enrich(pdf_path, line_df, page_df, pages, method):
    """Re-use a previous enrichment if one already ran with the same method."""
    done = page_df[
        (page_df.page_num.isin(pages)) &
        (page_df.parsing_method == method)
    ]
    missing = [p for p in pages if p not in done.page_num.values]
    if not missing:
        return line_df, page_df                         # already enriched
    return enrich(pdf_path, line_df, page_df, missing, method)

Over time, the cache fills with the enrichments that questions triggered. The most-queried regions of the most-queried documents get the deep-parsed version automatically. The rarely-touched regions stay on the cheap parse. The system learns where to invest deep parsing based on actual usage, not on a guess made at ingestion time.

Parsing quality is a multi-check decision spread across the four bricks, ordered cheap-first: deterministic signals during parsing (char density, flat-table fingerprint, chunk integrity), then intent-aware routing during question parsing, then score-gap and context-gap checks during retrieval, then LLM self-eval as the last line of defence. The stress test in section 4 makes the ordering non-negotiable: the LLM’s binary verdict on its own context is not stable enough to lead the cascade, and a separate fresh judge is what catches the fabrications.

The data model carries the cascade: parsing_method as a column on every parsing table makes mixed parses, escalation, audit, and caching reduce to filtering on a column. The same skeleton supports any new failure shape: adding a new shape adds a row to the routing map, adding a new check adds a column. Article 11 keeps the same self-correction frame and applies it to cross-references.