One RAG Pipeline, Four Very Different PDFs: Same Four Bricks, Every Answer Typed and Cited

Article 1 (minimal RAG)’s diagram showed four boxes in a row with one input pill per gap. Enough to introduce the bricks, not enough to capture what the upgraded pipeline actually carries. The version below adds the two side channels that the production code thr

Read the diagram in three passes.

The four bricks on the main row. Same names as Article 1 (minimal RAG). Each brick consumes the typed object the previous one emitted (line_df out of parsing, ParsedQuestion out of question parsing, anchor + context out of retrieval) and the contract is the Pydantic schema, not a plain dict. The four bricks stay independent: none of them imports any other.

The first side channel: parsing_summary feeding question parsing. Question parsing in Article 1 (minimal RAG) saw only the raw text of the question. In the upgraded pipeline it also receives a compact projection of the doc-level synthesis the parsing brick computed (doc_type, n_pages, typical_fields, summary). The same question “what is the name?” parses very differently on a CV (doc_type=resume, typical_fields=[name, email, ...]) and on a 200-page annual report. The channel is dashed because it travels through PromptContext in the user content of the LLM call, not through a kwarg of the brick API. The brick API stays parse_question(question, *, context=PromptContext(...)) regardless of what doc_type is.

The second side channel: the feedback loop. Generation does not always succeed in one shot. The LLM may flag “the retrieved context covered only part of the answer” (complete_answer_found=False) or “the document uses ‘excess’ where the question said ‘deductible’” (llm_discovered_keywords). The orchestrator reads those signals and routes back, typically to retrieval with an expanded keyword list, sometimes to question parsing for a query rewrite. Article 13 (the workflow pipeline) walks the loop in detail. This article runs the single-pass version, which is what the typical short paper or compliance document needs.

Each of the four bricks composed here is developed, with its code in detail, in its own articles (this article shows only the wiring, not the per-brick code again):

Part A ran the bricks one at a time. Production code wraps them in a single function. The name says what the function does and what it does it on: pdf_qa does question-answering on a PDF. Its sibling excel_qa would do the same on an Excel workbook; pdf_translate would translate a PDF; pdf_compare would compare two. The four upgrades plug in at the four named steps; the rest is plumbing:

def pdf_qa(pdf_path, question, *, client=None, expert_dict=None,
           context: PromptContext | None = None):
    # Brick 1: Parsing: line_df + page_df + toc_df + parsing_summary
    parsed_pdf = parse_pdf(pdf_path, method="fitz")
    line_df, page_df, toc_df = parsed_pdf["line_df"], parsed_pdf["page_df"], parsed_pdf["toc_df"]
    parsing_summary = parsed_pdf["parsing_summary"]

    # Project parsing_summary into the DocContext that both LLM bricks read
    pipeline_ctx = (context or PromptContext()).with_doc_context(
        DocContext.from_parsing_summary(parsing_summary)
    )

    # Brick 2: Question parsing: one LLM call corrects + extracts, then expand
    parsed = parse_question(question, client=client, context=pipeline_ctx)
    keywords = expand_with_expert(parsed.keywords, expert_dict or 
    "answer": "RAG combines retrieval and generation by using a pre-traine…",
    "start_page_num": 2,
    "start_line_num": 46,
    "end_page_num": 3,
    "end_line_num": 40,
    "confidence": 0.98,
    "justification": "Lines 2:46 to 3:40 describe the mechanism: the retrieve…",
    "quotes": [
      "Figure 1: Overview of our approach. We combine a pre-traine…",
      "The retrieval component pη(z)
    shape = infer_answer_shape(question)

    # Brick 3: Retrieval: keywords on page_df + LLM TOC router
    kw_pages, _ = retrieve_pages(page_df, line_df, keywords, top_k=3)
    selection = reason_on_toc(question, toc_df, client=client)
    toc_pages = expand_sections_to_pages(toc_df, selection.section_ids)
    pages = sorted(set(kw_pages['page_num']) | toc_pages)
    filtered = line_df[line_df['page_num'].isin(pages)]

    # Brick 4: Generation: schema picked from expected_answer_shape,
    # pipeline_ctx threaded so the LLM sees doc_type / typical_fields
    schema = ListAnswer if shape == 'listing' else AnswerWithEvidence
    answer = llm_answer_with_evidence(question, filtered, client=client,
                                     context=pipeline_ctx)
    return answer, 
    "answer": "RAG combines retrieval and generation by using a pre-traine…",
    "start_page_num": 2,
    "start_line_num": 46,
    "end_page_num": 3,
    "end_line_num": 40,
    "confidence": 0.98,
    "justification": "Lines 2:46 to 3:40 describe the mechanism: the retrieve…",
    "quotes": [
      "Figure 1: Overview of our approach. We combine a pre-traine…",
      "The retrieval component pη(z

Production signature. The lib’s pdf_qa carries a few more kwargs the simplified form above hides:

• store: the per-brick cache.
• method: the parsing engine.
• top_k: retrieval breadth.
• use_toc: switches the LLM TOC router off when the document has no usable outline.
• include_bbox: for layout-sensitive cases.

They all default to behavior compatible with the short version. The retrieval body itself delegates to dispatch_page_retrieval from docintel.retrieval, the same shared helper the cross-document corpus_pdf_qa calls per-document, so single-doc and project-wide QA stay on the same TOC-routing logic. The body stays the same four-brick wiring.

Here is what each brick took in and produced on this run:

Brick 1, parsing. In: the Attention PDF path. Out: line_df, page_df, toc_df. The bootstrap chunk at the top of the article already showed each one; no separate figure here.

Brick 2, question parsing. In: the noisy question. Out: the parsed brief.

{
  "in": 
    "answer": "RAG combines retrieval and generation by using a pre-traine…",
    "start_page_num": 2,
    "start_line_num": 46,
    "end_page_num": 3,
    "end_line_num": 40,
    "confidence": 0.98,
    "justification": "Lines 2:46 to 3:40 describe the mechanism: the retrieve…",
    "quotes": [
      "Figure 1: Overview of our approach. We combine a pre-traine…",
      "The retrieval component pη(z,
  "out": 
    "answer": "RAG combines retrieval and generation by using a pre-traine…",
    "start_page_num": 2,
    "start_line_num": 46,
    "end_page_num": 3,
    "end_line_num": 40,
    "confidence": 0.98,
    "justification": "Lines 2:46 to 3:40 describe the mechanism: the retrieve…",
    "quotes": [
      "Figure 1: Overview of our approach. We combine a pre-traine…",
      "The retrieval component pη(z
}

Brick 3, retrieval. In: final_keywords + page_df + toc_df. Out: the retrieval state.

{
  "in": {
    "final_keywords": ["positional encoding", "sinusoidal", "learned"],
    "scope": "page_df + toc_df"
  },
  "out": {
    "keyword_candidates": [
      {"page": 6, "matched_keywords": ["positional encoding", "sinusoidal", "learned"], "match_count": 3},
      {"page": 9, "matched_keywords": ["positional encoding", "sinusoidal", "learned"], "match_count": 3},
      {"page": 4, "matched_keywords": ["learned"], "match_count": 1}
    ],
    "toc_routing": {
      "section_ids": ["10"],
      "reasoning": "Section 10, 'Positional Encoding', explicitly focuses on…",
      "selected_sections": [
        {"section_id": "10", "title": "Positional Encoding", "start_page": 6, "end_page": 6}
      ]
    },
    "merged_pages": [4, 6, 9]
  }
}

Brick 3 merges the keyword pages and the TOC-picked pages into one set. The table below shows the reasoning: the paper’s full TOC as backbone, with for each section the keywords detected in its lines and three Y/. flags showing whether that section was picked by keyword retrieval, by TOC retrieval, and whether it ended up in merged_pages. The merger is a deterministic union; this table makes the decision auditable section by section.

Brick 4, generation. In: question + the lines on merged_pages. Out: a ListAnswer with one item per option, line spans, verbatim quotes, plus the four context-quality indicators.

{
  "in": {
    "question": "What are the optoins for posiitional encoding?",
    "merged_pages": [4, 6, 9],
    "filtered_line_df": "256 rows"
  },
  "out": {
    "items": [
      {"text": "Sinusoidal positional encodings", "start_page_num": 6, "start_line_num": 33, "end_page_num": 6, "end_line_num": 35, "quote": "PE(pos,2i) = sin(pos/100002i/dmodel)\nPE(pos,2i+1) = cos(po…"},
      {"text": "Learned positional embeddings", "start_page_num": 6, "start_line_num": 41, "end_page_num": 6, "end_line_num": 41, "quote": "We also experimented with using learned positional embeddin…"},
      {"text": "positional embedding instead of sinusoids", "start_page_num": 9, "start_line_num": 111, "end_page_num": 9, "end_line_num": 111, "quote": "positional embedding instead of sinusoids"}
    ],
    "answer_found": true,
    "complete_answer_found": true,
    "context_completeness": 1.0,
    "context_structured": true,
    "confidence": 1.0,
    "caveats": []
  }
}

The line spans on each item are not abstract numbers. They map back to a real page in the source PDF, and the helper below draws a red rectangle around each item’s line range so the reader can see the answer on the page itself:

Twenty lines of Python, four LLM calls (two in question parsing and generation, plus the bricks that do not call the LLM). The provenance dict makes every step replayable: log the call, an auditor can trace from question to citation without re-running the pipeline.

The Attention paper above is a research paper. The same pipeline, with no code change, runs on documents shaped very differently. Three more tests: a compliance PDF, a research paper, and a stress case on a document with a broken TOC.

The NIST Cybersecurity Framework 2.0 is 32 pages, native TOC, single-column layout. The kind of document an enterprise team parses for retrieval. This time the question targets a single concept, not a list of options. The question parser’s shape inference flags it as single, and the pipeline dispatches to AnswerWithEvidence instead of ListAnswer. Same wrap function, different output schema. Listing questions are the subject of Article 12 (listing); here we exercise the other branch of the shape dispatch:

The four bricks walked on this run:

Brick 1, parsing. In: test2_pdf (the NIST CSWP-29 PDF). Out: line_df, page_df, toc_df.

Brick 2, question parsing. In: the noisy question. Out: the parsed brief.

{
  "in": {
    "raw_question": "How is a Profile defined in CSF 2.0?"
  },
  "out": {
    "raw_keywords": ["Profile", "CSF 2.0"],
    "corrected_keywords": ["Profile", "CSF 2.0"],
    "expert_keywords_added": [],
    "final_keywords": ["Profile", "CSF 2.0"],
    "expected_answer_shape": "single"
  }
}

Brick 3, retrieval. In: final_keywords + page_df + toc_df. Out: the retrieval state.

{
  "in": {
    "final_keywords": ["Profile", "CSF 2.0"],
    "scope": "page_df + toc_df"
  },
  "out": {
    "keyword_candidates": [
      {"page": 2, "matched_keywords": ["Profile", "CSF 2.0"], "match_count": 2},
      {"page": 5, "matched_keywords": ["Profile", "CSF 2.0"], "match_count": 2},
      {"page": 4, "matched_keywords": ["Profile"], "match_count": 1}
    ],
    "toc_routing": {
      "section_ids": ["3", "2", "11"],
      "reasoning": "Section 3.1 (CSF Profiles) directly addresses the definit…",
      "selected_sections": [
        {"section_id": "2", "title": "3. Introduction to CSF Profiles and Tiers", "start_page": 11, "end_page": 14},
        {"section_id": "3", "title": "3.1. CSF Profiles", "start_page": 11, "end_page": 12},
        {"section_id": "11", "title": "Appendix C. Glossary", "start_page": 31, "end_page": 32}
      ]
    },
    "merged_pages": [2, 4, 5, 11, 12, 13, 14, 31, 32]
  }
}

The TOC-by-section audit on the same run:

Brick 4, generation. In: question + the lines on merged_pages. Out: an AnswerWithEvidence with one answer string, one citable span, verbatim quotes, plus the quality indicators (complete_answer_found, context_structured, confidence).

{
  "in": {
    "question": "How is a Profile defined in CSF 2.0?",
    "merged_pages": [2, 4, 5, 11, 12, 13, 14, 31, 32],
    "filtered_line_df": "280 rows"
  },
  "out": {
    "answer": "A Profile in CSF 2.0, specifically a CSF Organizational Pro…",
    "start_page_num": 11,
    "start_line_num": 8,
    "end_page_num": 12,
    "end_line_num": 20,
    "confidence": 1.0,
    "justification": "Lines 11:8-12:20 provide a detailed definition of CSF P…",
    "quotes": [
      "A CSF Organizational Profile describes an organization's cu…",
      "Every Organizational Profile includes one or both of the fo…",
      "A Community Profile is a baseline of CSF outcomes that is c…"
    ],
    "caveats": [],
    "complete_answer_found": true,
    "context_structured": true,
    "llm_discovered_keywords": ["Organizational Profile", "Current Profile", "Target Profile", "Community Profile"]
  }
}

The same span turned into a rectangle on the source page:

Because the question wording does not trigger LISTING_TRIGGERS (no options, all, which, what are), the shape inference returns single, and the pipeline dispatches the call to AnswerWithEvidence instead of ListAnswer. Same four bricks, same audit trail, different output schema.

The single span lands on the page that defines a CSF Profile, and the annotated page above shows the one rectangle highlighted in context. The quality indicators (complete_answer_found, context_structured, confidence) come back clean, so the downstream router would ship this answer as is.

Listing questions get their own dedicated treatment in Article 12 (listing); here the takeaway is that the wrap function carries both shapes without a separate code path on the caller side.

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (Lewis et al. 2020) is the paper that introduced the RAG architecture this series is about. 18 pages, native TOC, research-paper style with a results-table-heavy mid-section. Another single-answer question, this time about the paper’s central architectural pattern:

Same four-brick walk on this run:

Brick 1, parsing. In: test3_pdf (the original RAG paper, 18 pages). Out: line_df, page_df, toc_df.

Brick 2, question parsing. In: the noisy question. Out: the parsed brief.

{
  "in": {
    "raw_question": "How does RAG combine retrieval and generation?"
  },
  "out": {
    "raw_keywords": ["RAG", "retrieval", "generation"],
    "corrected_keywords": ["RAG", "retrieval", "generation"],
    "expert_keywords_added": [],
    "final_keywords": ["RAG", "retrieval", "generation"],
    "expected_answer_shape": "single"
  }
}

Brick 3, retrieval. In: final_keywords + page_df + toc_df. Out: the retrieval state.

{
  "in": {
    "final_keywords": ["RAG", "retrieval", "generation"],
    "scope": "page_df + toc_df"
  },
  "out": {
    "keyword_candidates": [
      {"page": 1, "matched_keywords": ["RAG", "retrieval", "generation"], "match_count": 3},
      {"page": 2, "matched_keywords": ["RAG", "retrieval", "generation"], "match_count": 3},
      {"page": 3, "matched_keywords": ["RAG", "retrieval", "generation"], "match_count": 3}
    ],
    "toc_routing": {
      "section_ids": ["2", "3", "4"],
      "reasoning": "Section 2.1 Models likely explains the overall architectu…",
      "selected_sections": [
        {"section_id": "2", "title": "2.1 Models", "start_page": 3, "end_page": 3},
        {"section_id": "3", "title": "2.2 Retriever: DPR", "start_page": 3, "end_page": 3},
        {"section_id": "4", "title": "2.3 Generator: BART", "start_page": 3, "end_page": 3}
      ]
    },
    "merged_pages": [1, 2, 3]
  }
}

The TOC-by-section audit on the same run:

Brick 4, generation. In: question + the lines on merged_pages. Out: an AnswerWithEvidence describing how RAG couples the retriever to the generator, with one cited span and the quality indicators.

{
  "in": {
    "question": "How does RAG combine retrieval and generation?",
    "merged_pages": [1, 2, 3],
    "filtered_line_df": "200 rows"
  },
  "out": {
    "answer": "RAG combines retrieval and generation by using a pre-traine…",
    "start_page_num": 2,
    "start_line_num": 46,
    "end_page_num": 3,
    "end_line_num": 40,
    "confidence": 0.98,
    "justification": "Lines 2:46 to 3:40 describe the mechanism: the retrieve…",
    "quotes": [
      "Figure 1: Overview of our approach. We combine a pre-traine…",
      "The retrieval component pη(z|x) is based on DPR ... Calcula…",
      "The generator component pθ(yi|x, z, y1:i-1) could be modell…",
      "RAG-Sequence Model ... the model uses the same document to…",
      "In the RAG-Token model we can draw a different latent docum…"
    ],
    "caveats": [
      "The detailed algorithmic steps (e.g., training or more adva…"
    ],
    "complete_answer_found": true,
    "context_structured": true,
    "llm_discovered_keywords": ["retriever", "generator", "marginalize", "latent document", "seq2seq", "DPR", "BART", "Maximum Inner Product Search", "RAG-Sequence", "RAG-Token"]
  }
}

The same cited span drawn on the source page:

The shape inference returns single again, and the pipeline dispatches to AnswerWithEvidence. The question is broader than NIST’s Profile definition because it asks about an architectural pattern that spans the retriever, the generator, and the way the two are jointly trained. Retrieval pulls a handful of pages across the body; the generator synthesises one paragraph that ties them together, with one citable span. As with the NIST run, the four indicators all stay clean on this well-structured PDF.

The pipeline ran on every document above, answering three and honestly declining the fourth. The fair question: would a naive RAG (Article 1 (minimal RAG)’s baseline, keyword-match or embed pages, keep the top few, ask) have done as well? We ran pdf_qa_baseline against pdf_qa on the same questions, plus a few more standards to see where the two diverge.

Be honest about the result: on the two clean arXiv papers, the naive baseline answers correctly too. They are short and well-structured, and the answer keyword-matches. The gap is not on easy inputs. It opens on the standards. On NIST CSF 2.0, asked “How is a Profile defined?”, naive retrieves pages full of the word Profile but never the one that defines it, and returns “not defined in these lines” at 0.10. On the 400-plus-page NIST SP 800-53 catalog, asked for one control, it retrieves nothing usable and returns “NA” at 0.00. On NIST SP 800-207 it hands back three of the seven zero-trust tenets, confident at 0.95 that a partial list is the whole answer. On FIPS 199 it defines only the high impact level and admits low and moderate were never in its context. Our pipeline routes each question on the document’s own table of contents, anchors the right section, and returns the complete, cited answer every time.

That pattern carries the argument. A naive RAG is a keyword-or-cosine bet, and it pays off until the document is long enough, or its vocabulary far enough from the question, that the answer sinks below the top-k cutoff. Then the model is handed context that does not contain the answer, and it does one of two things: it says “not found” (the honest 0.10 or NA case) or it invents a plausible answer from the wrong pages (the confident, partial tenets list). The second is what teams report as a hallucination. It is rarely the model inventing from nothing. Usually it is answering the wrong context faithfully, and often both at once: handed pages that do not contain the answer, it fills the gap with something plausible. Article 7quinquies (most RAG hallucinations are retrieval failures) makes that case at the retrieval level; here it shows up end to end.

This is why the four bricks are context engineering, not retrieval tuning. Each brick shapes what the model finally sees: parsing keeps the structure, question parsing widens the vocabulary, retrieval routes on the document’s own map, generation binds the answer to a citable span. Get the context right and a confident wrong answer has nowhere to come from. Prompt engineering isn’t enough: how four bricks of context engineering stop RAG hallucinations (Article 9bis, link to come) takes this contrast much further, one failure per brick, with a naive baseline built to break at exactly that brick. This section is the short version; that article is the full diagnosis.

The three runs above worked without a single line of code change because of how the pipeline is decomposed. Each brick has one job and one typed output, so swapping the document changes what flows through the bricks, never the bricks themselves.

Each brick has a typed contract with its neighbours. Parsing returns a small relational set of DataFrames (line_df, page_df, toc_df) with known schemas. Question parsing returns a structured brief (corrected keywords, expert expansion, expected answer shape). Retrieval returns a merged page set and the filtered line_df rows on those pages. Generation returns a typed Pydantic object whose schema depends on the answer shape, carrying the items and the four quality indicators. Every intermediate output is a named, inspectable, dump-to-JSON-able object.

Because the contracts are explicit, each brick can be swapped without rewiring the rest. The four upgrades that built up this article are the proof:

Same paper, same question as Article 1 (minimal RAG), four bricks at their Part II level. The structured contract between bricks makes the pipeline composable: DataFrames out of parsing and retrieval, Pydantic schemas out of question parsing and generation. A team can adopt the upgrades one brick at a time without rewiring the rest, and an auditor can replay every step from the provenance dict.

Article 10 (adaptive parsing) continues from where this one stops. The pipeline assumes parsing built a clean toc_df; when parsing fails (scans without OCR, broken bookmarks, layouts the heuristics miss), retrieval has to fall back to body-only signals. The next article walks the cheap-parsing-first, deeper-parsing-on-demand pattern.