Retrieval-Augmented Generation (RAG) has evolved rapidly in recent times. The original RAG paradigm was designed to be straightforward: retrieve the most relevant chunks from a corpus and use them to answer a question. While effective for many document-centric queries, this approach begins to struggle when questions extend beyond the boundaries of a single document. Or when questions are temporal in nature, such as tracking the history of an entity over years.
A few examples would be the following:
- Which companies did we acquire over the last decade?
- How has our AI strategy evolved since 2018?
- Which sustainability commitments announced in previous annual reports were eventually fulfilled?
- How has management’s capital allocation philosophy changed over the years?
Recently, a new architectural pattern, LLM-Wiki, has proposed an interesting solution to this challenge. Rather than repeatedly searching raw documents during retrieval, an extensive semantic processing step during ingestion builds a persistent knowledge base consisting of canonical pages linked through an index. Future queries are answered primarily from this compiled knowledge rather than the original documents.
While an elegant idea, this raises important architectural questions such as:
- Should enterprise knowledge be semantically compiled before anyone asks for it?
- Is there a way to avoid paying this ingestion cost over an entire corpus of large documents (e.g. voluminous annual report, contracts etc), most of whose content may never actually be queried about?
- What should be present in the canonical pages such that almost every future query can be answered without referring to the raw source ? An over-engineered canonical page may itself be a very long one, and over time grow significantly larger than the source documents themselves, resulting in higher retrieval latency and cost.
In this article, I shall compare LLM-Wiki with the Proxy-Pointer (PP) architecture, a structure-aware retrieval framework that approaches the same problem from a fundamentally different perspective. Rather than eagerly compiling semantic knowledge during ingestion, Proxy-Pointer preserves the structural organization of documents and performs semantic synthesis only when a query requires it.
This kind of lazy semantic analysis of content is consistent with the premise of Proxy-Pointer to deliver precision without the ingestion tax, which I have detailed in my articles Proxy-Pointer RAG: Achieving Vectorless Accuracy at Vector RAG Scale and Cost and Proxy-Pointer RAG: Structure Meets Scale — 100% Accuracy with Smarter Retrieval.
Let’s see the comparison with the help of a real-life scenario.
The LLM-Wiki Approach
LLM-Wiki records the temporal context by moving much of the reasoning to ingestion time. Instead of treating documents as isolated artifacts, each incoming document is processed by an LLM that extracts concepts, entities, relationships, and facts. These are merged into a collection of canonical pages representing persistent knowledge about the organization.
For example, multiple annual reports may collectively update canonical pages such as “Acquisitions”, “Artificial Intelligence”, “Cloud Strategy”, “Sustainability”, “Product Portfolio”, “Supply chain” etc. Each new annual report contributes additional knowledge to these pages.
Alongside these canonical pages, the system maintains an index that helps locate the appropriate pages for future queries.
Conceptually, the architecture resembles the following:
When a user asks, “Which companies did we acquire during the last decade?“, the system primarily searches the index, locates the acquisition-related canonical page, and answers directly from the compiled knowledge.
The original documents remain available as an immutable data store, but ideally they are no longer part of the normal query path.
From an architectural perspective, this is an example of knowledge compilation.
How do we build this Wiki?
Let’s say the input is a 200 page annual report. In order to optimise on the compute cost, we would like to extract information related to all the canonical pages in one pass and update the relevant pages. This would mean asking the LLM to run dozens of extraction tasks simultaneously:
Detect: acquisitions, products, AI initiatives, cloud strategy, sustainability, geopolitical risks, partnership, supply chain, regulatory changes, customers, litigation, investments, subsidiaries, organizational restructuring, technology trends, …..”
As the number of semantic objectives grows, the model’s attention is divided across them. The model must continuously decide which pieces of information belong to which canonical pages, whether new pages should be created, and how newly discovered facts relate to existing knowledge. As prompt complexity increases, asking the model to optimize for many objectives simultaneously, LLMs exhibit high-dimensional information extraction degradation. Especially for long and dense documents, this can reduce recall, increase omissions and miss less salient information.
More fundamentally, the system faces a difficult prediction problem.
It must decide during ingestion which information is worth materializing into the canonical knowledge base while anticipating every possible question that users will ask in the future.
For instance, “How has the revenue contribution of Subsidiary X evolved since its acquisition?“
The relevant evidence for the above query may be scattered across multiple annual reports, embedded within detailed financial statements, footnotes, or segment reporting tables. Materializing every potentially relevant financial line item into canonical pages would significantly increase the size and maintenance cost of the knowledge base. Conversely, omitting such details means that some future queries will inevitably require revisiting the original documents.
This illustrates a fundamental trade-off of preemptive semantic compilation: the system must continually balance completeness, maintainability, and ingestion cost, despite not knowing which aspects of today’s documents will be necessary and sufficient for tomorrow’s questions.
The Proxy-Pointer Approach
Proxy-Pointer approaches the same problem from a different direction.
Rather than building a preemptive semantic knowledge base, Proxy-Pointer builds a structural representation (skeletal tree) of each document at ingestion. This process is purely regex based, not utilizing a LLM and incurring zero cost. The actual semantic compilation is postponed to the point of retrieval for a just-in-time reasoning and response. Let’s see how this will work:
Proxy-Pointer leverages standard vector indexes with important modifications. Chunks are created within section boundaries and are never allowed to overlap with subsequent sections. Each chunk is tagged with the metadata pointing to the section where it came from and boundaries (line numbers) for retrieval. When a query comes, the Vector search + LLM re-ranker pipeline selects the most relevant top-k sections for context.
The above pipeline is designed to utilize the section/sub-section structure of a complex document as a semantic map to accurately surface the right content required to answer the user’s question. As is demonstrated in the links mentioned at the beginning of the article, the retrieval pipeline delivers surgical precision over complex queries.
Does this work for cross-document queries?
Let us consider the question
“What happened to the companies we acquired over the last decade?“
The answer is not stored in a single document. The answer must be constructed by correlating merger and acquisition information spread across multiple annual reports. A company acquired in 2016 may be mentioned repeatedly over the following years before eventually disappearing from later reports after being fully integrated into the parent organization.
The LLM-Wiki method would have created a canonical page recording all mergers and acquisitions upon ingestion of each annual report over the last decade. And can therefore, provide that page as the context to the LLM.
Proxy-Pointer on the other hand, would reason that the query requires annual reports of the company for the last 10 years. And given that each of the chunk metadata in the vector index would have reference to the year (metadata filtering using structural breadcrumbs — e.g. Year: 2024 > Section: M&A), the pipeline would be able to retrieve top-k sections from each of the reports relevant for mergers and acquisitions.
Assuming k=5, this would result in 50 sections. If this exceeds the LLM context window, the documents can be analysed in groups to extract the acquisition information and then provided to the final response generation LLM call as context.
Key differences with the LLM-Wiki approach
The most obvious difference, as has been mentioned before, is that there is no upfront ingestion cost, beyond creating a standard vector index. The semantic compilation and analysis is deferred (lazy); it occurs if the query needs it.
The second is that even during retrieval, the analysis of the query in the previous section may need only 50 sections. This is much more efficient than analysing 10 annual reports, each of 200 pages, upfront to form the canonical pages in the LLM-Wiki method.
The third difference is that Proxy-Pointer leverages the structure tree as a semantic map, which enables it to reduce the sections to be analysed based on the user query. LLM-Wiki on the other hand, does an exhaustive analysis of the entire document to extract entities and relationships to build the knowledge base the user wants to store, most of which may never be asked for.
Finally, the user is spared from making the difficult design decisions upfront as to what queries are likely to come in future. Building the compiler that can accurately organize facts, concepts, relationships and other information with the expectation that almost every query can be answered without referring the source document is a challenging ask.
Large organizations routinely maintain repositories containing millions of pages. These include annual reports, regulatory filings, engineering specifications, policies, contracts, technical manuals and many more. Only a small fraction of this is ever queried by the users. Most of the queries are directed to the current and latest version of reports and policies. Even within the queries asked by users, temporal queries form a small subset.
Proxy-Pointer would only analyse as much as is required, when it is required, potentially reducing costs significantly. For a one-off temporal query, it analyses only the sections needed by the question, whereas LLM-Wiki has already analysed every annual report during ingestion. If similar temporal queries recur, the synthesized response (or intermediate summaries) can be cached and reused, allowing Proxy-Pointer to amortize the semantic reasoning cost without requiring every document to be semantically compiled upfront.
Explainability of the responses
Enterprise AI systems increasingly operate in regulated environments where explainability is as important as correctness. To a user “What is the answer?” always comes with “Why should I trust it?“.
Proxy-Pointer performs reasoning directly over source sections, and the traceability is naturally preserved. The evidence used during reasoning is immediately available for inspection.
LLM-Wiki can preserve explainability through citations back to source documents and pages, but the primary reasoning for responses occurs over compiled representations. As those representations evolve through repeated updates, this becomes another aspect of the system that must be maintained and verified over time.
Conclusion
The debate between LLM-Wiki and Proxy-Pointer is not really about retrieval. It is about when semantic understanding should occur.
LLM-Wiki assumes that enterprise knowledge is valuable enough to justify compiling it during ingestion, so that future queries become inexpensive. Proxy-Pointer makes the opposite assumption. It argues that most enterprise content will never be queried, and therefore semantic reasoning should be performed only when a question demonstrates that the computation is necessary.
Organizations with stable corpora and highly repetitive conceptual queries may find value in maintaining a compiled knowledge base. Conversely, large enterprise repositories characterized by evolving documents, unpredictable questions, and occasional temporal analysis may benefit from deferring semantic compilation until retrieval.
The broader lesson extends beyond these two architectures. As enterprise AI systems evolve from document retrieval to knowledge reasoning, architects must increasingly balance cost against value. Should knowledge be compiled because it might be useful one day, or should it be synthesized only when a user asks for it?
Proxy-Pointer makes a clear architectural choice. Instead of predicting tomorrow’s questions, it lets tomorrow’s questions decide what knowledge needs to be created.
Open-Source Repository
Proxy-Pointer is fully open-source (MIT License) and can be accessed at Proxy-Pointer Github repository.
Clone the repo. Try the use cases. Let me know your thoughts.
Connect with me and share your comments at www.linkedin.com/in/partha-sarkar-lets-talk-AI