with an enormous corpus of text data, where they, during their pre-training stage, essentially consume the entire internet. LLMs thrive when they have access to all relevant data to respond to user questions appropriately. However, in many cases, we limit the capabilities of our LLMs by not providing them enough data. In this article, I’ll discuss why you should care about feeding our LLM more data, how to fetch this data, and specific applications.
I’ll also start with a new feature in my articles: Writing out my main goal, what I want to achieve with the article, and what you should know after reading it. If successful, I’ll start writing it into each of my articles:
My goal for this article is to highlight the importance of providing LLMs with relevant data, and how you can feed it into your LLMs for improved performance
You can also read my articles on How to Analyze and Optimize Your LLMs in 3 Steps and Document QA using Multimodal LLMs.
Table of contents
Why add more data to LLMs?
I’ll start my article off by pointing out why it’s important. LLMs are incredibly data hungry, meaning that they require a lot of data to -work well. This is commonly shown in the pre-training corpus of LLMs, which consists of trillions of text tokens being used for training the LLM.
However, the concept of utilizing a lot of data also applies to LLMs during inference time (when you utilize the LLM in production). You need to provide the LLM with all necessary data to answer a user request.
In a lot of cases, you inadvertently reduce the LLM’s performance by not providing relevant information.
For example, if you create a question answering system, where users can upload files and talk to them. Naturally, you provide the text contents of each file so that the user can chat with the document; however, you could, for example, forget to add the filenames of the documents to the context the user is chatting with. This will impact the LLM’s performance, for example, if some information is only present in the filename or the user references the filename in the chat. Some other specific LLM applications where additional data is useful are:
- Classification
- Information extraction
- Keyword search for finding relevant documents to feed to LLM
In the rest of the article, I’ll discuss where you can find such data, techniques to retrieve additional data, and some specific use cases for the data.
In this section, I’ll discuss data that you likely already have available in your application. One example is my last analogy, where you have a question answering system for files, but forget to add the filename to the context. Some other examples are:
- File extensions (.pdf, .docx, .xlsx)
- Folder path (if the user uploaded a folder)
- Timestamps (for example, if a user asks about the most recent document, this is required)
- Page numbers (the user might ask the LLM to fetch specific information located on page 5)
There are a ton of other such examples of data you likely already have available, or that you can quickly fetch and add to your LLM’s context.
The type of data you have available will vary widely from application to application. A lot of the examples I’ve provided in this article are tailored to text-based AI, since that’s the space I spend the most time in. However, if you, for example, work more on visual AI or audio-based AI, I urge you to find similar examples in your space.
For visual AI, it could be:
- Location data for where the image/video was taken
- The filename of the image/video file
- The author of the image/video file
Or for audio AI, it could be
- Metadata about who is speaking when
- Timestamps for each sentence
- Location data from where the audio was recorded
My point being, there is a plethora of available data out there; all you need to do is look for it and consider how it can be useful for your application.
Sometimes, the data you already have available is not enough. You want to provide your LLM with even more data to help it answer questions appropriately. In this case, you need to retrieve additional data. Naturally, since we are in the age of LLMs, we’ll utilize LLMs to fetch this data.
Retrieving information beforehand
The easiest approach is to retrieve additional data by fetching it before processing any live requests. For document AI, this means extracting specific information from documents during processing. You might extract the type of document (legal document, tax document, or sales brochure) or specific information contained in the document (dates, names, locations, …).
The advantage of fetching the information beforehand is:
- Speed (in production, you only need to fetch the value from your database)
- You can take advantage of batch processing to reduce costs
Today, fetching this kind of information is rather simple. You set up an LLM with a specific system prompt to fetch information, and feed the prompt along with the text into the LLM. The LLM will then process the text and extract the relevant information for you. You might want to consider evaluating the performance of your information extraction, in which case you can read my article on Evaluating 5 Millions LLM Requests with Automated Evals.
You likely also want to map out all the information points to retrieve, for example:
When you have created this list, you can retrieve all your metadata and store it in the database.
However, the main downside of fetching information beforehand is that you have to predetermine which information to extract. This is difficult in a lot of scenarios, in which case you can do live information retrieval, which I cover in the next section.
On-demand information retrieval
When you can’t determine which information to retrieve beforehand, you can fetch it on demand. This means setting up a generic function that takes in a data point to extract and the text to extract it from. For example
import json
def retrieve_info(data_point: str, text: str) -> str:
prompt = f"""
Extract the following data point from the text below and return it in a JSON object.
Data Point: {data_point}
Text: {text}
Example JSON Output: {{"result": "example value"}}
"""
return json.loads(call_llm(prompt))You define this function as a tool your LLM has access to, and which it can call whenever it needs information. This is essentially how Anthropic has set up their deep research system, where they create one orchestrator agent that can spawn sub-agents to fetch additional information. Note that giving your LLM access to use additional prompts can lead to a lot of token usage, so you should pay attention to you’re LLM’s token spend.
Until now, I’ve discussed why you should utilize additional data and how to get a hold of it. However, to fully grasp the content of this article, I’ll also provide specific applications where this data improves LLM performance.
Metadata filtering search
My first example is that you can perform a search with metadata filtering. Providing information such as:
- file-type (pdf, xlsx, docx, …)
- file size
- Filename
It can help your application when fetching relevant information. This can, for example, be information fetched to be fed into your LLM’s context, like when performing RAG. You can utilize the additional metadata to filter away irrelevant files.
A user might have asked a question pertaining to only Excel documents. Using RAG to fetch chunks from files other than Excel documents is, therefore, bad usage of the LLM’s context window. You should instead filter available chunks to only find Excel documents, and utilize chunks from Excel documents to best answer the user’s query. You can learn more about handling LLM contexts in my article on building effective AI agents.
AI agent internet search
Another example is if you’re asking your AI agent questions about recent history that happened after the pre-training cutoff for the LLM. LLMs typically has a training data cutoff for pre-training data, because the data needs to be carefully curated, and keeping it fully up to date is challenging.
This presents a problem when users ask questions about recent history, for example, about recent events in the news. In this case, the AI agent answering the query needs access to an internet search (essentially performing information extraction on the internet). This is an example of on-demand information extraction.
Conclusion
In this article, I’ve discussed how to significantly enhance your LLM by providing it with additional data. You can either find this data in your existing metadata (filenames, file-size, location data), or you can retrieve the data through information extraction (document type, names mentioned in a document, etc). This information is often critical to an LLM’s ability to successfully answer user queries, and in many instances, the lack of this data essentially guarantees the LLM’s failure to answer a question correctly.
👉 Find me on socials:
🧑💻 Get in touch
✍️ Medium