, I have written quite a lot about RAG, starting with the Hitchhiker’s Guide to RAG with ChatGPT API and LangChain, and then exploring various topics related to RAG and AI, like chunking, hybrid search, reranking, contextual retrieval, and a three-part series on evaluating retrieval quality. In other words, we have covered a lot of ground on the RAG side of things.
What we haven’t talked about as explicitly is the other major technique people reach for when they want to improve an LLM app for a specific domain. That is fine-tuning. And in particular, we have not talked about what happens when you put the two side by side and try to figure out which one you actually need.
If you search “RAG vs fine-tuning” online, you will find a lot of content that treats this as a competition with a winner. For some, RAG wins because it is cheaper to set up, for some others, fine-tuning wins because it produces better results, and so on. The problem with this framing is that it is fundamentally misleading, since RAG and fine-tuning are not competing techniques, but rather techniques that solve different problems at different layers of an AI application. Understanding what each one actually does is the prerequisite for making a good decision.
So, let’s take a look!
🍨 DataCream is a newsletter about AI, data, and tech. If you are interested in these topics, subscribe here!
What is RAG and what does it actually do?
If you have followed this series, you already have a solid intuition for RAG. But let’s state it one more time, because the precise definition matters for the comparison with fine-tuning that follows.
So, RAG, or Retrieval-Augmented Generation, is a technique that enhances an LLM’s response by retrieving relevant external information at inference time and injecting it into the prompt. The model itself is not changed in any way. What changes is what it sees as input.
The pipeline looks something like this:
- Firstly, external documents (the knowledge base we want to utilize) are processed into vector embeddings and stored in a vector database.
- When a user submits a query, the query is also converted to an embedding, and the most semantically similar document chunks are retrieved from the database.
- Those chunks are then passed to the LLM along with the user’s query, so the model can generate a response grounded in that specific retrieved context.
And that’s it.
Here is a minimal RAG example using the OpenAI API:
from openai import OpenAI
import numpy as np
client = OpenAI(api_key="your_api_key")
# our tiny knowledge base
documents = [
"pialgorithms is an AI-powered document management platform.",
"pialgorithms allows teams to search, extract, and automate document workflows.",
"pialgorithms was founded in Athens, Greece.",
]
# embed the knowledge base
def embed(texts):
response = client.embeddings.create(
model="text-embedding-3-small",
input=texts
)
return [r.embedding for r in response.data]
doc_embeddings = embed(documents)
# embed the user query and retrieve the most relevant chunk
query = "Where is pialgorithms based?"
query_embedding = embed([query])[0]
# cosine similarity
def cosine_similarity(a, b):
return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))
similarities = [cosine_similarity(query_embedding, doc_emb) for doc_emb in doc_embeddings]
best_match = documents[np.argmax(similarities)]
# inject retrieved context into the prompt
response = client.chat.completions.create(
model="gpt-4o-mini",
messages=[
{
"role": "system",
"content": f"Answer the user's question using only the following context:\n\n{best_match}"
},
{
"role": "user",
"content": query
}
]
)
print(response.choices[0].message.content)
# pialgorithms is based in Athens, Greece.Let’s take a moment to understand what is really happening here. Of course, the model has no idea what pialgorithms is from its training, but because we retrieved the right document chunk and injected it into the prompt, the model is able to answer accurately. The knowledge comes from outside the model, at the moment of the query, and the model itself is untouched.
This is the core of what RAG does: it gives the model access to external knowledge it was never trained on, dynamically, at inference time.
And based on the way it works, RAG does well on specific types of tasks, as for instance:
- Answering questions about documents, knowledge bases, or data that the model has never seen
- Staying up to date without retraining, since the knowledge base can be independently updated at any time
- Providing traceable, citable answers, since you know exactly which document chunk was retrieved
- Handling private or proprietary information safely, without including that information in the model
On the flip side, here is what RAG won’t do: it is not going to change the model’s behaviour, tone, reasoning style, or task performance. If your model tends to be verbose, RAG won’t make it more concise. If it struggles with a particular output format, RAG will not fix that.
What is fine-tuning and what does it actually do?
Fine-tuning is the process of taking a pre-trained model and continuing to train it on a new, task-specific dataset, updating its weights in the process. To put it differently, while RAG changes the inputs of the model, fine-tuning changes the model itself.
More specifically, a base model like GPT-4o-mini is pre-trained on a massive general dataset. Fine-tuning takes that model and runs an additional, shorter training loop on specific examples relevant to our specific use case. Those examples are typically in the form of input-output pairs. In this way, the model’s weights are adjusted so that it produces outputs that look more like those example pairs.
Here is what a fine-tuning job would look like using the OpenAI API:
from openai import OpenAI
import json
client = OpenAI(api_key="your_api_key")
# Step 1: prepare training data as a JSONL file
# each example is a conversation with a desired output
training_examples = [
{
"messages": [
{"role": "system", "content": "You are a concise technical assistant. Always respond in one sentence."},
{"role": "user", "content": "What is a vector database?"},
{"role": "assistant", "content": "A vector database stores and retrieves data as high-dimensional numerical vectors, enabling fast semantic similarity search."}
]
},
{
"messages": [
{"role": "system", "content": "You are a concise technical assistant. Always respond in one sentence."},
{"role": "user", "content": "What is chunking in RAG?"},
{"role": "assistant", "content": "Chunking is the process of splitting large documents into smaller pieces before embedding them, so they fit within model context limits and improve retrieval precision."}
]
},
# in practice you would want at least 50-100 examples
]
# save as JSONL
with open("training_data.jsonl", "w") as f:
for example in training_examples:
f.write(json.dumps(example) + "\n")
# upload the training file
with open("training_data.jsonl", "rb") as f:
training_file = client.files.create(file=f, purpose="fine-tune")
# create the fine-tuning job
fine_tune_job = client.fine_tuning.jobs.create(
training_file=training_file.id,
model="gpt-4o-mini-2024-07-18"
)
print(fine_tune_job.id)Once the fine-tuning job completes, OpenAI returns a unique model identifier for your newly fine-tuned model, in the format ft:base-model:your-org:your-suffix:unique-id. This is now a distinct model that lives in your OpenAI account, separate from the base gpt-4o-mini.
On print, we would get back an id for that fine-tuned model, looking something like this:
ft:gpt-4o-mini-2024-07-18:your-org:your-suffix:abc123
We can then call it exactly like any other model, just by passing that identifier in the model parameter:
# once the job is complete, use the fine-tuned model
response = client.chat.completions.create(
model="ft:gpt-4o-mini-2024-07-18:your-org:your-suffix:abc123",
messages=[
{"role": "user", "content": "What is prompt caching?"}
]
)
print(response.choices[0].message.content)The difference is that this model has already internalised the behaviour we trained it on: in our example, it will now consistently respond in one concise sentence, without us having to instruct it to do so in every system prompt. That is the kind of thing fine-tuning is genuinely good at: consistent formatting, specific tone, adherence to a particular output structure, or improved performance on a very specific task type. And that, in essence, is what fine-tuning does.
Notice how fine-tuning doesn’t have any impact on incorporating specific information in the model. Unlike what one might intuitively assume, fine-tuning a model on your company’s documents won’t make the model “learn” that information and be able to answer questions about it reliably. It may indeed result in the model memorizing specific facts from training examples here and there, but this memorization is brittle and unreliable. The most probable outcome would be a model hallucinating about topics appearing in the training examples, rather than a model accurately recalling specific details appearing in those examples. Thus, if knowledge retrieval is what you need, RAG is the right tool, not fine-tuning.
More specifically, fine-tuning actually does well on the following:
- “Teaching” the model a consistent output format, tone, or style
- Improving performance on a specific, narrow task type (e.g. always generating valid JSON, always summarising in three bullet points, and so on)
- Reducing the need for long, repetitive system prompts by integrating those instructions into the model
- Adapting the model to domain-specific language or terminology, so it understands and uses the right vocabulary
Nonetheless, fine-tuning does not do that well in:
- Adding reliable factual knowledge, the model can recall accurately
- Keeping the model up to date with changing information
- Providing traceable, citable answers
So, when do we use each and when do we use both?
Now that we understand what each technique actually does, the “RAG vs fine-tuning” question becomes much easier to answer, because in most cases it is not really a “vs” type of question at all.
RAG and fine-tuning operate at different layers of an AI application. RAG operates at the knowledge layer, meaning it controls what information the model has access to. On the flip side, fine-tuning operates at the behaviour layer, meaning it defines the way the model processes the provided information and generates responses. These two layers are independent of each other, which means you can use RAG, fine-tuning, or both, depending on what you are trying to achieve.
So, here is a practical decision framework for deciding what to use:
The scenario where we use both RAG and fine-tuning is actually the most common in real production systems. The simplest way to keep the two straight is this: fine-tune for behaviour, use RAG for knowledge.
Imagine, for example, we are building a customer support assistant for a software product, and we need it to:
- Always respond in a specific tone and format, consistent with our software brand
- Have accurate, up-to-date knowledge of our product’s documentation
For such a task, we would need to utilize both RAG and fine-tuning. In particular, fine-tuning would handle the first requirement by allowing the model to learn from examples of ideal customer support responses, teaching it the right tone, the right level of detail, and the right output format. The second requirement would be covered by RAG: at inference time, the most relevant information from the product’s documentation is retrieved and injected into the prompt, allowing the model to provide reliable answers grounded in the documentation.
So, in practice, we can combine both fine-tuning and RAG by calling a fine-tuned model the same way we would call any other model, but also inject retrieved context into the system prompt, exactly as we would do in a standard RAG pipeline.
# combining fine-tuned model with RAG
response = client.chat.completions.create(
model="ft:gpt-4o-mini-2024-07-18:your-org:support-style:abc123", # fine-tuned for tone/format
messages=[
{
"role": "system",
"content": f"You are a helpful support assistant for pialgorithms. "
f"Use only the following documentation to answer:\n\n{retrieved_context}" # RAG context
},
{
"role": "user",
"content": user_question
}
]
)Fine-tuning makes the model know how to appropriately respond, and RAG tells it what to say. Thus, this isn’t a “fine-tuning vs RAG” question, but rather fine-tuning and RAG complement one another and do different things.
On my mind
What I find most interesting about the RAG vs fine-tuning debate is how often it is framed as a question about which technique is better, when the more useful question is what problem you are actually trying to solve.
RAG and fine-tuning address different failure modes of a base LLM. If a base model fails because it does not know something, that is a knowledge problem, and RAG solves it. If a base model fails because it behaves inconsistently or produces outputs in the wrong format, that is a behaviour problem, and fine-tuning solves it. If your model is failing for both reasons at once, you may genuinely need both.
✨ Thank you for reading! ✨
If you made it this far, you might find pialgorithms useful: a platform we’ve been building that helps teams securely manage organisational knowledge in one place.
Loved this post? Join me on 💌 Substack and 💼 LinkedIn
All images by the author, except where mentioned otherwise.