Building Trustworthy Production RAG Systems Through Continuous Evaluation

then you’ve already been through this situation when it looks like your RAG system is working fine but it is still wrong. The retrieval fetches some chunks, passes it to the generative model, and it writes a fluent answer. Nothing throws an error, and yet the answer might be built on the wrong document, or missing half the information it needed, or grounded in something that’s technically true but three versions out of date. The only way to actually know and mitigate this is to evaluate it properly on a regular basis.

This article walks through how to build that evaluation pipeline yourself, starting with the parts that feel too basic to write about, through automated evaluation with RAGAS, past the point where RAGAS stops being enough, and into what it takes to run this as an actual process rather than a notebook you open before a stakeholder meeting. The steps are written so you can follow them for your own RAG application, not just read them as theory.

This is part 4 of The RAG for Enterprise series, and if you have missed the earlier parts I would strongly recommend you to check out the part 3 here: Hybrid Search and Re-Ranking in Production RAG

What’s in this article

1. Building the Golden Dataset
2. Doing a simple manual pass check before automating
3. Automate scoring with RAGAS
4. Adding a custom LLM judge for what RAGAS can’t check
5. Using a human in the loop
6. Watch for drift after shipping the system
7. Running it as a pipeline
8. Conclusion

Building the Golden Dataset

Before touching any evaluation library, you need a set of questions with known correct answers. This is called a golden dataset, and skipping it is the most common mistake teams make while working on a RAG system. Jumping straight to running RAGAS on random queries with nothing to compare against tells you almost nothing about whether the system is actually right or wrong.

A good golden dataset entry has three parts: the question, a correct answer written by someone who knows the domain, and which document contains the answer. The third field is what lets you tell a retrieval failure (wrong chunk fetched) apart from a generation failure (right chunk, wrong answer written from it). These are different bugs with different fixes, and a dataset without this field will leave you debugging blind.

golden_set = [
    {
        "question": "What is the maximum file size for uploads?",
        "ground_truth": "25 MB per file on the free plan, 200 MB on paid plans.",
        "source_doc": "upload_limits.md",
        "category": "single_fact",
    },
    {
        "question": "Can I cancel my subscription mid-cycle and get a refund?",
        "ground_truth": "No, cancellations take effect at the end of the billing cycle. No partial refunds.",
        "source_doc": "billing_policy.md",
        "category": "single_fact",
    },
]

Twenty to thirty questions is enough to get real signal at the start. The category field matters more than the count of entries, and this is usually where teams underinvest the most. A dataset made only of easy factual lookups will pass everything and tell you nothing, because easy factual lookups are the one failure mode RAG systems rarely have. The categories worth adding, in the order they tend to actually bite in the production:

• Multi-hop – the answer needs two or more chunks combined, which is where retrieval quietly under-fetches
• No answer expected – the correct behavior is refusing to answer, not guessing from the nearest semantically similar chunk
• Conflicting or stale documents – an old and a current version of the same policy exist in the corpus, and only one is right
• Adversarial phrasing – the same question asked with different terminology than the source document uses

The third category “Conflicting or stale documents” is the most skipped category in the golden sets, and it’s also the one that tends to produce more production incidents about an answer that was fluent, well-cited, but built entirely on a document nobody had gotten around to archiving. If your corpus accumulates old versions of things (most enterprise document stores do), your eval set needs to test for it, or your pipeline will be just as blind to it as your reviewers were.

Doing a simple manual pass check before automating

Run the golden set through your pipeline and read the answers next to the ground truth before setting up any scoring tool. This step gets skipped constantly because it feels too basic to be real engineering work but it tells you whether your dataset itself is sound, and gives you a feel for what “wrong” looks like in your specific system before you trust a metric to catch it for you.

results = []
for item in golden_set:
    answer = rag_pipeline.query(item["question"])
    results.append({
        "question": item["question"],
        "generated_answer": answer,
        "ground_truth": item["ground_truth"],
        "correct": None,  # fill in manually
    })

This catches the embarrassing failures early like a broken prompt template, a retriever returning nothing, a model ignoring context entirely, etc. Automating the scoring of a broken pipeline just gives you a precise number for something that was never going to be useful.

Automate scoring with RAGAS

Once the basics hold up, RAGAS gives you a fast, repeatable way to score the pipeline on four dimensions without reading every answer by hand.

from ragas import evaluate
from ragas.metrics import (
    context_precision,
    context_recall,
    faithfulness,
    answer_relevancy,
)
from datasets import Dataset

eval_dataset = Dataset.from_list([
    {
        "question": item["question"],
        "answer": rag_pipeline.query(item["question"]),
        "contexts": rag_pipeline.retrieve(item["question"]),
        "ground_truth": item["ground_truth"],
    }
    for item in golden_set
])

result = evaluate(
    eval_dataset,
    metrics=[context_precision, context_recall, faithfulness, answer_relevancy],
)
print(result)

• Context precision – of the retrieved chunks, how many were actually relevant, penalising noise
• Context recall – estimates whether retrieval captured the information needed to produce the reference answer
• Faithfulness – is the answer grounded in what was retrieved, catching hallucination
• Answer relevancy – does the answer address the actual question, not a nearby one

A typical run looks something like this (the numbers are for explaining each metrics):

A high faithfulness score gets read as “the answer is correct,” and that’s the exact misreading that lets bad answers through. Faithfulness only checks whether the answer is supported by the retrieved context or not, it doesn’t tell if that context was the right thing to retrieve. A stale or outdated document, faithfully summarised, produces an answer that’s fully grounded and fully wrong at the same time. This is the structural ceiling of RAGAS: it’s very good at catching hallucination and very bad at catching a confidently wrong source.

Adding a custom LLM judge for what RAGAS can’t check

Most RAGAS metrics rely on LLM-based evaluation using predefined prompts that know nothing about your domain. If correctness for you depends on something specific like exact figures, recency of the source, a required disclaimer, tone, etc. the default prompts won’t catch it, because it was never asked to do so.

To resolve this problem you can use an LLM as a judge and pass a custom prompt to it which tells it to judge the answers based on your specific needs.

import json
from google import genai
from google.genai.types import GenerateContentConfig

client = genai.Client(
    vertexai=True,
    project="YOUR_GCP_PROJECT_ID",
    location="us-central1",
)

JUDGE_PROMPT = """
Compare the generated answer to the ground truth. Score 1-5 on each dimension.

Question: {question}
Ground Truth: {ground_truth}
Generated Answer: {answer}
Source Document Date: {doc_date}

1. numeric_accuracy - are all numbers and facts correct, not just plausible?
2. recency_awareness - if the source is outdated, does the answer flag
   uncertainty instead of stating it as current fact?

Return JSON only:
{{
    "numeric_accuracy": int,
    "recency_awareness": int,
    "reasoning": str
}}
"""

def custom_judge(question, ground_truth, answer, doc_date, client):
    prompt = JUDGE_PROMPT.format(
        question=question,
        ground_truth=ground_truth,
        answer=answer,
        doc_date=doc_date,
    )

    response = client.models.generate_content(
        model="gemini-2.5-flash",
        contents=prompt,
        config=GenerateContentConfig(
            temperature=0,
            max_output_tokens=250,
            response_mime_type="application/json",
        ),
    )

    return json.loads(response.text)

Don’t run this on the full dataset every time because it’s slower and costs more per call than RAGAS. Scope it to the categories where RAGAS evaluation is actually limited, which in practice means conflicting_docs and no_answer_expected. Running it everywhere else is paying for precision you don’t need on questions RAGAS already scores reliably.

Using a human in the loop

Both RAGAS and a custom judge are still LLMs scoring another LLM’s output, and they won’t always agree with a human. The honest number to know here, which most teams never measure, is how often your judge actually agrees with a person. In practice, LLM-judge-to-human agreement in the low-to-mid 80% range is common, not a sign something is broken, but a real ceiling is worth knowing rather than assuming.

def needs_human_review(ragas_score, judge_score, threshold=1.0):
    return abs(ragas_score - judge_score) > threshold

This keeps the human review queue small and targeted, only the cases where two scoring methods disagree, not every answer. It’s also worth occasionally having two people score the same handful of borderline answers independently. If domain experts disagree with each other close to a third of the time on a given question type, that’s not a scoring pipeline problem the ground truth itself is ambiguous, and no amount of tooling fixes that. It usually means the golden set entry needs to be rewritten, not the judge.

Watch for drift after shipping the system

A pipeline that only runs against a fixed golden set has a blind spot because the live corpus changes underneath it. New documents get added, old ones get archived, and real user queries drift away from whatever you originally tested against. None of that shows up in a dataset that was frozen the day you wrote it.

import random
from datetime import datetime, timedelta

def sample_production_queries(logs, n=50, days=7):
    recent = [q for q in logs if q["timestamp"] > datetime.now() - timedelta(days=days)]
    return random.sample(recent, min(n, len(recent)))

Sampling a small slice of live traffic weekly and running it through RAGAS’s context precision and faithfulness, both of these metrics work without a ground truth answer, gives you a second signal. A sudden drop with no corresponding code change usually means something changed in the corpus, not the pipeline, and it’s a different failure mode than anything a merge-time check will ever catch.

Running it as a pipeline

Two things make it a real pipeline instead of a one-off exercise: cost-aware scheduling, and a CI gate.

Running the full stack: RAGAS, the custom judge, and human review on every single commit is expensive enough that most teams quietly stop doing it within a month. Tiering it works better in practice:

• RAGAS on the full golden set: every pull request touching retrieval or prompts
• Custom judge on flagged categories only: every pull request, scoped to a handful of examples
• Human review: weekly, on the disagreement queue only
• Production sampling: weekly, on a rotating slice of live traffic

def check_regression(current_scores, baseline_scores, threshold=0.03):
    regressions = [
        (metric, baseline_scores[metric], score)
        for metric, score in current_scores.items()
        if baseline_scores[metric] - score > threshold
    ]
    if regressions:
        raise SystemExit(f"Blocking merge — regressions found: {regressions}")
    print("No regressions. Safe to merge.")

Wire this into CI so it runs on every pull request touching retrieval, chunking, or prompts, and let it fail the build if any metric drops past your threshold. This is the most important part of the whole setup, and it’s also the one that actually prevents incidents. For example, a chunking change that quietly breaks a class of queries gets caught here, before it becomes a support ticket three weeks later instead of a failed check today.

Conclusion

None of the above six steps is impressive on its own when used independently, but what makes the difference is running them together, on a schedule, as something the team trusts rather than something someone remembers once in a while. That’s the intent of this article, from running evaluation as a one-time gut check to making it a part of the RAG system infrastructure, sitting quietly in CI and catching the change that would otherwise have shipped.

If you’re starting from nothing, don’t try to build all six steps at once. A golden dataset with twenty good questions and a RAGAS score you check by hand is already ahead of most RAG systems in production today. The rest can be added as the system and your patience for reading answers by hand grows.