Plain-language summary: large language model applications are hard to test because their outputs are free-form text rather than fixed values. This tutorial shows how to bring unit-testing discipline to a retrieval-augmented generation (RAG) system using DeepEval, an open-source framework that scores model outputs with “LLM-as-a-judge” metrics. The full code is available in the linked repository.
This tutorial configures an evaluation environment focused on integrating the DeepEval framework to bring unit-testing rigor to LLM applications. Bridging the gap between raw retrieval and final generation, it builds a system that treats model outputs as testable artifacts and uses LLM-as-a-judge metrics to measure performance. Instead of relying on manual inspection, it constructs a structured pipeline in which each question, its retrieved context, and the generated response are validated against rigorous, reproducible metrics.
import sys, os, textwrap, json, math, re
from getpass import getpass
print("🔧 Hardening environment (prevents common Colab/py3.12 numpy corruption)...")
!pip -q uninstall -y numpy || true
!pip -q install --no-cache-dir --force-reinstall "numpy==1.26.4"
!pip -q install -U deepeval openai scikit-learn pandas tqdm
print("✅ Packages installed.")
import numpy as np
import pandas as pd
from tqdm.auto import tqdm
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity
from deepeval import evaluate
from deepeval.test_case import LLMTestCase, LLMTestCaseParams
from deepeval.metrics import (
AnswerRelevancyMetric,
FaithfulnessMetric,
ContextualRelevancyMetric,
ContextualPrecisionMetric,
ContextualRecallMetric,
GEval,
)
print("✅ Imports loaded successfully.")
OPENAI_API_KEY = getpass("🔑 Enter OPENAI_API_KEY (leave empty to run without OpenAI): ").strip()
openai_enabled = bool(OPENAI_API_KEY)
if openai_enabled:
os.environ("OPENAI_API_KEY") = OPENAI_API_KEY
print(f"🔌 OpenAI enabled: {openai_enabled}")The environment is initialized by pinning key dependencies and installing the DeepEval framework to ensure a stable testing pipeline. Specialized metrics such as Faithfulness and Contextual Recall are imported alongside the API credentials needed to enable automated, high-fidelity evaluation of the system’s outputs.
DOCS = (
{
"id": "doc_01",
"title": "DeepEval Overview",
"text": (
"DeepEval is an open-source LLM evaluation framework for unit testing LLM apps. "
"It supports LLM-as-a-judge metrics, custom metrics like G-Eval, and RAG metrics "
"such as contextual precision and faithfulness."
),
},
{
"id": "doc_02",
"title": "RAG Evaluation: Why Faithfulness Matters",
"text": (
"Faithfulness checks whether the answer is supported by retrieved context. "
"In RAG, hallucinations occur when the model states claims not grounded in context."
),
},
{
"id": "doc_03",
"title": "Contextual Precision",
"text": (
"Contextual precision evaluates how well retrieved chunks are ranked by relevance "
"to a query. High precision means relevant chunks appear earlier in the ranked list."
),
},
{
"id": "doc_04",
"title": "Contextual Recall",
"text": (
"Contextual recall measures whether the retriever returns enough relevant context "
"to answer the query. Low recall means key information was missed in retrieval."
),
},
{
"id": "doc_05",
"title": "Answer Relevancy",
"text": (
"Answer relevancy measures whether the generated answer addresses the user's query. "
"Even grounded answers can be irrelevant if they don't respond to the question."
),
},
{
"id": "doc_06",
"title": "G-Eval (GEval) Custom Rubrics",
"text": (
"G-Eval lets you define evaluation criteria in natural language. "
"It uses an LLM judge to score outputs against your rubric (e.g., correctness, tone, policy)."
),
},
{
"id": "doc_07",
"title": "What a DeepEval Test Case Contains",
"text": (
"A test case typically includes input (query), actual_output (model answer), "
"expected_output (gold answer), and retrieval_context (ranked retrieved passages) for RAG."
),
},
{
"id": "doc_08",
"title": "Common Pitfall: Missing expected_output",
"text": (
"Some RAG metrics require expected_output in addition to input and retrieval_context. "
"If expected_output is None, evaluation fails for metrics like contextual precision/recall."
),
},
)
EVAL_QUERIES = (
{
"query": "What is DeepEval used for?",
"expected": "DeepEval is used to evaluate and unit test LLM applications using metrics like LLM-as-a-judge, G-Eval, and RAG metrics.",
},
{
"query": "What does faithfulness measure in a RAG system?",
"expected": "Faithfulness measures whether the generated answer is supported by the retrieved context and avoids hallucinations not grounded in that context.",
},
{
"query": "What does contextual precision mean?",
"expected": "Contextual precision evaluates whether relevant retrieved chunks are ranked higher than irrelevant ones for a given query.",
},
{
"query": "What does contextual recall mean in retrieval?",
"expected": "Contextual recall measures whether the retriever returns enough relevant context to answer the query, capturing key missing information issues.",
},
{
"query": "Why might an answer be relevant but still low quality in RAG?",
"expected": "An answer can address the question (relevant) but still be low quality if it is not grounded in retrieved context or misses important details.",
},
)
A structured knowledge base holds documentation snippets that serve as ground-truth reference material for the RAG system. A set of evaluation questions and their expected outputs forms a “gold dataset,” which makes it possible to assess how accurately the model retrieves information and generates grounded responses.
class TfidfRetriever:
def __init__(self, docs):
self.docs = docs
self.texts = (f"{d('title')}n{d('text')}" for d in docs)
self.vectorizer = TfidfVectorizer(stop_words="english", ngram_range=(1, 2))
self.matrix = self.vectorizer.fit_transform(self.texts)
def retrieve(self, query, k=4):
qv = self.vectorizer.transform((query))
sims = cosine_similarity(qv, self.matrix).flatten()
top_idx = np.argsort(-sims)(:k)
results = ()
for i in top_idx:
results.append(
{
"id": self.docs(i)("id"),
"score": float(sims(i)),
"text": self.texts(i),
}
)
return results
retriever = TfidfRetriever(DOCS)A custom TF-IDF retriever class transforms the documents into a searchable vector space using bigram-aware TF-IDF vectorization. This supports cosine-similarity search against the knowledge base, so the top-k most relevant text fragments can be retrieved programmatically for any query.
def extractive_baseline_answer(query, retrieved_contexts):
"""
Offline fallback: we create a short answer by extracting the most relevant sentences.
This keeps the notebook runnable even without OpenAI.
"""
joined = "n".join(retrieved_contexts)
sents = re.split(r"(?<=(.!?))s+", joined)
keywords = (w.lower() for w in re.findall(r"(a-zA-Z){4,}", query))
scored = ()
for s in sents:
s_l = s.lower()
score = sum(1 for k in keywords if k in s_l)
if len(s.strip()) > 20:
scored.append((score, s.strip()))
scored.sort(key=lambda x: (-x(0), -len(x(1))))
best = (s for sc, s in scored(:3) if sc > 0)
if not best:
best = (s.strip() for s in sents(:2) if len(s.strip()) > 20)
ans = " ".join(best).strip()
if not ans:
ans = "I could not find enough context to answer confidently."
return ans
def openai_answer(query, retrieved_contexts, model="gpt-4.1-mini"):
"""
Simple RAG prompt for demonstration. DeepEval metrics can still evaluate even if
your generation prompt differs; the key is we store retrieval_context separately.
"""
from openai import OpenAI
client = OpenAI()
context_block = "nn".join((f"(CTX {i+1})n{c}" for i, c in enumerate(retrieved_contexts)))
prompt = f"""You are a concise technical assistant.
Use ONLY the provided context to answer the query. If the answer is not in context, say you don't know.
Query:
{query}
Context:
{context_block}
Answer:"""
resp = client.chat.completions.create(
model=model,
messages=({"role": "user", "content": prompt}),
temperature=0.2,
)
return resp.choices(0).message.content.strip()
def rag_answer(query, retrieved_contexts):
if openai_enabled:
try:
return openai_answer(query, retrieved_contexts)
except Exception as e:
print(f"⚠️ OpenAI generation failed, falling back to extractive baseline. Error: {e}")
return extractive_baseline_answer(query, retrieved_contexts)
else:
return extractive_baseline_answer(query, retrieved_contexts)A hybrid answer mechanism prioritizes high-fidelity generation via an LLM while keeping a keyword-based extractive baseline as a reliable fallback. Separating the retrieval context from the final generation step means answers can be either synthesized by the model or extracted programmatically — a useful safeguard when the generative path is unavailable.
print("n🚀 Running RAG to create test cases...")
test_cases = ()
K = 4
for item in tqdm(EVAL_QUERIES):
q = item("query")
expected = item("expected")
retrieved = retriever.retrieve(q, k=K)
retrieval_context = (r("text") for r in retrieved)
actual = rag_answer(q, retrieval_context)
tc = LLMTestCase(
input=q,
actual_output=actual,
expected_output=expected,
retrieval_context=retrieval_context,
)
test_cases.append(tc)
print(f"✅ Built {len(test_cases)} LLMTestCase objects.")
print("n✅ Metrics configured.")
metrics = (
AnswerRelevancyMetric(threshold=0.5, model="gpt-4.1", include_reason=True, async_mode=True),
FaithfulnessMetric(threshold=0.5, model="gpt-4.1", include_reason=True, async_mode=True),
ContextualRelevancyMetric(threshold=0.5, model="gpt-4.1", include_reason=True, async_mode=True),
ContextualPrecisionMetric(threshold=0.5, model="gpt-4.1", include_reason=True, async_mode=True),
ContextualRecallMetric(threshold=0.5, model="gpt-4.1", include_reason=True, async_mode=True),
GEval(
name="RAG Correctness Rubric (GEval)",
criteria=(
"Score the answer for correctness and usefulness. "
"The answer must directly address the query, must not invent facts not supported by context, "
"and should be concise but complete."
),
evaluation_params=(
LLMTestCaseParams.INPUT,
LLMTestCaseParams.ACTUAL_OUTPUT,
LLMTestCaseParams.EXPECTED_OUTPUT,
LLMTestCaseParams.RETRIEVAL_CONTEXT,
),
model="gpt-4.1",
threshold=0.5,
async_mode=True,
),
)
if not openai_enabled:
print("n⚠️ You did NOT provide an OpenAI API key.")
print("DeepEval's LLM-as-a-judge metrics (AnswerRelevancy/Faithfulness/Contextual* and GEval) require an LLM judge.")
print("Re-run this cell and provide OPENAI_API_KEY to run DeepEval metrics.")
print("n✅ However, your RAG pipeline + test case construction succeeded end-to-end.")
rows = ()
for i, tc in enumerate(test_cases):
rows.append({
"id": i,
"query": tc.input,
"actual_output": tc.actual_output(:220) + ("..." if len(tc.actual_output) > 220 else ""),
"expected_output": tc.expected_output(:220) + ("..." if len(tc.expected_output) > 220 else ""),
"contexts": len(tc.retrieval_context or ()),
})
display(pd.DataFrame(rows))
raise SystemExit("Stopped before evaluation (no OpenAI key).")Running the RAG pipeline produces an LLMTestCase object that combines the retrieved context with the model-generated answer and the ground-truth expectation. A comprehensive suite of DeepEval metrics — including G-Eval and RAG-specific indicators — then evaluates system performance using the LLM-as-a-judge approach.
print("n🧪 Running DeepEval evaluate(...) ...")
results = evaluate(test_cases=test_cases, metrics=metrics)
summary_rows = ()
for idx, tc in enumerate(test_cases):
row = {
"case_id": idx,
"query": tc.input,
"actual_output": tc.actual_output(:200) + ("..." if len(tc.actual_output) > 200 else ""),
}
for m in metrics:
row(m.__class__.__name__ if hasattr(m, "__class__") else str(m)) = None
summary_rows.append(row)
def try_extract_case_metrics(results_obj):
extracted = ()
candidates = ()
for attr in ("test_results", "results", "evaluations"):
if hasattr(results_obj, attr):
candidates = getattr(results_obj, attr)
break
if not candidates and isinstance(results_obj, list):
candidates = results_obj
for case_i, case_result in enumerate(candidates or ()):
item = {"case_id": case_i}
metrics_list = None
for attr in ("metrics_data", "metrics", "metric_results"):
if hasattr(case_result, attr):
metrics_list = getattr(case_result, attr)
break
if isinstance(metrics_list, dict):
for k, v in metrics_list.items():
item(f"{k}_score") = getattr(v, "score", None) if v is not None else None
item(f"{k}_reason") = getattr(v, "reason", None) if v is not None else None
else:
for mr in metrics_list or ():
name = getattr(mr, "name", None) or getattr(getattr(mr, "metric", None), "name", None)
if not name:
name = mr.__class__.__name__
item(f"{name}_score") = getattr(mr, "score", None)
item(f"{name}_reason") = getattr(mr, "reason", None)
extracted.append(item)
return extracted
case_metrics = try_extract_case_metrics(results)
df_base = pd.DataFrame(({
"case_id": i,
"query": tc.input,
"actual_output": tc.actual_output,
"expected_output": tc.expected_output,
} for i, tc in enumerate(test_cases)))
df_metrics = pd.DataFrame(case_metrics) if case_metrics else pd.DataFrame(())
df = df_base.merge(df_metrics, on="case_id", how="left")
score_cols = (c for c in df.columns if c.endswith("_score"))
compact = df(("case_id", "query") + score_cols).copy()
print("n📊 Compact score table:")
display(compact)
print("n🧾 Full details (includes reasons):")
display(df)
print("n✅ Done. Tip: if contextual precision/recall are low, improve retriever ranking/coverage; if faithfulness is low, tighten generation to only use context.")The final step executes the evaluation function, which triggers the LLM-as-a-judge process to score each test case against the defined metrics. The scores and their accompanying qualitative rationales are aggregated into a single dataframe, revealing where the RAG pipeline performs well and where retrieval or generation needs further work.
In sum, DeepEval turns complex textual outputs into actionable data using metrics such as Faithfulness, Contextual Recall and a G-Eval rubric. This systematic approach helps diagnose “silent failures” in retrieval and hallucinations in generation, and provides the rationale needed to justify architectural changes — moving a project from an experimental prototype toward a production-ready RAG system backed by a verifiable, metric-driven safety net.
Limitations and what to watch
LLM-as-a-judge evaluation is powerful but not infallible. Because the judge is itself a language model, its scores carry the biases and occasional errors of that model, and results can vary between runs unless prompts, model versions and temperature are held fixed; DeepEval’s G-Eval mitigates but does not eliminate this through techniques such as chain-of-thought scoring and token-weight summation. The quality of any evaluation also depends heavily on the gold dataset — a small or unrepresentative set of questions will give a misleadingly clean picture. Metric scores should be read as directional signals rather than absolute truths, ideally cross-checked against periodic human review and tracked over time as the underlying models and data change. Running these evaluations also incurs API cost and latency, since each metric typically calls a judge model per test case.
The bottom line
Treating LLM outputs as testable artifacts, rather than inspecting them by hand, is what separates a fragile demo from a system that can be maintained and improved. Frameworks like DeepEval make that discipline practical — provided the metrics are understood as approximate, reproducible signals rather than perfect verdicts.
