Evaluation that gates releases

Measure first, then improve

Agents fail in ways that demos hide. A demo shows five good runs. Production sends five thousand, and the sixty that go wrong are the ones your client remembers. An evaluation set is how you find those sixty before the client does.

At NexOps, we do not start an agent build with the model. We start with a set of real cases and a definition of "good". The model, the prompt and the tools are then variables that we change and measure. In this lesson we build that measurement: the set, the scorers, the check on the judges, and the regression gate.

Step 1: build the evaluation set from real cases

An evaluation set is a list of inputs, and for many of them the expected outcome. Start small and real.

• Collect 30 to 50 real tasks from the people who do the work today: tickets, requests, documents. Real cases hold the odd phrasing, the missing field and the angry customer that invented cases lack.
• Cover the edges. Add cases where the right answer is "I cannot do this", cases with missing data, and cases that try to push the agent off its task.
• Write expectations. For each case, write what a correct result contains: facts that must appear, tools that must be called, actions that must not happen.
• Keep a held-out part. Do not tune on every case. Keep a share aside to check that your improvements generalise.

On Databricks, MLflow evaluation datasets are stored in Unity Catalog and attached to an MLflow experiment. You need CREATE TABLE on the target schema. You can add records from captured traces or write them yourself.

import mlflow
import mlflow.genai.datasets
 
# Create a dataset in Unity Catalog (name format: catalog.schema.table).
eval_dataset = mlflow.genai.datasets.create_dataset(
    name="workspace.default.support_agent_eval",
)
 
# Add real traces from production that finished without an error.
traces = mlflow.search_traces(
    filter_string="attributes.status = 'OK' AND tags.environment = 'production'",
    max_results=10,
)
eval_dataset = eval_dataset.merge_records(traces)
 
# Add a labelled case with a known expected outcome.
# The docs use both "expected" and "expectations" as key names: check your MLflow version.
eval_dataset = eval_dataset.merge_records([
    {
        "inputs": {"question": "Where is order SO-100234?"},
        "expected": {"expected_facts": ["order status", "delivery date"]},
    },
])

The documentation lists limits of 2,000 rows per dataset and 20 expectations per record, so a focused set is a better fit than a huge one anyway.

Step 2: choose scorers

A scorer turns a trace into a verdict. There are two families.

MLflow ships built-in judges such as Safety, RetrievalGroundedness, Correctness (which needs expectations), ToolCallCorrectness and ToolCallEfficiency. It also offers Guidelines, which checks a response against criteria you write in plain language, and custom judges and code scorers in Python.

Prefer a rule over a judge wherever you can. A rule never has a bad day.

Step 3: check the judges against people

An LLM judge is a model, and models are wrong. Before you trust a judge, measure how often it agrees with a person.

1. Take 30 cases and have a domain expert label each one pass or fail.
2. Run the judge on the same cases.
3. Compute the agreement and read every disagreement.

If agreement is low, the judge instructions are unclear, or the criterion is too vague for anyone to apply. Rewrite the criterion until two people and the judge agree. The Databricks documentation describes aligning judges with human feedback to improve accuracy for your domain. Treat it as a tool for this step, and still keep a sample of human review in operation.

Step 4: run it and gate on it

mlflow.genai.evaluate() runs your scorers over a dataset and records the results so you can compare versions.

# Schematic: argument names may differ between MLflow versions. Check the current docs.
import mlflow
from mlflow.genai.scorers import Safety, Guidelines
 
def predict(question: str) -> str:
    return my_agent.answer(question)   # your agent, called once per case
 
results = mlflow.genai.evaluate(
    data=eval_dataset,
    predict_fn=predict,
    scorers=[
        Safety(),
        Guidelines(name="polite_and_short", guidelines="Reply in at most 120 words and stay polite."),
    ],
)

Then turn the result into a regression gate in your pipeline. A gate is a rule that stops a release:

• Pass rate on the evaluation set must not drop against the current production version.
• No case in the safety group may fail.
• Cost per task must stay within the budget (lesson 6).

The same scorers can run on a sample of production traffic, so the quality you measured before release keeps being measured after it. Every production failure you find becomes a new case in the dataset. That loop is how the set gets better.

Common mistakes

• Testing on the cases you tuned on. Keep a held-out part.
• One overall score. Report per group, so a safety failure cannot hide behind a high average.
• Trusting an unchecked judge. Measure agreement with people first.
• Never adding cases. Each incident is a free test case.

Summary

Build the set from real cases, write expectations, prefer rules to judges, check the judges against people and gate every release on the result. Next lesson: guardrails and safety, where the evaluation set gets its hardest cases.