Evaluating AI agents requires systematically checking whether the agent is accurate, helpful, safe, relevant, and efficient enough for real business use, before and after it goes live. Databricks provides this natively through managed MLflow. This post is a follow-up to my previous blog: “How Do You Know Your AI Agent is Working”.
Why a dedicated evaluation process is needed: AI agents produce varied, non-exact answers, and traditional software tests that rely on exact matches are therefore inadequate. What is needed is an evaluation-driven loop:
- Define what “good” looks like for your business.
- Test against a representative set of examples.
- Score the results automatically wherever possible.
- Bring in human experts for calibration.
- Improve the agent, re-test, and then monitor in production.
Working through this loop reduces risk, builds business trust in the agent, and makes continuous improvement practical. Databricks supports offline evaluation during development as well as online monitoring in production, leveraging consistent scoring tools.
How Databricks handles it: MLflow records what the agent did at every step of a request, applies automated graders to that record, routes the cases that require human judgment to business experts through a review interface, and then runs those same checks against sampled live traffic once the agent is deployed. Traces, evaluation datasets and expert feedback all reside in Unity Catalog, alongside the data the agent reads. As a result, the evidence of agent quality is governed like any other data asset rather than sitting in a separate tool.
In the rest of this post, I give a phased task list for implementing agent evaluation in an enterprise. For each task I identify what the Business SME owns, what IT owns, and which Databricks/MLflow component does the work. The intent is something you can lift into a project plan.
The Components
Five building blocks are good to define upfront.
- Traces are detailed automatic logs of everything the agent did on a single request: the question it received, the documents it retrieved, the systems it queried, its intermediate reasoning, and the answer it produced. All other components are dependent on traces.
- Scorers attach a score and a written rationale to a trace. They come in four varieties. Built-in judges are ready-made graders for the dimensions most agents need, covering correctness against known answers, groundedness in retrieved sources, relevance to the question, retrieval quality, and safety. Guidelines judges turn plain-English business rules into automated checks. Custom judges handle domain-specific quality that no generic grader can assess. Code-based scorers are deterministic checks for rules that have known right answers, such as whether the correct system was queried or whether a figure matches the source record.
- Evaluation datasets are curated collections of realistic questions and tasks, with expected answers or key facts attached where they exist. They live as governed, versioned tables in Unity Catalog, alongside the data the agent reads.
- The Review App is a web interface where business experts read the agent’s answers, rate them, supply corrected expectations, and leave comments. No technical skill is required to use it.
- Metrics and monitoring aggregate scores into quality, cost, and latency figures, enforce the thresholds the business has agreed to, and run the identical set of scorers against a sample of production traffic after launch.
Who Owns What and Why It Matters
Defining the right ownership of activities is key to avoiding unfavorable outcomes. Business SMEs own the definition of quality. They live the process and therefore know what a correct and useful answer looks like, which questions the agent must refrain from answering, and what the company is willing to say in writing. IT owns instrumentation, automation, scorer construction, release gates, and production monitoring.
When IT writes the quality criteria, the agent scores well on generic tests and disappoints real users. When the business is handed raw evaluation output with no explanation, nobody trusts the numbers. Therefore, the task list below keeps both parties in their lane and defines the handoffs explicitly.
Phase 1 — Define What Good Looks Like
This phase produces written agreement on what is being measured and what score is good enough to deploy.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 1.1 | Inventory every data source and system the agent touches | Scoping only, no component yet | Validate. Confirms the data sources for the agent to use. | Lead. Documents each source, how the agent reaches it, and what the agent calls it. |
| 1.2 | Agree the quality dimensions to measure | Scorer selection mlflow.genai.scorers |
Own. Names the dimensions that matter, such as accuracy, completeness, tone, and refusal behavior. | Support. Maps each dimension to a scorer type and flag any that cannot be measured reliably. |
| 1.3 | Set the quality gates: which dimensions block a release, and the threshold each one must clear | Quality gates on metrics results.metrics[“<scorer>/mean”] |
Own. Decides that a safety failure stops a release while a tone failure does not and signs off that a minimum of 98 percent accuracy is acceptable. | Advise. Explains the reliability and cost tradeoff behind each check, and what each threshold implies for release frequency and rework. |
| 1.4 | Write the plain-language rules the agent must never break | Guidelines judge Guidelines(name=…, guidelines=…) |
Own. Writes the rules verbatim, such as never discussing pricing or discounts. | Implement. Converts each written rule into a Guidelines judge without changing the wording. |
Phase 2 — Instrument the Agent
IT owns this phase almost entirely. It is a pre-requisite for the rest of the phases.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 2.1 | Turn on tracing across the agent | MLflow Tracing @mlflow.trace, mlflow.<library>.autolog() |
None. | Own. Enables tracing so every request produces a complete step-by-step record. |
| 2.2 | Distinguish document retrieval from system queries in the trace | Span types SpanType.RETRIEVER, SpanType.TOOL |
None. | Own. Labels each step by type so graders can tell a document lookup from a database call. |
| 2.3 | Record who is asking on every request | Trace tags and attributes mlflow.update_current_trace(tags=…) |
Review. Confirms which caller details must be captured for later review. | Own. Attaches the requester identity to the trace so entitlement can be checked. |
| 2.4 | Store traces as governed data | Unity Catalog trace ingestion set_experiment_trace_location(UCSchemaLocation(…)) |
None. | Own. Links the experiment to a Unity Catalog schema so traces land in governed tables. |
| 2.5 | Confirm traces are readable before building anything on them | Experiments Traces view mlflow.search_traces() |
Review. Reads one complete trace and confirms it is understandable without a developer present. | Own. Verifies the step sequence renders correctly and fixes any gaps. |
Phase 3 — Build the Evaluation Sets
This phase has the most business commitment and is the phase that most often sets the schedule. Best practice is to build several focused suites rather than one consolidated set, because they have different owners, different pass thresholds, and different refresh cycles.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 3.1 | Seed the set from real questions | Traces as a dataset source mlflow.search_traces() |
Lead. Confirms the harvested questions reflect how people actually ask. | Support. Pulls real questions from logs and existing traces as starting examples. |
| 3.2 | Build the document and knowledge suite | Dataset expectations expectations.expected_facts |
Own. Supplies the key facts a correct answer must contain for each question. | Format. Structures those facts into dataset records. |
| 3.3 | Build the structured data suite | Expected values plus a code-based scorer expectations.expected_value, @scorer |
Review. Confirms the questions are ones customers genuinely ask about their orders. | Own. Computes ground-truth figures from a pinned version of the source data so expectations do not drift. |
| 3.4 | Build the adversarial and entitlement suite | Custom expectations read by a custom scorer @scorer over SpanType.RETRIEVER and SpanType.TOOL |
Review. Confirms which information must never surface, and to whom. | Own. Designs probes with Security, including attempts to retrieve another customer’s records. |
| 3.5 | Build the refusal and out-of-scope suite | Guidelines judges Guidelines(…), ExpectationsGuidelines() |
Own. With Legal, lists the questions the agent must decline and how it should decline them. | Format. Converts each refusal rule into an evaluation case. |
| 3.6 | Add multi-turn conversations | Conversation-format inputs mlflow.genai.evaluate(data=…) |
Own. Writes realistic follow-up exchanges that depend on what was said earlier. | Format. Structures the exchanges, so the agent is scored across the full conversation. |
| 3.7 | Publish each suite as a governed, versioned table | Managed evaluation datasets mlflow.genai.datasets.create_dataset(uc_table_name=…), .merge_records() |
None. | Own. Registers each suite in Unity Catalog so results stay comparable over time. |
Phase 4 — Score the Agent and Set a Baseline
This is when the first real measurement happens. Initial scores are expected to require adjustment to address inaccuracies.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 4.1 | Switch on the built-in judges | Built-in scorers Correctness(), RetrievalGroundedness(), RelevanceToQuery(), Safety() |
Review. Reads a sample of judge rationales and confirms the reasoning matches business expectations. | Own. Selects and configures the built-in judges that apply to this agent. |
| 4.2 | Build deterministic checks for exact rules | Code-based scorers @scorer returning Feedback |
Define. States which rules have one correct answer and no room for interpretation. | Own. Writes the checks, such as confirming the right system was queried. |
| 4.3 | Create a custom judge for domain quality | Custom LLM judge make_judge(…), .register() |
Own. Writes the scoring rubric describing what a 1 through 5 answer looks like. | Implement. Builds and registers the judge from that rubric. |
| 4.4 | Run each suite as a named baseline | Evaluation run mlflow.start_run(run_name=…), mlflow.genai.evaluate() |
None. | Own. Runs each suite under a named baseline so later versions can be compared against it. |
| 4.5 | Review failures together | Traces and evaluation results mlflow.search_traces(run_id=…) |
Lead. Reviews 15 to 20 failing cases and rules on whether each is a real failure or a flawed test. | Prepare. Groups failures by cause and walks the SME through representative examples. |
| 4.6 | Correct the test set and record the baseline | Dataset update and re-run dataset.merge_records(), mlflow.genai.evaluate() |
Approve. Confirms the corrected cases now reflect the right expectation. | Own. Updates the flawed cases, re-runs, and records the agreed baseline numbers. |
Task 4.5 deserves attention out of proportion to its size. It is where the business decides whether to trust the scoring at all.
Phase 5 — Calibrate the Judges Against Expert Judgment
Automated judges are capable but not automatically correct about your business. While deterministic checks don’t require calibration, this phase tunes the custom judge until its scores track what your experts would have said.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 5.1 | Select the responses experts will review | Trace tagging and labeling datasets mlflow.set_trace_tag(), create_dataset() |
Advise. Indicates which kinds of cases are most worth expert time. | Own. Assembles the review batch and loads it into a labeling dataset. |
| 5.2 | Set up the review form | Label schema matched to the judge label_schemas.create_label_schema() |
Review. Confirms the rating scale and instructions read clearly to a non-technical reviewer. | Own. Creates the review schema so expert ratings can be matched to judge scores. |
| 5.3 | Run the labeling session | Review App and labeling sessions mlflow.genai.create_labeling_session() |
Own. Rates 100 to 150 responses against the rubric and comments on what is wrong. | Facilitate. Creates the session, assigns reviewers, and distributes the link. |
| 5.4 | Align the judge to expert ratings | Judge alignment get_scorer(name=…).align(optimizer=MemAlignOptimizer(…)) |
None. | Own. Runs alignment so the judge absorbs the patterns in the expert feedback. |
| 5.5 | Re-run and confirm agreement | Evaluation run and Review App mlflow.genai.evaluate() |
Verify. Spot-checks 20 held-out responses and confirms the aligned judge now agrees with expert opinion. | Own. Re-runs the baseline with the aligned judge and reports the change. |
It’s worth setting business expectations before this phase starts since the aligned judge will usually report lower scores than it did beforehand. The agent has not declined but the grading has simply become more accurate.
Phase 6 — Improve, Launch, and Monitor
Evaluation starts prior to agent deployment and continues beyond launch as it becomes part of how the agent is released and operated.
| # | Task | Databricks / MLflow Component and Function | Business SME | IT |
| 6.1 | Sort failures by cause and route them | Trace search and analysis mlflow.search_traces(filter_string=…) |
Prioritize. Says which failure categories hurt the business most and should be fixed first. | Own. Groups failures by root cause and assign each to retrieval, permissions, or instructions. |
| 6.2 | Fix one layer at a time | Evaluation run comparison mlflow.search_runs() |
None. | Own. Makes a single change, re-runs the affected suite, and compares against the baseline. |
| 6.3 | Fix entitlement problems at the data layer | Unity Catalog permissions GRANT, row filters, index filters |
Review. Confirms the corrected access rules match business policy. | Own. Resolves access failures through governed permissions rather than through agent instructions. |
| 6.4 | Promote changes only on evidence | Prompt registry and aliases mlflow.genai.register_prompt(), mlflow.genai.set_prompt_alias() |
Approve. Signs off on promoting a version once the scores support it. | Execute. Registers the new version and promotes it only when the score improved. |
| 6.5 | Make evaluation a release gate | Evaluation in a scheduled job mlflow.genai.evaluate() checked against results.metrics |
Own. Confirms which suites must pass on every change and which can run nightly. | Own. Automates the run and fails the release when a blocking threshold is missed. |
| 6.6 | Score live traffic with the same checks | Production monitoring scorer.register(), .start(ScorerSamplingConfig(sample_rate=…)) |
Agree. Accepts the coverage and cost tradeoff, such as sampling every conversation for safety and a subset for quality. | Own. Registers and starts the scorers against sampled production traffic, after set_databricks_monitoring_sql_warehouse_id(). |
| 6.7 | Review the weakest production cases weekly | AI/BI dashboard over UC trace tables, plus the Review App | Own. Spends about an hour a week on the lowest-scoring real conversations. | Prepare. Builds the dashboard and queues the cases for review. |
| 6.8 | Feed what you learn back into the test sets | Evaluation dataset update dataset.merge_records() |
Own. Adds newly discovered failure patterns as permanent test cases. | Own. Merges them into the governed dataset and re-baselines. |
| 6.9 | Revisit the thresholds quarterly | Quality gate review, no component | Own. Reconsiders whether the agreed pass marks still reflect business risk. | Support. Reports where thresholds are consistently exceeded or missed. |
What Actually Governs the Schedule
The first implementation along these lines typically runs seven to eight weeks. The distribution is the part worth planning around.
Phases 1 and 2 take about a week and are mostly IT. Phase 3 takes two to three weeks and is almost entirely business time. Phase 4 takes a week. Phase 5 takes a week, however long the experts need to finish labeling. Phase 6 takes about two weeks to stand up and continues as long as the agent is in use.
The second thing worth budgeting for is expert attention after launch. An hour a week of informed review, fed back into the test set, is what keeps quality from drifting as information and user behavior change.
If you are planning an agent deployment and want help setting up the evaluation framework around it, reach out to Perficient for a discussion with our AI and Databricks specialists.
