Build an Eval Suite
This notebook builds an evaluation suite for a small support agent. You define three cases with stable IDs, score each run for task success and for correct escalation, and print a per-case report that shows coverage and failed rows. Then you run the same cases against a second agent configuration. The agent is plain Elixir code with fixed replies, so the notebook runs offline with no credentials and no model calls.
Setup
Mix.install([
{:tune, "~> 0.5.0"}
])
Define the agent
The agent takes a configuration and one ticket. It escalates the ticket types
listed in config.escalate and resolves the rest. A ticket for the
acme-outage account returns an error, the way a real agent run fails when a
backend times out.
defmodule SupportAgent do
@moduledoc "A support agent with fixed replies for offline evaluation."
def run(_config, %{account: "acme-outage"}), do: {:error, :inventory_service_timeout}
def run(config, %{ticket: ticket}) do
if ticket in config.escalate do
%{action: :escalate, reply: "Escalating to payments"}
else
%{action: :resolve, reply: reply(ticket)}
end
end
defp reply(:password_reset), do: "Reset link sent"
defp reply(:order_status), do: "Order shipped"
defp reply(_ticket), do: "Ticket closed"
end
A runner can return a plain value, {:ok, output}, or {:error, reason}.
Tune records an error as a failed row and keeps evaluating the other cases.
Write the cases
Give every case an id. Tune uses it to name observations, and paired
comparisons use it to match rows. Keep IDs stable so reports from different
runs line up.
cases = [
%{
id: "password-reset",
input: %{ticket: :password_reset, account: "north-bakery"},
expected: %{action: :resolve, reply: "Reset link sent"}
},
%{
id: "chargeback",
input: %{ticket: :chargeback, account: "lumen-books"},
expected: %{action: :escalate, reply: "Escalating to payments"}
},
%{
id: "order-status",
input: %{ticket: :order_status, account: "acme-outage"},
expected: %{action: :resolve, reply: "Order shipped"}
}
]
The first case is a routine request. The second needs an escalation. The third fails because the agent run returns an error.
Build the suite
Tune.new/2 takes the configuration to evaluate and a run: function.
Tune.data/2 adds the cases as the test split. Each Tune.score/4 call adds
one named metric. A two-argument callback receives the case and the output.
task_success = fn tune_case, output ->
if output.reply == tune_case.expected.reply, do: 1, else: 0
end
correct_escalation = fn tune_case, output ->
if output.action == tune_case.expected.action, do: 1, else: 0
end
suite = fn config ->
Tune.new(config, run: &SupportAgent.run/2)
|> Tune.data(test: cases)
|> Tune.score(:task_success, task_success)
|> Tune.score(:correct_escalation, correct_escalation)
end
The builder calls only return a %Tune{} value. Nothing runs until you call
Tune.evaluate/2.
Run the evaluation
{:ok, evaluation} = Tune.evaluate(suite.(%{escalate: [:chargeback]}))
{evaluation.complete?, length(evaluation.observations)}
{true, 3}
Tune.evaluate/2 returns a Tune.Evaluation. It holds one Tune.Observation
per case, in dataset order, and one Tune.Score.Summary per score.
complete? is true because Tune attempted every case, even though one case
failed.
Print a per-case report
Write the report as an ordinary function so you can reuse it for every configuration.
defmodule SuiteReport do
@moduledoc "Prints observations, summaries, and failed rows for one evaluation."
def print(evaluation, metrics) do
Enum.each(evaluation.observations, &print_row(&1, metrics))
Enum.each(metrics, fn name ->
summary = Tune.Evaluation.summary(evaluation, name)
IO.puts(
"#{name}: mean #{summary.mean}, scored #{summary.count} of " <>
"#{length(evaluation.observations)}, coverage #{Float.round(summary.coverage, 2)}"
)
end)
failed = for %{status: :error} = row <- evaluation.observations, do: {row.id, row.error}
IO.puts("failed rows: #{inspect(failed)}")
end
defp print_row(row, metrics) do
values = Enum.map_join(metrics, " ", &"#{&1}=#{Map.get(row.scores, &1, "-")}")
IO.puts(String.pad_trailing(row.id, 16) <> String.pad_trailing("#{row.status}", 7) <> values)
end
end
SuiteReport.print(evaluation, [:task_success, :correct_escalation])
password-reset ok task_success=1 correct_escalation=1
chargeback ok task_success=1 correct_escalation=1
order-status error task_success=- correct_escalation=-
task_success: mean 1.0, scored 2 of 3, coverage 0.67
correct_escalation: mean 1.0, scored 2 of 3, coverage 0.67
failed rows: [{"order-status", :inventory_service_timeout}]
Read the mean together with the coverage. Both metrics have a mean of 1.0,
but they cover only two of the three cases. The failed row has no scores, and
its error field keeps the reason the runner returned.
Test a second configuration
Run the same suite on a configuration that escalates password resets too. The cases and scores stay the same, so the two reports compare row by row.
{:ok, eager} = Tune.evaluate(suite.(%{escalate: [:chargeback, :password_reset]}))
SuiteReport.print(eager, [:task_success, :correct_escalation])
password-reset ok task_success=0 correct_escalation=0
chargeback ok task_success=1 correct_escalation=1
order-status error task_success=- correct_escalation=-
task_success: mean 0.5, scored 2 of 3, coverage 0.67
correct_escalation: mean 0.5, scored 2 of 3, coverage 0.67
failed rows: [{"order-status", :inventory_service_timeout}]
The password-reset row shows where the second configuration loses: it
escalates a routine request. Use Tune.compare/3 when you need the per-row
changes and paired coverage as data instead of two printed reports.
Next steps
- Gate a release compares a baseline and a search winner on the same held-out cases.
- Score transcripts scores ordered agent turns.
- Programs and evaluation covers typed programs and score callbacks.