Evaluate a Function
In this notebook you measure a plain Elixir function against labeled cases. You write a shipping-price function, give Tune three test rows, add two scores, and read the per-case results and the score summaries. The notebook makes no model calls and needs no credentials.
Setup
Mix.install([
{:tune, "~> 0.5.0"}
])
Write the function under test
The function prices a parcel from its weight and delivery zone. It is ordinary Elixir code. Tune does not need to know anything about it.
defmodule Shipping do
@rates %{"local" => 5.0, "national" => 9.0}
def quote(%{weight_kg: weight, zone: zone}) do
%{price: Map.get(@rates, zone, 12.0) + weight * 1.5}
end
end
Shipping.quote(%{weight_kg: 1, zone: "local"})
%{price: 6.5}
Describe the test cases
Each row has an id, an input, and an optional expected value. Tune passes
only input to the function. Scores see the whole case, so labels never reach
the code under test.
The third row has no expected value. You use it to see how Tune reports a
score that fails on one case.
rows = [
%{id: "small-local", input: %{weight_kg: 1, zone: "local"}, expected: %{price: 6.5}},
%{id: "heavy-national", input: %{weight_kg: 10, zone: "national"}, expected: %{price: 24.0}},
%{id: "unlabeled", input: %{weight_kg: 2, zone: "local"}}
]
length(rows)
3
Build the pipeline
Tune.new/2 takes the target and a runner. The runner receives the target and
one case input, and returns the output. Here the target is the Shipping
module, and the runner calls quote/1. In your own code, pass the function you
already use in production.
Tune.data/2 attaches the rows as the :test role. Tune reads them only when
the pipeline runs.
Add two independent scores with Tune.score/4:
:exactis a two-argument scorer. It reads the case and the output, and compares the price with the label.:under_20is a one-argument scorer. It reads only the output.
With more than one score, Tune.select/2 names the score that decides the
headline value.
pipeline =
Shipping
|> Tune.new(run: fn shipping, input -> shipping.quote(input) end)
|> Tune.data(test: rows)
|> Tune.score(:exact, fn tune_case, output ->
if output.price == tune_case.expected.price, do: 1, else: 0
end)
|> Tune.score(:under_20, fn output -> if output.price < 20, do: 1, else: 0 end)
|> Tune.select(:under_20)
pipeline.scores |> Enum.map(& &1.name)
[:exact, :under_20]
Building the pipeline runs nothing. Each builder call returns a new %Tune{}
value.
Run the evaluation
Tune.evaluate/2 runs the function on every :test row and scores each
output. It returns {:ok, %Tune.Evaluation{}}.
{:ok, evaluation} = Tune.evaluate(pipeline)
{evaluation.complete?, evaluation.eligible?, evaluation.stop_reason}
{true, true, :completed}
Inspect each case
evaluation.observations keeps one Tune.Observation per case, in row order.
Each observation holds the case id, its status, the output, the numeric
scores, and any score_errors.
for observation <- evaluation.observations do
IO.puts(
"#{observation.id}: #{observation.status} " <>
"price=#{observation.output.price} " <>
"scores=#{inspect(observation.scores)} " <>
"score_errors=#{inspect(Map.keys(observation.score_errors))}"
)
end
:ok
small-local: ok price=6.5 scores=%{exact: 1, under_20: 1} score_errors=[]
heavy-national: ok price=24.0 scores=%{exact: 1, under_20: 0} score_errors=[]
unlabeled: error price=8.0 scores=%{under_20: 1} score_errors=[:exact]
The function produced an output for unlabeled, but the :exact scorer
raised because the row has no expected map. Tune records that as a score
error on the case and marks the observation :error. The :under_20 score for
the same case still counts.
Read the error to see what went wrong:
unlabeled = List.last(evaluation.observations)
Exception.message(unlabeled.score_errors.exact)
"the score raised: expected a map, got:\n\n nil\n"
The error is a Tune.Error.Execution.Callback. A score that raises, exits, or
returns something other than a number or %{score: number, feedback: text}
produces this error. The evaluation itself still returns {:ok, evaluation}.
Read the score summaries
evaluation.summaries holds one Tune.Score.Summary per score.
Tune.Evaluation.summary/2 returns one by name. count is the number of
scored cases, and coverage is the share of cases that produced that score.
for name <- [:exact, :under_20] do
summary = Tune.Evaluation.summary(evaluation, name)
IO.puts("#{name}: mean=#{summary.mean} count=#{summary.count} coverage=#{Float.round(summary.coverage, 2)}")
end
evaluation.selected_score
exact: mean=1.0 count=2 coverage=0.67
under_20: mean=0.6666666666666666 count=3 coverage=1.0
0.6666666666666666
The :exact mean is 1.0, but it covers only two of three cases. Always read a
mean together with its coverage. selected_score is the mean of the selected
score, :under_20.
The evaluation stays eligible because the selected score covers every case. If
:exact were the selected score, the missing case would make the evaluation
ineligible and selected_score would be nil. Pass failure_score: to
Tune.score/4 when a failed case should count as a fixed value instead.
Next steps
- Programs shows how to evaluate typed LLM programs with the same pipeline.
- Typed Predictors builds a predictor that you can
pass to
Tune.new/2as the target. - Search API varies values and searches for a better configuration.