Powered by AppSignal & Oban Pro

Score Retrieval

guides/examples/score-retrieval.livemd

Score Retrieval

This notebook scores one saved answer from a retrieval-augmented application. Tune.Scores.Faithfulness checks whether the retrieved passages support the answer's claims. Tune.Scores.ContextPrecision checks whether the relevant passages were ranked first. Both scores call a model judge; here the judge replies are scripted, so the notebook runs offline with no credentials.

Setup

Mix.install([
  {:tune, "~> 0.5.0"},
  {:plug, "~> 1.20"}
])

Script the judge

The Scripted module serves OpenAI-compatible replies to ReqLLM through an in-process Plug. It stands in for the provider so the notebook runs offline. In real use you pass a real model spec and the host's credentials instead.

Judge scores request structured objects. With openai_structured_output_mode: :tool_strict, ReqLLM asks for the object as a tool call, so a {:tool, json} reply returns the object's JSON. A {:http_error, status, body} reply makes the judge request fail.

defmodule Scripted do
  @moduledoc "Serves scripted OpenAI-compatible judge replies to ReqLLM through an in-process Plug."

  def model, do: %{provider: :openai, id: "gpt-4-turbo", extra: %{wire: %{protocol: "openai_chat"}}}

  def judge(replies) do
    {:ok, queue} = Agent.start_link(fn -> replies end)

    plug = fn conn ->
      reply = Agent.get_and_update(queue, fn [next | rest] -> {next, rest}; [] -> {nil, []} end)
      respond(conn, reply)
    end

    [
      model: model(),
      request_options: [
        api_key: "scripted",
        req_http_options: [plug: plug],
        max_retries: 0,
        openai_structured_output_mode: :tool_strict
      ]
    ]
  end

  defp respond(conn, nil), do: send_json(conn, 500, %{error: %{message: "no scripted reply left"}})

  defp respond(conn, {:http_error, status, body}), do: send_json(conn, status, body)

  defp respond(conn, {:tool, arguments}) do
    send_json(conn, 200, %{
      id: "scripted",
      model: "gpt-4-turbo",
      choices: [
        %{
          message: %{
            role: "assistant",
            tool_calls: [
              %{
                id: "call_1",
                type: "function",
                function: %{name: "structured_output", arguments: arguments}
              }
            ]
          },
          finish_reason: "tool_calls"
        }
      ],
      usage: %{prompt_tokens: 10, completion_tokens: 5, total_tokens: 15}
    })
  end

  defp send_json(conn, status, body) do
    conn
    |> Plug.Conn.put_resp_content_type("application/json")
    |> Plug.Conn.send_resp(status, JSON.encode!(body))
  end
end

Save the answer and its retrieval results

The saved row puts the question and the ordered passages in input. The first passage is irrelevant, and the two relevant passages follow it. The answer is in output, and the reference answer is in expected.

rows = [
  %{
    id: "paris",
    input: %{
      user_input: "Where is Paris?",
      retrieved_contexts: [
        "A bicycle has two wheels.",
        "Paris is in France.",
        "Paris is its capital."
      ]
    },
    output: "Paris is in France.",
    expected: "Paris is in France."
  }
]

Build the two scores

Faithfulness makes one judge call to extract the answer's claims, then one call per claim to check support. This script extracts one claim and supports it, so faithfulness uses two calls.

Context precision makes one judge call per passage, in passage order. concurrency: 1 keeps the calls in order, so the scripted replies line up with the passages.

relevance = fn relevant, reasoning ->
  {:tool, JSON.encode!(%{relevant: relevant, reasoning: reasoning})}
end

support_replies = [
  {:tool, ~s({"statements":[{"text":"Paris is in France."}]})},
  {:tool, ~s({"supported":true,"reasoning":"A retrieved passage supports the answer."})}
]

precision_replies = [
  relevance.(false, "The passage is unrelated."),
  relevance.(true, "The passage locates Paris."),
  relevance.(true, "The passage identifies the country.")
]

{:ok, faithfulness} = Tune.Scores.Faithfulness.new(Scripted.judge(support_replies))

{:ok, precision} =
  Tune.Scores.ContextPrecision.new(Scripted.judge(precision_replies) ++ [concurrency: 1])

Evaluate

Pass both scores to Tune.score/4 and bound the judge work with model_calls: 5, the two faithfulness calls plus three precision calls.

{:ok, report} =
  rows
  |> Tune.replay()
  |> Tune.score(:faithfulness, faithfulness)
  |> Tune.score(:precision, precision)
  |> Tune.limit(model_calls: 5)
  |> Tune.evaluate()

IO.puts("faithfulness: #{report.summaries.faithfulness.mean}")
IO.puts("precision: #{Float.round(report.summaries.precision.mean, 4)}")
IO.puts("judge calls: #{inspect(report.calls)}")
faithfulness: 1.0
precision: 0.5833
judge calls: %{judge: 5}

The answer is fully supported, but precision is below 1.0 because an irrelevant passage came first. Precision averages the precision at each relevant position: (1/2 + 2/3) / 2.

Read the evidence

Each observation keeps the score feedback. The details for each score are in metadata.scores.

[observation] = report.observations
IO.puts("faithfulness: #{observation.feedback.faithfulness}")
IO.puts("precision: #{observation.feedback.precision}")

evidence = observation.metadata.scores

for statement <- evidence.faithfulness.details.statements do
  IO.puts("claim: #{statement.text} supported: #{statement.supported}")
end

for context <- evidence.precision.details.contexts do
  IO.puts("#{context.index} relevant: #{context.relevant} - #{context.reasoning}")
end
faithfulness: 1/1 statements supported
precision: 2/3 contexts relevant
claim: Paris is in France. supported: true
0 relevant: false - The passage is unrelated.
1 relevant: true - The passage locates Paris.
2 relevant: true - The passage identifies the country.

metadata.scores also holds judge_calls, one entry per judge request with its model, usage, and latency.

Keep one score when the other judge fails

Script the first precision request to fail. Faithfulness gets fresh replies.

failure = {:http_error, 422, %{error: %{message: "scripted ranking failure"}}}
supported = relevance.(true, "The passage supports the answer.")

{:ok, faithfulness} = Tune.Scores.Faithfulness.new(Scripted.judge(support_replies))

{:ok, precision} =
  Tune.Scores.ContextPrecision.new(
    Scripted.judge([failure, supported, supported]) ++ [concurrency: 1]
  )

{:ok, report} =
  rows
  |> Tune.replay()
  |> Tune.score(:faithfulness, faithfulness)
  |> Tune.score(:precision, precision)
  |> Tune.limit(model_calls: 5)
  |> Tune.evaluate()

[observation] = report.observations

IO.puts("status: #{observation.status}")
IO.puts("scores: #{inspect(observation.scores)}")
IO.puts("failed scores: #{inspect(Map.keys(observation.score_errors))}")
IO.puts("precision count: #{report.summaries.precision.count}")
IO.puts("judge calls: #{inspect(report.calls)}")
status: error
scores: %{faithfulness: 1.0}
failed scores: [:precision]
precision count: 0
judge calls: %{judge: 5}

The row is marked :error, but the faithfulness score stays available. Every issued request counts against model_calls, including the failed one.

Next steps