Powered by AppSignal & Oban Pro

Score Transcripts

guides/examples/score-transcripts.livemd

Score Transcripts

This notebook replays saved support conversations and scores two things separately: whether the agent escalated when it should, and whether the conversation ended with the expected resolution. One transcript has no final answer, so its resolution score fails while its escalation score still counts. The notebook runs offline and needs no credentials.

Setup

Mix.install([
  {:tune, "~> 0.5.0"}
])

Save the transcripts

Each row stores the conversation as an ordered list of turns in output. input holds the ticket and the resolution the support team expects. expected says whether the ticket should be escalated.

transcripts = [
  %{
    id: "refund",
    input: %{ticket: "refund", resolution: "Refund sent"},
    expected: %{escalate: true},
    output: [
      %{role: :customer, text: "I'd like a refund"},
      %{role: :agent, text: "Escalating to payments"},
      %{role: :agent, text: "Refund sent"}
    ]
  },
  %{
    id: "password",
    input: %{ticket: "password reset", resolution: "Reset link sent"},
    expected: %{escalate: false},
    output: [
      %{role: :customer, text: "I can't log in"},
      %{role: :agent, text: "Reset link sent"}
    ]
  },
  %{
    id: "blocked",
    input: %{ticket: "blocked payment", resolution: "Payment restored"},
    expected: %{escalate: true},
    output: [
      %{role: :customer, text: "Payment is blocked"},
      %{role: :agent, text: "Escalating to payments"}
    ]
  }
]

The blocked conversation stops after the hand-off, so it has no final answer.

Define the scorers

Each scorer is a module with a score/2 function that receives the case and the saved turns. Keep the scorers separate so one can fail without hiding the other.

Escalation checks whether any agent turn escalates and compares that with expected.escalate.

defmodule Escalation do
  @moduledoc "Scores whether the agent escalated exactly when the ticket required it."

  def score(row, turns) do
    escalated? = Enum.any?(turns, &escalation?/1)
    if escalated? == row.expected.escalate, do: 1, else: 0
  end

  def escalation?(%{role: :agent, text: "Escalating" <> _}), do: true
  def escalation?(_turn), do: false
end

Resolution reads the last turn. If the conversation ends on a customer turn or an escalation, there is no final answer to judge, so the scorer returns an error instead of a zero.

defmodule Resolution do
  @moduledoc "Scores whether the final agent turn delivers the expected resolution."

  def score(row, turns) do
    case List.last(turns) do
      %{role: :agent, text: text} = turn ->
        cond do
          Escalation.escalation?(turn) -> {:error, :missing_final_answer}
          text == row.input.resolution -> 1
          true -> 0
        end

      _turn ->
        {:error, :missing_final_answer}
    end
  end
end

Replay and evaluate

Tune.replay/1 turns each row into a case. The saved turns reach the scorers unchanged and in order.

{:ok, report} =
  transcripts
  |> Tune.replay()
  |> Tune.score(:escalation, Escalation)
  |> Tune.score(:resolution, Resolution)
  |> Tune.evaluate()

for name <- [:escalation, :resolution] do
  summary = report.summaries[name]
  IO.puts("#{name}: mean #{summary.mean}, count #{summary.count}, coverage #{Float.round(summary.coverage, 3)}")
end
escalation: mean 1.0, count 3, coverage 1.0
resolution: mean 1.0, count 2, coverage 0.667

Escalation scored every transcript. Resolution scored two of three. Its mean covers only the scored rows, and coverage shows the missing one.

Inspect row errors

Each observation keeps the scores that succeeded and the errors for the ones that failed.

for observation <- report.observations do
  IO.puts(
    "#{observation.id} #{observation.status} scores: #{inspect(observation.scores)} " <>
      "errors: #{inspect(observation.score_errors)}"
  )
end
refund ok scores: %{resolution: 1, escalation: 1} errors: %{}
password ok scores: %{resolution: 1, escalation: 1} errors: %{}
blocked error scores: %{escalation: 1} errors: %{resolution: :missing_final_answer}

The blocked row has status :error because one of its scores failed. Its escalation score is still in the summary.

The saved turns are kept on each observation, in their original order.

refund = hd(report.observations)
IO.inspect(Enum.map(refund.output, & &1.role))
[:customer, :agent, :agent]

Next steps