Typed Predictors
In this notebook you build a predictor that reads a support ticket and returns a typed category and confidence. You preview the prompt in the Chat, JSON, and XML formats, then run the predictor against scripted model replies and see both a validated output and a rejected one. The notebook runs offline and needs no credentials.
Setup
Mix.install([
{:tune, "~> 0.5.0"},
{:plug, "~> 1.20"}
])
Script the model replies
The Scripted module below is a Plug that stands in for the provider. ReqLLM
sends each request to it, and it answers with the next scripted reply, so the
notebook runs offline. In your application, pass a real model spec and the
host's credentials instead.
defmodule Scripted do
@moduledoc "Serves scripted OpenAI-compatible replies to ReqLLM through an in-process Plug."
def model, do: %{provider: :openai, id: "gpt-4-turbo", extra: %{wire: %{protocol: "openai_chat"}}}
def options(replies) do
{:ok, queue} = Agent.start_link(fn -> replies end)
plug = fn conn ->
reply = Agent.get_and_update(queue, fn [next | rest] -> {next, rest}; [] -> {nil, []} end)
respond(conn, reply)
end
[api_key: "scripted", req_http_options: [plug: plug], max_retries: 0]
end
def chat(fields) do
Enum.map_join(fields, "\n\n", fn {name, value} -> "[[ ## #{name} ## ]]\n#{value}" end) <>
"\n\n[[ ## completed ## ]]"
end
defp respond(conn, nil), do: send_json(conn, 500, %{error: %{message: "no scripted reply left"}})
defp respond(conn, text) do
send_json(conn, 200, %{
id: "scripted",
model: "gpt-4-turbo",
choices: [%{message: %{role: "assistant", content: text}, finish_reason: "stop"}],
usage: %{prompt_tokens: 10, completion_tokens: 5, total_tokens: 15}
})
end
defp send_json(conn, status, body) do
conn
|> Plug.Conn.put_resp_content_type("application/json")
|> Plug.Conn.send_resp(status, JSON.encode!(body))
end
end
Declare the predictor
A predictor pairs a signature with optional demonstrations. The signature
lists the instructions, the input fields, and the output fields. Each field
has a Zoi schema. Here label must be one of two strings, and confidence
must be a float.
{:ok, predictor} =
Tune.Predictor.new(
signature: [
instructions: "Extract the ticket category and confidence.",
inputs: [ticket: Zoi.string()],
outputs: [label: Zoi.enum(["billing", "bug"]), confidence: Zoi.float()]
]
)
ticket = %{ticket: "I was charged twice for my card purchase"}
Enum.map(predictor.signature.outputs, & &1.name)
["label", "confidence"]
Preview the prompt in three formats
The adapter decides how Tune renders the prompt and parses the reply. The
target model configuration names the adapter. Make one configuration per
format so that each preview is independent. Tune.Adapter.JSON requires a
:mode option; :text asks for a plain-text JSON object.
configs = %{
chat: [model: Scripted.model(), adapter: Tune.Adapter.Chat],
json: [model: Scripted.model(), adapter: Tune.Adapter.JSON, adapter_options: [mode: :text]],
xml: [model: Scripted.model(), adapter: Tune.Adapter.XML]
}
show = fn format ->
{:ok, prepared} =
predictor
|> Tune.new(model: Map.fetch!(configs, format))
|> Tune.preview(ticket)
for message <- prepared.context.messages do
IO.puts("--- #{message.role}")
IO.puts(Enum.map_join(message.content, "", & &1.text))
end
:ok
end
Tune.preview/2 returns a Tune.Prepared struct. It holds the rendered
ReqLLM context and the output descriptor. It makes no model call, so these
configurations need no request options.
The Chat format marks each field with [[ ## name ## ]]:
show.(:chat)
--- system
Your input fields are:
1. `ticket` (string)
Your output fields are:
1. `label` (enum)
2. `confidence` (number)
All interactions will be structured in the following way, with the appropriate values filled in.
[[ ## ticket ## ]]
{ticket}
[[ ## label ## ]]
{label} # note: the value you produce must exactly match (no extra characters) one of: billing; bug
[[ ## confidence ## ]]
{confidence} # note: the value you produce must adhere to the JSON schema: {"type":"number"}
[[ ## completed ## ]]
In adhering to this structure, your objective is:
Extract the ticket category and confidence.
--- user
[[ ## ticket ## ]]
I was charged twice for my card purchase
Respond with the corresponding output fields, starting with the field `[[ ## label ## ]]`, then `[[ ## confidence ## ]]` (must be formatted as valid JSON: number), and then ending with the marker for `[[ ## completed ## ]]`.
The JSON format keeps the marked inputs and asks for one JSON object:
show.(:json)
--- system
Your input fields are:
1. `ticket` (string)
Your output fields are:
1. `label` (enum)
2. `confidence` (number)
All interactions will be structured in the following way, with the appropriate values filled in.
Inputs will have the following structure:
[[ ## ticket ## ]]
{ticket}
Outputs will be a JSON object with the following fields.
{
"label": "{label}", # note: the value you produce must exactly match (no extra characters) one of: billing; bug
"confidence": "{confidence}" # note: the value you produce must adhere to the JSON schema: {"type":"number"}
}
In adhering to this structure, your objective is:
Extract the ticket category and confidence.
--- user
[[ ## ticket ## ]]
I was charged twice for my card purchase
Respond with a JSON object in the following order of fields: `label`, then `confidence` (must be formatted as valid JSON: number).
The XML format wraps every field in an element:
show.(:xml)
--- system
Your input fields are:
1. `ticket` (string)
Your output fields are:
1. `label` (enum)
2. `confidence` (number)
All interactions will be structured in the following way, with the appropriate values filled in.
<ticket>
{ticket}
</ticket>
<label>
{label} # note: the value you produce must exactly match (no extra characters) one of: billing; bug
</label>
<confidence>
{confidence} # note: the value you produce must adhere to the JSON schema: {"type":"number"}
</confidence>
In adhering to this structure, your objective is:
Extract the ticket category and confidence.
--- user
<ticket>
I was charged twice for my card purchase
</ticket>
Respond with the corresponding output fields wrapped in XML tags, starting with `<label>`, then `<confidence>`.
The signature is the same in all three. Only the adapter changed.
Run with a valid reply
Tune.Runtime.run/3 renders the prompt, sends it through ReqLLM, parses the
reply, and validates every output field against its schema. Add
request_options to the model configuration so the request reaches the
scripted Plug.
run = fn format, reply ->
model = Map.fetch!(configs, format) ++ [request_options: Scripted.options([reply])]
Tune.Runtime.run(predictor, ticket, model: model)
end
{:ok, prediction} = run.(:chat, Scripted.chat(label: "billing", confidence: "0.8"))
{prediction.output, prediction.calls}
{%{"confidence" => 0.8, "label" => "billing"}, %{target: 1}}
The result is a Tune.Prediction. Its output map uses string keys and holds
values that passed the Zoi schemas. calls counts one target model call.
The same predictor parses a reply in each adapter's own format:
{:ok, from_json} = run.(:json, ~s({"label": "bug", "confidence": 0.6}))
{:ok, from_xml} = run.(:xml, "<label>bug</label>\n<confidence>0.7</confidence>")
{from_json.output, from_xml.output}
{%{"confidence" => 0.6, "label" => "bug"},
%{"confidence" => 0.7, "label" => "bug"}}
Run with an invalid field
Now the model answers with a label outside the enum. Tune parses the reply, then rejects it during validation. It does not repair the value or retry.
{:error, error} = run.(:chat, Scripted.chat(label: "refund", confidence: "0.8"))
{error.__struct__, error.cause.__struct__, error.cause.message}
{Tune.Error.Execution.Failed, Tune.Error.Output.Invalid,
"invalid outputs: label: invalid enum value: expected one of billing, bug"}
The error keeps the evidence you need to debug the reply. error.cause.value
holds the parsed values, and error.result holds the partial prediction with
its call count.
{error.cause.value, error.result.calls}
{%{"confidence" => 0.8, "label" => "refund"}, %{target: 1}}
A reply that cannot be parsed fails earlier with Tune.Error.Output.Parse.
Here confidence is not a JSON number:
{:error, parse_error} = run.(:chat, Scripted.chat(label: "billing", confidence: "high"))
{parse_error.cause.__struct__, parse_error.cause.message}
{Tune.Error.Output.Parse, "field \"confidence\" is not valid JSON"}
Next steps
- Evaluate a Function shows how to score outputs over labeled cases.
- Programs composes predictors into multi-step programs.
- Host Integration shows how a host sends the prepared request through its own inference path.