DSEx 03: Evaluate And Optimize

repo = if File.exists?("mix.exs"), do: File.cwd!(), else: Path.expand("..", __DIR__)
Mix.install([{:dsex, path: repo}])

Build Train And Dev Sets

Livebook 02 made the program inspectable. This chapter gives that program a scorecard. Optimizers are only useful after examples and metrics say what better means.

Optimizers need two kinds of examples. The train set is material they can use to build candidates. The dev set is the held-out scorecard used to choose between those candidates.

trainset = [
  DSEx.example(question: "2+2?", answer: "4") |> DSEx.with_inputs(:question),
  DSEx.example(question: "3+3?", answer: "6") |> DSEx.with_inputs(:question)
]

devset = [
  DSEx.example(question: "2 plus 2?", answer: "4") |> DSEx.with_inputs(:question)
]

Define A Metric

The metric is the telos of an optimization run. DSEx can search demos, instructions, or artifacts, but it can only improve what the metric can see.

metric = DSEx.exact_match(:answer)

Evaluate A Baseline

Start with a baseline before optimizing. A baseline tells you whether the metric, dev set, adapter, and LM are wired together before search adds motion.

lm = %{
  module: DSEx.LM.Static,
  opts: [
    handler: fn messages, _opts ->
      prompt = Enum.map_join(messages, "\n", & &1.content)
      if prompt =~ "answer: 4" or prompt =~ "Always answer 4",
        do: %{answer: "4"},
        else: %{answer: "unknown"}
    end
  ]
}

program = DSEx.predict("question -> answer", lm: lm)
DSEx.evaluate(program, devset, metric)

Labeled Few-Shot

The simplest improvement is to attach known-good examples as demos. This is not magic training; it is ordinary data being rendered by the adapter.

compiled =
  DSEx.Optimizer.LabeledFewShot.new(k: 1)
  |> then(&DSEx.optimize(program, &1, trainset))

DSEx.evaluate(compiled, devset, metric)

Random Search

Random search tries several demo subsets and keeps the candidate that scores best. Its value is not sophistication; its value is that it creates an inspectable optimization report.

compiled =
  metric
  |> DSEx.Optimizer.RandomSearch.new(candidates: 3, demos_per_candidate: 1)
  |> then(&DSEx.optimize(program, &1, trainset, devset))

{
  DSEx.evaluate(compiled, devset, metric),
  DSEx.Optimizer.Report.fetch(compiled)
}

Instruction Search

Instruction search changes the program instructions and keeps the candidate that scores best against the dev set.

Use this when the examples are fine but the task wording is the bottleneck.

compiled =
  DSEx.Optimizer.InstructionSearch.compile(
    program,
    metric,
    trainset,
    devset,
    ["Answer unknown.", "Always answer 4."]
  )

DSEx.Optimizer.Report.fetch(compiled)

Optimize Anything

Use artifact optimization when the thing you want to improve is not a DSEx program yet: a config file, policy text, rubric, template, or other named artifact.

This is the broader DSEx philosophy in miniature: define an artifact, define how to score it, then let the system propose and evaluate changes.

artifact = DSEx.Optimize.Anything.new_artifact(:config, "mode=slow")

report =
  DSEx.Optimize.Anything.optimize(
    artifact,
    fn artifact, _examples ->
      cond do
        artifact.text =~ "mode=fast" and artifact.text =~ "timeout=5" -> 1.0
        artifact.text =~ "mode=fast" -> 0.5
        true -> 0.0
      end
    end,
    trials: 2,
    mutation_fn: fn _artifact, trial, _seed ->
      case trial do
        1 -> "mode=fast"
        2 -> "timeout=5"
      end
    end
  )

report.best

Live Evaluation Proof

This cell spends a few provider calls when credentials are present. It proves that the same evaluate -> optimize -> report path works with a real LM, while keeping the dataset intentionally tiny.

live_lm = fn opts ->
  if System.get_env("OPENAI_API_KEY") && System.get_env("OPENAI_MODEL") do
    {:ok,
     DSEx.req_llm("openai:#{System.fetch_env!("OPENAI_MODEL")}",
       Keyword.merge(
         [
           api_key: System.fetch_env!("OPENAI_API_KEY"),
           temperature: 0,
           max_completion_tokens: 100
         ],
         opts
       )
     )}
  else
    {:skip, "Set OPENAI_API_KEY and OPENAI_MODEL to run the live proof cell."}
  end
end

case live_lm.(max_completion_tokens: 100) do
  {:ok, lm} ->
    live_program =
      DSEx.predict(
        DSEx.signature(
          "question -> answer: string",
          "Return JSON only. The answer field must be exactly the requested numeral."
        ),
        lm: lm,
        adapter: DSEx.Adapter.JSON,
        config: [json_retries: 1]
      )

    live_trainset = [
      DSEx.example(question: "Return the answer exactly 4.", answer: "4")
      |> DSEx.with_inputs(:question)
    ]

    live_devset = [
      DSEx.example(question: "Return the answer exactly 4.", answer: "4")
      |> DSEx.with_inputs(:question)
    ]

    live_metric = fn _example, prediction ->
      prediction
      |> DSEx.get(:answer, "")
      |> to_string()
      |> String.trim()
      |> Kernel.==("4")
    end

    baseline = DSEx.evaluate(live_program, live_devset, live_metric)

    unless baseline.score == 1.0 do
      raise "live provider failed the evaluation proof: #{inspect(baseline)}"
    end

    compiled =
      DSEx.optimize(
        live_program,
        DSEx.Optimizer.RandomSearch.new(live_metric, candidates: 2, demos_per_candidate: 1),
        live_trainset,
        live_devset
      )

    %{
      baseline_score: baseline.score,
      compiled_score: DSEx.evaluate(compiled, live_devset, live_metric).score,
      optimizer_report: DSEx.Optimizer.Report.fetch(compiled)
    }

  skip ->
    skip
end

GEPA-Style Reflection

GEPA-style optimization keeps per-example diagnostics, reflects on misses, and uses Pareto pressure so an improvement for one case does not erase performance on another.

The key idea is not “make a bigger prompt.” The key idea is to preserve useful diagnostics from failures and use them as material for the next candidate.

reflection_lm = %{
  module: DSEx.LM.Static,
  opts: [handler: fn _messages, _opts -> %{mutation: "Paris\nconcise"} end]
}

artifact = DSEx.Optimize.Anything.new_artifact(:prompt, "Base")

report =
  DSEx.Optimize.GEPA.optimize(
    artifact,
    fn artifact, examples ->
      %{
        per_example_scores: Enum.map(examples, &if(String.contains?(artifact.text, &1), do: 1.0, else: 0.0)),
        asi: Enum.reject(examples, &String.contains?(artifact.text, &1))
      }
    end,
    examples: ["Paris", "concise"],
    dev_examples: ["Paris"],
    generations: 1,
    reflection_lm: reflection_lm
  )

report.best

Next: open livebooks/04_tools_agents_mcp_rlm.livemd when the program needs explicit tools, external catalogs, or bounded recursive exploration.