Imp 03: Evaluate And Optimize

imp_checkout? = fn path ->
  is_binary(path) and File.regular?(Path.join(path, "mix.exs")) and
    File.regular?(Path.join(path, "lib/imp.ex"))
end

explicit_repo = System.get_env("IMP_PATH")

if explicit_repo && not imp_checkout?.(Path.expand(explicit_repo)) do
  raise "IMP_PATH does not point to an Imp source checkout or unpacked package"
end

repo =
  [explicit_repo, Path.expand("..", __DIR__), File.cwd!()]
  |> Enum.reject(&is_nil/1)
  |> Enum.map(&Path.expand/1)
  |> Enum.find(imp_checkout?)

if repo do
  # Prefer the notebook's own Imp checkout even when Livebook was launched
  # from an unrelated Mix project. Unpacked package archives may omit
  # mix.lock, so pin it only when the source checkout actually provides it.
  install_opts =
    if File.regular?(Path.join(repo, "mix.lock")),
      do: [lockfile: Path.join(repo, "mix.lock")],
      else: []

  Mix.install([{:imp, path: repo}], install_opts)
else
  # Standalone notebook: install the tagged Git source release.
  Mix.install([{:imp, github: "deepfates/imp", tag: "v0.4.0"}])
end
live_provider_enabled? = System.get_env("LIVE_PROVIDER") == "1"

Build Train And Dev Sets

Livebook 02 made the program inspectable. This chapter gives that program a scorecard. Optimizers are only useful after examples and metrics say what better means.

Optimizers need two kinds of examples. The train set is material they can use to build candidates. The dev set is the held-out scorecard used to choose between those candidates.

trainset = [
  Imp.example(question: "2+2?", answer: "4") |> Imp.with_inputs(:question),
  Imp.example(question: "3+3?", answer: "6") |> Imp.with_inputs(:question)
]

devset = [
  Imp.example(question: "2 plus 2?", answer: "4") |> Imp.with_inputs(:question)
]

Define A Metric

The metric is the telos of an optimization run. Imp can search demos, instructions, or artifacts, but it can only improve what the metric can see.

metric = Imp.exact_match(:answer)

Evaluate A Baseline

Start with a baseline before optimizing. A baseline tells you whether the metric, dev set, adapter, and LM are wired together before search adds motion.

lm =
  Imp.LM.Static.new(
    handler: fn messages, _opts ->
      prompt = Enum.map_join(messages, "\n", & &1.content)
      if prompt =~ "answer: 4" or prompt =~ "Always answer 4",
        do: %{answer: "4"},
        else: %{answer: "unknown"}
    end
  )

program = Imp.predict("question -> answer", lm: lm)
Imp.evaluate(program, devset, metric)

Labeled Few-Shot

The simplest improvement is to attach known-good examples as demos. This is not magic training; it is ordinary data being rendered by the adapter.

compiled =
  Imp.Optimizer.LabeledFewShot.new(k: 1)
  |> then(&Imp.optimize!(program, &1, trainset))

Imp.evaluate(compiled, devset, metric)

Random Search

Random search tries several demo subsets and keeps the candidate that scores best. Its value is not sophistication; its value is that it creates an inspectable optimization report.

compiled =
  metric
  |> Imp.Optimizer.RandomSearch.new(candidates: 3, demos_per_candidate: 1)
  |> then(&Imp.optimize!(program, &1, trainset, devset))

{
  Imp.evaluate(compiled, devset, metric),
  Imp.Optimizer.Report.fetch(compiled)
}

Instruction Search

Instruction search changes the program instructions and keeps the candidate that scores best against the dev set.

Use this when the examples are fine but the task wording is the bottleneck.

compiled =
  Imp.Optimizer.InstructionSearch.compile(
    program,
    metric,
    trainset,
    devset,
    ["Answer unknown.", "Always answer 4."]
  )

Imp.Optimizer.Report.fetch(compiled)

Optimize Anything

Use artifact optimization when the thing you want to improve is not an Imp program yet: a config file, policy text, rubric, template, or other named artifact.

This is the broader Imp philosophy in miniature: define an artifact, define how to score it, then let the system propose and evaluate changes.

result =
  Imp.Optimize.Anything.run(
    "mode=slow",
    fn candidate ->
      cond do
        candidate =~ "mode=fast" and candidate =~ "timeout=5" -> 1.0
        candidate =~ "mode=fast" -> 0.5
        true -> 0.0
      end
    end,
    config: [
        engine: [max_candidate_proposals: 2, parallel: false],
        reflection: [
          custom_candidate_proposer: fn candidate, component, _records, _iteration ->
            current = Map.fetch!(candidate, component)

            if current =~ "mode=fast",
              do: current <> "\ntimeout=5",
              else: current <> "\nmode=fast"
          end
        ]
    ]
  )

IO.inspect(result, label: "Optimize Anything result")

Evaluate A Live Provider

This cell spends a few provider calls only when LIVE_PROVIDER=1 and provider credentials are present. It proves that the same evaluate -> optimize -> report path works with a real LM, while keeping the dataset intentionally tiny.

live_lm = fn opts ->
  if live_provider_enabled? && System.get_env("OPENAI_API_KEY") && System.get_env("OPENAI_MODEL") do
    {:ok,
     Imp.req_llm("openai:#{System.fetch_env!("OPENAI_MODEL")}",
       Keyword.merge(
         [
           api_key: System.fetch_env!("OPENAI_API_KEY"),
           temperature: 0,
           max_completion_tokens: 100
         ],
         opts
       )
     )}
  else
    {:skip, "Set LIVE_PROVIDER=1, OPENAI_API_KEY, and OPENAI_MODEL to run the live provider cell."}
  end
end

case live_lm.(max_completion_tokens: 100) do
  {:ok, lm} ->
    live_program =
      Imp.predict(
        Imp.signature(
          "question -> answer: string",
          "Return JSON only. The answer field must be exactly the requested numeral."
        ),
        lm: lm,
        adapter: Imp.Adapter.JSON,
        config: [json_retries: 1]
      )

    live_trainset = [
      Imp.example(question: "Return the answer exactly 4.", answer: "4")
      |> Imp.with_inputs(:question)
    ]

    live_devset = [
      Imp.example(question: "Return the answer exactly 4.", answer: "4")
      |> Imp.with_inputs(:question)
    ]

    live_metric = fn _example, prediction ->
      prediction
      |> Imp.get(:answer, "")
      |> to_string()
      |> String.trim()
      |> Kernel.==("4")
    end

    baseline = Imp.evaluate(live_program, live_devset, live_metric)

    unless baseline.score == 1.0 do
      raise "live provider returned an invalid evaluation: #{inspect(baseline)}"
    end

    compiled =
      Imp.optimize!(
        live_program,
        Imp.Optimizer.RandomSearch.new(live_metric, candidates: 2, demos_per_candidate: 1),
        live_trainset,
        live_devset
      )

    %{
      baseline_score: baseline.score,
      compiled_score: Imp.evaluate(compiled, live_devset, live_metric).score,
      optimizer_report: Imp.Optimizer.Report.fetch(compiled)
    }

  skip ->
    skip
end

GEPA-Style Reflection

GEPA-style optimization keeps per-example diagnostics, reflects on misses, and uses Pareto pressure so an improvement for one case does not erase performance on another.

The key idea is not "make a bigger prompt." The key idea is to preserve useful diagnostics from failures and use them as material for the next candidate.

result =
  Imp.Optimize.Anything.run(
    "Base",
    fn candidate, requirement ->
      if String.contains?(candidate, requirement), do: 1.0, else: {0.0, %{feedback: requirement}}
    end,
    dataset: ["Paris", "concise"],
    valset: ["Paris"],
    config: [
        engine: [max_candidate_proposals: 1, parallel: false],
        reflection: [
          custom_candidate_proposer: fn _candidate, _component, _records, _iteration ->
            "Paris\nconcise"
          end
        ]
    ]
  )

IO.inspect(result, label: "GEPA-style reflection result")

Next: open livebooks/04_tools_agents_mcp_rlm.livemd when the program needs explicit tools, external catalogs, or bounded recursive exploration.