DSEx 02: Programming, Not Prompting

repo = if File.exists?("mix.exs"), do: File.cwd!(), else: Path.expand("..", __DIR__)
Mix.install([{:dsex, path: repo}])

The Idea

Livebook 01 showed the real-provider shape. This chapter removes provider variance so you can inspect the same DSEx ideas with a deterministic local LM.

It also includes a live-provider proof cell near the end. The point is that DSEx.LM.Static is a development tool, not a different product path.

DSEx-style programming means declaring the task boundary first:

The signature is the smallest useful unit in DSEx. It says what data enters the task and what data must come back. It does not say which provider to use, how many examples to show, or how the final prompt should look.

signature = DSEx.signature("question -> answer")
DSEx.Signature.to_spec(signature)

Then you attach an LM and call a program:

This notebook uses a deterministic local LM so the behavior is stable. The LM is deliberately boring; the point is to see the DSEx program shape without network calls or provider variance.

lm = %{
  module: DSEx.LM.Static,
  opts: [
    handler: fn messages, _opts ->
      prompt = Enum.map_join(messages, "\n", & &1.content)
      if prompt =~ "France", do: %{answer: "Paris"}, else: %{answer: "unknown"}
    end
  ]
}

DSEx.configure(lm: lm, adapter: DSEx.Adapter.Chat)

qa_program = DSEx.predict("question -> answer")
{:ok, prediction} = DSEx.call(qa_program, %{question: "Capital of France?"})
DSEx.to_map(prediction)

Inspect The Generated Messages

Predictions keep trace metadata:

The generated messages are still available. DSEx is not asking you to trust a black box; it is giving the prompt an owner, a contract, and a place in normal Elixir data.

prediction.metadata.trace.messages

The point is not to hide prompts forever. The point is to make them a generated artifact of a typed program.

Chain Of Thought

Some tasks benefit from a separate reasoning field. In DSEx that is still a structured output, not an informal convention hidden in a prompt.

cot_lm = %{
  module: DSEx.LM.Static,
  opts: [handler: fn _messages, _opts -> %{reasoning: "2 + 2 is 4.", answer: "4"} end]
}

cot_program = DSEx.chain_of_thought("question -> answer", lm: cot_lm)
{:ok, pred} = DSEx.call(cot_program, %{question: "What is 2+2?"})
DSEx.to_map(pred)

Schema-Constrained Output

When another part of your application consumes the result, prefer constraints over hoping the model follows prose. The JSON adapter validates the parsed fields and returns feedback when output does not fit the signature.

signature =
  DSEx.signature(%{
    inputs: [:text],
    outputs: [
      %{name: :sentiment, type: :string, constraints: %{enum: ["positive", "negative"]}},
      %{name: :confidence, type: :number, constraints: %{min: 0.0, max: 1.0}}
    ]
  })

{:ok, pred} =
  DSEx.Adapter.JSON.parse(
    signature,
    ~s({"sentiment":"positive","confidence":0.91}),
    []
  )

DSEx.to_map(pred)

Try a bad output:

DSEx.Adapter.JSON.parse(signature, ~s({"sentiment":"mixed","confidence":2.0}), [])

Examples

Examples are rows of data. Marking inputs tells DSEx which fields the model can see and which fields are labels for evaluation or optimization.

example =
  DSEx.example(question: "2+2?", answer: "4")
  |> DSEx.with_inputs(:question)

{DSEx.to_map(DSEx.inputs(example)), DSEx.to_map(DSEx.labels(example))}

Live Provider Proof

This is the same programming model against a real LM. The cell validates the typed JSON result when OPENAI_API_KEY and OPENAI_MODEL are present.

live_lm = fn opts ->
  if System.get_env("OPENAI_API_KEY") && System.get_env("OPENAI_MODEL") do
    {:ok,
     DSEx.req_llm("openai:#{System.fetch_env!("OPENAI_MODEL")}",
       Keyword.merge(
         [
           api_key: System.fetch_env!("OPENAI_API_KEY"),
           temperature: 0,
           max_completion_tokens: 120
         ],
         opts
       )
     )}
  else
    {:skip, "Set OPENAI_API_KEY and OPENAI_MODEL to run the live proof cell."}
  end
end

case live_lm.(max_completion_tokens: 120) do
  {:ok, lm} ->
    live_program =
      DSEx.predict(
        DSEx.signature(
          "text -> sentiment: enum[positive,negative], confidence: number",
          "Classify the text. Return JSON only."
        ),
        lm: lm,
        adapter: DSEx.Adapter.JSON,
        config: [json_retries: 1]
      )

    {:ok, live_prediction} =
      DSEx.call(live_program, %{text: "DSEx makes LM programs testable and useful."})

    sentiment = DSEx.get(live_prediction, :sentiment)
    confidence = DSEx.get(live_prediction, :confidence)

    unless sentiment == "positive" and is_number(confidence) do
      raise "live provider returned an invalid typed prediction: #{inspect(DSEx.to_map(live_prediction))}"
    end

    %{
      sentiment: sentiment,
      confidence: confidence,
      trace_messages: length(live_prediction.metadata.trace.messages)
    }

  skip ->
    skip
end

Save And Load

Program state can be saved without serializing provider secrets. That makes it reasonable to treat compiled programs as artifacts while rebinding live credentials at runtime.

path = Path.join(System.tmp_dir!(), "dsex-livebook-program.json")
DSEx.save!(qa_program, path)
loaded = DSEx.load!(path)
File.rm(path)

loaded.signature |> DSEx.Signature.to_spec()

Next: open livebooks/03_evaluate_and_optimize.livemd to turn examples into a metric, evaluate the baseline, and compile a better program.