DSEx 05: Operate And Live Checks

repo = if File.exists?("mix.exs"), do: File.cwd!(), else: Path.expand("..", __DIR__)
Mix.install([{:dsex, path: repo}])

Local Quality Gates

The earlier notebooks built DSEx programs. This chapter is about trust: what maintainers run before shipping the library, and what application developers should decide before putting provider calls in production.

Run these in a terminal from the source checkout repo root:

mix production.check
mix integration.check
mix protocol.check

These gates are the local project contract. They check formatting, compilation, deterministic tests, package shape, docs, local service integration, and provider-compatible protocol behavior. Run the executable notebook gate when you change teaching material in the source checkout:

mix livebook.execute.check

Application Checklist

Before swapping a useful deterministic program to a live provider, decide the host application’s operational boundaries:

[
  runtime_config: [:model_name, :api_key, :timeout, :temperature],
  tests: [:static_lm_unit_tests, :opt_in_live_smoke],
  safety: [:redaction, :tool_policy, :provider_failure_behavior],
  operations: [:telemetry, :cost_limits, :rate_limits, :rollout_plan]
]

Security Posture

Unknown external keys remain strings:

key = "external_key_#{System.unique_integer([:positive])}"
example = DSEx.example(%{key => "value"})

{
  DSEx.get(example, key),
  try do
    String.to_existing_atom(key)
  rescue
    ArgumentError -> :not_interned
  end
}

Trace redaction:

echo = DSEx.tool(:echo, "echo", fn input -> input end)

agent =
  DSEx.Agent.new(:redactor, fn agent, _input, runtime ->
    DSEx.Agent.call_tool(agent, :echo, %{api_key: "sk-live", nested: %{token: "secret"}}, runtime)
  end, tools: [echo])

{:ok, _output, runtime} = DSEx.Agent.run(agent, %{})
runtime.traces

Optional Live Provider

This cell expects OPENAI_API_KEY and OPENAI_MODEL to be present in your environment. Do not paste secrets into the notebook.

if System.get_env("OPENAI_API_KEY") && System.get_env("OPENAI_MODEL") do
  model = System.fetch_env!("OPENAI_MODEL")

  lm =
    DSEx.req_llm("openai:#{model}",
      api_key: System.fetch_env!("OPENAI_API_KEY"),
      temperature: 0,
      max_completion_tokens: 40
    )

  program = DSEx.predict("question -> answer", lm: lm)
  DSEx.call(program, %{question: "Reply with exactly: pong"})
else
  {:skip, "Set OPENAI_API_KEY and OPENAI_MODEL before running the live provider cell."}
end

Live Gate

In a terminal from the source checkout:

set -a
. ./.env
set +a
LIVE_PROVIDER=1 mix live.check

Live Operations Proof

This cell validates the same live provider path and asserts the expected output when credentials are present.

if System.get_env("OPENAI_API_KEY") && System.get_env("OPENAI_MODEL") do
  model = System.fetch_env!("OPENAI_MODEL")

  lm =
    DSEx.req_llm("openai:#{model}",
      api_key: System.fetch_env!("OPENAI_API_KEY"),
      temperature: 0,
      max_completion_tokens: 40
    )

  program = DSEx.predict("question -> answer", lm: lm)
  {:ok, prediction} = DSEx.call(program, %{question: "Reply with exactly: pong"})
  answer = prediction |> DSEx.get(:answer, "") |> to_string() |> String.downcase()

  unless String.contains?(answer, "pong") do
    raise "live operations proof failed: #{inspect(DSEx.to_map(prediction))}"
  end

  DSEx.to_map(prediction)
else
  {:skip, "Set OPENAI_API_KEY and OPENAI_MODEL before running the live operations proof."}
end

Saving Without Secrets

lm = DSEx.req_llm("openai:gpt-test", api_key: "not-persisted", temperature: 0)
program = DSEx.predict("question -> answer", lm: lm)

path = Path.join(System.tmp_dir!(), "dsex-no-secret.json")
DSEx.save!(program, path)
state = File.read!(path)
loaded = DSEx.load!(path)
File.rm(path)

{
  String.contains?(state, "not-persisted"),
  Keyword.has_key?(loaded.lm.opts, :api_key),
  loaded.lm
}