Powered by AppSignal & Oban Pro

Axon Learning Rate

axon-learning-rate.livemd

Axon Learning Rate

In this notebook you pick a learning rate for a small Axon model. Tune tries three rates. For each one, your runner trains a fresh model on a fixed batch and reports the validation loss. Tune keeps the rate with the lowest loss and saves it as an artifact, while the trained model stays in your application. The notebook runs on the CPU, offline, and needs no credentials.

Setup

Mix.install([
  {:tune, "~> 0.5.0"},
  {:nx, "~> 1.0"},
  {:axon, "~> 0.9"}
])

Use the pure Elixir backend so the notebook needs no compiled accelerator:

Nx.default_backend(Nx.BinaryBackend)

Prepare the data

The model learns y = 2x. Training uses one fixed batch of three points. Validation uses two points the model does not train on.

train_batches = [{Nx.tensor([[0.0], [1.0], [2.0]]), Nx.tensor([[0.0], [2.0], [4.0]])}]
validation = {Nx.tensor([[0.5], [1.5]]), Nx.tensor([[1.0], [3.0]])}

Train inside the runner

The runner receives one configuration and one validation case. It builds a fresh one-unit dense model, trains it for five epochs with SGD at the configured learning rate, and measures mean squared error on the validation points.

The runner returns the loss tensor and the trained model state. Tune treats the output as an opaque value: only the score function reads it.

fit = fn config, {features, expected} ->
  model =
    Axon.input("features", shape: {nil, 1})
    |> Axon.dense(1, kernel_initializer: :zeros, bias_initializer: :zeros)

  optimizer = Polaris.Optimizers.sgd(learning_rate: config.learning_rate)

  loop_state =
    model
    |> Axon.Loop.trainer(:mean_squared_error, optimizer, seed: 3, log: 0)
    |> Axon.Loop.run(train_batches, Axon.ModelState.empty(), epochs: 5)

  trained = loop_state.step_state.model_state
  predicted = Axon.predict(model, trained, features)
  loss = predicted |> Nx.subtract(expected) |> Nx.pow(2) |> Nx.mean()

  %{loss: loss, trained: trained}
end

Zero initializers and a fixed loop seed make every run repeatable.

Search the learning rates

The target %{learning_rate: 0.02} is the original setting. Tune.vary/2 lists the rates to try, and grid search tries each one in order.

The score converts the loss tensor to an Elixir float with Nx.to_number/1. Tune compares plain numbers, not tensors. direction: :minimize makes a lower loss better.

study =
  Tune.new(%{learning_rate: 0.02}, run: fit)
  |> Tune.data(validation: [validation], test: [validation])
  |> Tune.vary(learning_rate: [0.02, 0.1, 0.2])
  |> Tune.score(:loss, &Nx.to_number(&1.loss), direction: :minimize)
  |> Tune.search(:grid)

{:ok, result} = Tune.tune(study)

for candidate <- result.history do
  loss = candidate.evaluation.summaries.loss.mean
  IO.puts("learning_rate #{candidate.value.learning_rate}: loss #{Float.round(loss, 4)}")
end

IO.puts("Selected #{inspect(result.value)}")
learning_rate 0.02: loss 1.7769
learning_rate 0.1: loss 0.0918
learning_rate 0.2: loss 0.0572
Selected %{learning_rate: 0.2}

With only five epochs, the smallest rate barely moves the weights. The largest rate gets closest to y = 2x.

The validation split here is tiny and the search picks from it, so the loss is optimistic. In real work, keep a separate test split that the search never sees.

Compare with the original rate

Tune.compare/3 trains a model with the original rate and one with the selected rate on the same split, then pairs their losses.

{:ok, comparison} = Tune.compare(study, result, split: :test)
metric = comparison.metrics.loss

IO.puts("Original loss #{Float.round(metric.baseline_mean, 4)}")
IO.puts("Selected loss #{Float.round(metric.challenger_mean, 4)}")
IO.puts("Change #{Float.round(metric.delta, 4)} (#{metric.status})")
Original loss 1.7769
Selected loss 0.0572
Change -1.7196 (improved)

For a minimized score, a negative change is an improvement.

Save and reload the setting

Tune.dump/1 builds an artifact from the result. The artifact holds only the learned scalar setting. Tune.load/2 applies it to a fresh target.

{:ok, artifact} = Tune.dump(result)
{:ok, config} = Tune.load(artifact, %{learning_rate: 0.02})

IO.puts("Artifact parameters #{inspect(artifact.parameters)}")
IO.puts("Reloaded #{inspect(config)}")
Artifact parameters %{"learning_rate" => 0.2}
Reloaded %{learning_rate: 0.2}

Keep the model in the application

The trained weights stay in the observation output of the best candidate. They are not part of the artifact.

[observation] = result.best.evaluation.observations
IO.puts("Trained state #{inspect(observation.output.trained.__struct__)}")
IO.puts("Artifact keys #{inspect(Map.keys(artifact.parameters))}")
Trained state Axon.ModelState
Artifact keys ["learning_rate"]

Tune refuses to put tensors or model state in artifact parameters. Store and version the trained model with your application's own tooling. Use the artifact to record which learning rate produced it, and train again with the reloaded configuration when you need a fresh model.

Next steps

  • Batch Settings uses the same grid search on a simulated workload.
  • Artifacts explains what an artifact stores and how loading checks it.
  • Search API lists the grid and genetic search options.