Hooks Guide

Section

This guide explains what hooks are, when to use them, and what shape hook payloads must have to be routed back to callers correctly.

What hooks give you

With streaming enabled, BatchServing.dispatch_many/2 returns a stream of events:

• {:batch, output} - batch result payloads
• {hook_name, payload} - custom runtime events emitted by your serving implementation

Hooks are best for:

• progress reporting in LiveView/UIs
• runtime telemetry (latency, retries, token/cost estimates)
• phase/state transitions (queued, embedding, persisting, etc.)

Hook payload shape

Hook payloads are sliced back to each caller using the same batch boundaries as {:batch, output} events. In practice, this means a hook payload should be an ordered collection aligned with the batch input, where each element corresponds to one input item.

Good fits:

• [%{processed?: true, token_estimate: 42}, ...]
• [token_count_1, token_count_2, ...]
• [%{job_id: "job-1", latency_ms: 30}, ...]

Poor fits for the current transport:

• a single aggregate map for the whole execution
• one scalar count/cost value for the whole batch
• grouped-by-user maps that do not preserve item order

If you need aggregate metrics, derive them on the caller side from the per-item hook entries you receive.

Concrete embeddings use case

Imagine a “Document Indexing” screen where users upload thousands of text chunks.

• {:batch, embeddings} events carry vectors/results.
• {:progress, meta} events carry one metadata entry per chunk.

Minimal shape:

[
  %{
    processed_delta: 1,
    latency_ms: 420,
    estimated_cost_usd: 0.012
  },
  ...
]

LiveView can render a progress bar and running status from those events while indexing continues.

LiveView progress

# in handle_batch
if hook = state.hooks[:progress] do
  hook.(
    Enum.map(results, fn result ->
      %{token_estimate: result.token_estimate}
    end)
  )
end

# in your liveview
parent = self()

DemoServing
|> BatchServing.dispatch_many!(items_with_job_id)
|> Enum.reduce(%{processed: 0, total_tokens: 0}, fn
  {:progress, items}, acc ->
    processed = acc.processed + Enum.count(items)
    total_tokens = acc.total_tokens + Enum.sum_by(items, & &1.token_estimate)

    send(
      parent,
      {
        :progress,
        %{processed: processed, total_tokens: total_tokens}
      }
    )

    %{processed: processed, total_tokens: total_tokens}

  {:batch, batch_output}, acc ->
    send(parent, {:results, batch_output})
    acc
end)

LiveView can use hook events to update progress as batches finish while keeping a running total of token usage from the metadata emitted for each item.

LiveView example

See:

• examples/liveview_batch_progress_demo.livemd

This demo shows streaming progress updates in a LiveView page while work executes in batches.