The Build Pipeline

DocShell.Build.run/1 is the top of DocShell's generation pipeline. It reads modules, Markdown guides, Livebook notebooks, release notes, and an OpenAPI document, then projects them into the JSON artifacts a renderer can consume.

This tutorial follows one complete build from source files to artifacts. The examples create a disposable documentation set under your system temp directory, so running the notebook does not touch priv/doc_shell/ or depend on files in this repository.

What you will learn

• How to prepare guide and notebook sources for a build
• What DocShell.Build.run/1 returns in memory
• Which files are written to disk
• Where filtering, paths, search text, and OpenAPI fit in the pipeline
• How DocShell fails when a configured source is invalid

Setup

Run this notebook from Livebook's default standalone runtime. The setup cell installs the matching DocShell release from Hex. Release tooling updates this version together with the package and the notebook links.

Mix.install([
  {:doc_shell, "== 0.3.0"}
])

Application.ensure_all_started(:doc_shell)

Create a small documentation set

A real host project already has modules, guides, and maybe notebooks. For a tutorial, a tiny generated fixture is easier to reason about. The fixture has:

• one Markdown guide with YAML frontmatter
• one .livemd notebook
• one real module from DocShell itself, DocShell.Config

workspace =
  Path.join(
    System.tmp_dir!(),
    "doc_shell_build_pipeline_#{System.unique_integer([:positive])}"
  )

File.rm_rf!(workspace)

guide_dir = Path.join(workspace, "guides")
livebook_dir = Path.join(workspace, "notebooks")
public_dir = Path.join(workspace, "public")
private_dir = Path.join(workspace, "private")

Enum.each([guide_dir, livebook_dir, public_dir, private_dir], &File.mkdir_p!/1)

guide_path = Path.join(guide_dir, "getting-started.md")

File.write!(guide_path, """
---
id: getting-started
title: Getting Started
audience: developers
locale: en
---

# Getting Started

Start with a small documentation set and add sources deliberately.
""")

livebook_path = Path.join(livebook_dir, "incident-playbook.livemd")

File.write!(livebook_path, """
# Incident Playbook

This notebook is documentation too. DocShell indexes the source; it does not
evaluate the code cell.

```elixir
System.system_time(:second)
```
""")

%{
  workspace: workspace,
  guide_path: guide_path,
  livebook_path: livebook_path,
  public_dir: public_dir,
  private_dir: private_dir
}

Run the build

DocShell.Build.run/1 accepts per-call options. These override host configuration, which overrides DocShell defaults. In tutorials and release tasks, explicit options make the example deterministic.

build_opts = [
  modules: [DocShell.Config],
  guide_bases: [guide_dir],
  changelog_source: nil,
  livebook_base: livebook_dir,
  public_dir: public_dir,
  private_dir: private_dir
]

{:ok, result} = DocShell.Build.run(build_opts)

Map.keys(result) |> Enum.sort()

The return value always has the same top-level keys:

• :modules — module documentation extracted from the BEAM docs chunk
• :guides — Markdown guides with parsed body and metadata
• :livebooks — .livemd notebooks with parsed body and source metadata
• :changelog — release-note entries from the configured source
• :openapi — a valid OpenAPI document
• :presentation — navigation, search, and content indexes

Loading release notes from another store

CHANGELOG.md is only the default source. A host whose documentation lives in a graph, database, CMS, or service implements one callback and returns the same renderer-neutral changelog entries:

defmodule MyApp.Docs.ReleaseNotes do
  @behaviour DocShell.Generate.Changelog.Source

  @impl true
  def load(opts) do
    opts
    |> Keyword.fetch!(:markdown)
    |> DocShell.Generate.Changelog.from_markdown("graph://docs/releases")
  end
end

DocShell.Build.run(
  changelog_source: MyApp.Docs.ReleaseNotes,
  changelog_options: [markdown: "## v1.0.0 (2026-08-28)\n\nInitial release"],
  write: false
)

Sources that already store structured entries can return those maps directly; DocShell validates them before projection or artifact writing.

Inspect module extraction

Module documentation is read from compiled BEAM documentation. That is the same source IEx uses for h DocShell.Config, which keeps the artifact aligned with what developers see locally.

module_entry = Enum.find(result.modules, &(&1["id"] == "DocShell.Config"))

Map.take(module_entry, ["id", "title", "kind"])

The module metadata carries structural information useful for renderers and coverage tooling.

module_entry["meta"]
|> Map.take(["module", "language", "moduledoc"])

Function, callback, and type docs are exposed as member metadata. Member docs stay as Markdown strings; renderers can parse them lazily instead of paying the cost for every member up front.

module_entry["meta"]["members"]
|> Enum.map(&Map.take(&1, ["kind", "name", "arity", "signatures"]))
|> Enum.take(5)

Inspect guide extraction

Guides are Markdown files. Frontmatter is optional, but when it exists DocShell passes it through as metadata. audience and locale get a small amount of special treatment later because the search index exposes them as first-class filter fields.

guide_entry = List.first(result.guides)

Map.take(guide_entry, ["id", "title", "kind", "meta"])

The parsed body lives under "ast" in the in-memory result.

guide_entry["ast"] |> List.first()

Inspect Livebook extraction

Livebooks are indexed from .livemd files. Unlike guides, they do not use YAML frontmatter; Livebook owns the top of the file, and DocShell takes the id from the filename and the title from the first Markdown H1.

livebook_entry = List.first(result.livebooks)

Map.take(livebook_entry, ["id", "title", "kind", "meta"])

Code cells remain source text in the AST. DocShell is an extractor, not a notebook runner.

livebook_entry["ast"]
|> Enum.find(fn node ->
  node["tag"] == "pre"
end)

Inspect the OpenAPI output

If no OpenAPI adapter is configured, DocShell still emits a valid empty OpenAPI 3.1 document. That means renderers can always expect openapi.json to exist.

Map.take(result.openapi, ["openapi", "info", "paths"])

Use the OpenAPI adapters notebook for Ash, OpenApiSpex, raw JSON, and custom adapter examples.

Inspect the presentation indexes

The presentation layer turns all extracted entries into three renderer-facing indexes:

• navigation — what to list
• search — what to index
• content — what to render for a page

Map.keys(result.presentation) |> Enum.sort()

Navigation items are structs in memory. The default generator keeps the tree flat because hierarchy is a host product decision.

result.presentation.navigation
|> Enum.map(&Map.take(&1, [:id, :title, :path, :kind, :children]))

Search entries contain the same identity and path information plus flattened plain text content. Tokens are present but empty unless search_tokens: true is set.

result.presentation.search
|> Enum.map(&Map.take(&1, [:id, :title, :path, :kind, :audience, :locale, :tokens]))

Content maps ids to AST node lists. The body is stored once, keyed by id, rather than duplicated in the navigation and search indexes.

Map.keys(result.presentation.content) |> Enum.sort()

Inspect the written files

Because write defaults to true, the build also wrote an artifact tree.

Path.wildcard(Path.join(public_dir, "*.json"))
|> Enum.map(&Path.basename/1)
|> Enum.sort()

Use DocShell.Artifact.read/1 to read an artifact payload. It validates the envelope before returning the "data" field.

{:ok, manifest} = DocShell.Artifact.read(Path.join(public_dir, "manifest.json"))

Map.take(manifest, ["artifacts"])

The private directory currently receives only its own manifest. Hosts can use the public and private directories with different cache processes and authorization rules.

DocShell.Artifact.read(Path.join(private_dir, "manifest.json"))

Skip file writes when you only need data

Hosts that ingest documentation into a database or knowledge graph can keep the in-memory result and skip the files.

dry_workspace =
  Path.join(
    System.tmp_dir!(),
    "doc_shell_build_pipeline_dry_#{System.unique_integer([:positive])}"
  )

dry_public_dir = Path.join(dry_workspace, "public")
dry_private_dir = Path.join(dry_workspace, "private")

dry_opts =
  build_opts
  |> Keyword.put(:public_dir, dry_public_dir)
  |> Keyword.put(:private_dir, dry_private_dir)
  |> Keyword.put(:write, false)

{:ok, dry_result} = DocShell.Build.run(dry_opts)

%{
  returned_keys: Map.keys(dry_result) |> Enum.sort(),
  wrote_public_dir?: File.exists?(dry_public_dir)
}

Fail loudly on invalid sources

Extraction stops at the first configured source that cannot be read. The error names the module or file at fault. That is intentional: silently losing a page is worse than failing the build.

DocShell.Build.run(Keyword.put(build_opts, :modules, [Nonexistent.Module]))

Guide frontmatter is treated the same way. A malformed configured guide returns an error instead of disappearing from the documentation set.

bad_guide_dir = Path.join(workspace, "bad-guides")
File.mkdir_p!(bad_guide_dir)

File.write!(Path.join(bad_guide_dir, "broken.md"), """
---
title: Broken

# This frontmatter never closes
""")

DocShell.Build.run(
  build_opts
  |> Keyword.put(:modules, [])
  |> Keyword.put(:guide_bases, [bad_guide_dir])
)

Build configuration checklist

When wiring DocShell into a host application, make these decisions explicitly:

The important boundary is simple: DocShell owns extraction and artifact shapes. Routing, visual hierarchy, authorization, and rendering stay in the host.

Changelog source validation

Every changelog source entry must be a valid entry map; nil and other invalid entries return {:error, {:invalid_changelog_entry, entry}}.

Guide line endings

YAML frontmatter accepts LF, CRLF, and CR line endings, including a closing --- delimiter at end of file.

Document identities

Document IDs must be nonempty and unique across all sources, including entries filtered from presentation. Duplicate IDs return an error naming both sources. Overlapping guide directories extract each normalized path once.

JSON metadata normalization

Metadata preserves JSON scalars and uses UTF-8 string keys. Unsupported terms become inspected text; improper list tails become a final array value. DocShell.Json.normalize/1 rejects converted-key collisions. The legacy stringify/1 keeps string keys when a collision occurs. Guides use normalize/1.

Configuration errors

Build.run/1 rejects malformed options and unknown per-call keys before extraction. Unknown application environment keys remain ignored. Invalid guide identities, titles, audience, and locale return errors naming the field and file. Guide IDs and titles accept nonempty strings or numeric/boolean scalars.

Output destinations

Public and private output directories must be disjoint. The optional raw OpenAPI destination must lie outside both. Conflicting paths fail before extraction or writes; use dedicated directories without symlink aliases.

Markdown titles

Guide and notebook titles come from the first top-level parsed H1, including Setext headings. Inline formatting is flattened, and headings inside code examples are ignored. Explicit guide frontmatter titles still take precedence.

Changelog Markdown context

Changelogs are parsed as complete Markdown documents before splitting on top-level release headings. Code examples remain within their release, and reference links resolve across the whole document. Parse errors in any part of the source return a source-tagged error.

Failed builds and recovery

Builds stage all JSON and back up existing files before publishing. Returned publication failures restore earlier files; rollback failures report retained backup paths. Cooperating builds use .doc-shell-build.lock directories. After a process or machine crash, recover retained backups and remove stale locks before rebuilding. Files still publish individually, so cache reloads validate generation IDs and keep the last complete snapshot. Use dedicated output directories without external writers or symlink aliases.