Powered by AppSignal & Oban Pro

Search and read web pages

guides/ai/basic-web-browser.livemd

Search and read web pages

Use the browser tools in a ReAct Agent to find pages, read their text, and cite their URLs. Each Agent must declare which tools and destinations it needs.

Configure the host

Set the allowlist in your application's runtime configuration. To use Brave search, export BRAVE_SEARCH_API_KEY before you start Mix or the release. For local use, copy the root .env.example to .env and export the values from your local file. The default ReAct model uses OPENAI_API_KEY. For other models, see Configure ReqLLM.

# config/runtime.exs
config :jido, :browser,
  allow: [
    %{"host" => "hexdocs.pm", "path_prefix" => "/jido/"},
    %{"host" => "jido.hexdocs.pm", "path_prefix" => "/"}
  ],
  search: :brave,
  brave_api_key: System.get_env("BRAVE_SEARCH_API_KEY")

Each rule uses an exact DNS hostname. Use "/" to permit all paths on that host. A more specific path prefix must end with /. URLs under that prefix must use plain ASCII path segments. Encoded paths, dot segments, and duplicate slashes are rejected for path-scoped rules.

The host list and the Agent list must both permit a page. Jido checks each redirect and connects to a DNS address only after it checks the resolved addresses. Public HTTP and HTTPS on their standard ports are supported. The fixed Brave endpoint receives the search query and the key. The key stays out of Agent definitions, tool arguments, and page requests.

Read a page as Markdown

Add {:html2markdown, "~> 0.3.2"} to your application's Mix dependencies. Select Markdown in your host configuration.

# config/runtime.exs
config :jido, :browser,
  allow: [%{"host" => "jido.hexdocs.pm", "path_prefix" => "/"}],
  fetch_format: :markdown

The default is :text. With :markdown, HTML headings, links, tables, and code blocks keep their structure in the returned text field. Plain text pages continue to return plain text. If the host selects Markdown and the dependency is missing, fetch returns :html2markdown_unavailable.

Use a fetch-only Agent when you already have a URL. This example can run with your configured model and does not need a Brave key.

defmodule JidoDocsReader do
  use Jido.AI.Agent, name: "jido_docs_reader"

  ai do
    reasoning Jido.AI.Reasoning.ReAct do
      instructions "Read the page at the given URL and summarize its main sections."
    end

    tools do
      browser do
        allow "jido.hexdocs.pm"
        expose [:fetch]
      end
    end
  end
end

reader = JidoDocsReader.instantiate!(id: "docs-reader")
request = JidoDocsReader.request_signal!(%{input: "Read https://jido.hexdocs.pm/readme.html"})
{:ok, _reader, [%Jido.Agent.Directive.Reply{signal: reply}]} =
  JidoDocsReader.cmd(reader, request)

reply.data.output

Find and read a page with Brave

Use the Brave setup above. Add fetch_format: :markdown to that same configuration to return HTML as Markdown. This example uses both the Brave Search API and your configured model.

defmodule JidoDocsResearcher do
  use Jido.AI.Agent,
    name: "jido_docs_researcher",
    description: "Answer questions with sources from the Jido documentation."

  ai do
    reasoning Jido.AI.Reasoning.ReAct do
      instructions "Search for Jido documentation on HexDocs, read a result, and cite its final URL."
    end

    tools do
      browser do
        allow "hexdocs.pm", path_prefix: "/jido/"
        allow "jido.hexdocs.pm"
        expose [:search, :fetch]
      end
    end
  end
end

agent = JidoDocsResearcher.instantiate!(id: "docs-research")
signal = JidoDocsResearcher.request_signal!(%{input: "How does a Jido Agent process a Signal?"})
{:ok, _agent, [%Jido.Agent.Directive.Reply{signal: reply}]} =
  JidoDocsResearcher.cmd(agent, signal)

reply.data.output

Configure the ReAct model for your application before you run the Agent.

  • search_web(query) returns up to five permitted results. Each has a title, URL, and snippet. If the result list is empty, try a query specific to an allowed site.
  • web_fetch(url) returns url and text. Use the returned URL for citations because the page can redirect. Fetch accepts uncompressed UTF-8 HTML and plain text. Plain-text HTML extraction removes tags and common non-content elements and has limited entity decoding.

Each call has an eight-second deadline and a 512 KiB response body limit. Fetch follows up to three redirects and returns up to 6,000 bytes in either output format. Treat returned text as source data. The source can contain instructions.

If search reports :search_not_configured, check that the host selects :brave and that the key is available when runtime configuration runs.

Migrate from Jido Browser

Replace standalone Jido.Browser.Actions.SearchWeb and Jido.Browser.Actions.WebFetch tool declarations with the browser block. Move the Brave key to the host configuration above. Move destination restrictions to the host and Agent allowlists. For fetch only, use expose [:fetch] and supply only the host allowlist. Update consumers to read results from search and url and text from fetch.

Keep Jido Browser for sessions, tabs, page interaction, screenshots, JavaScript, rendering, and rich document extraction.