ExArrow Quick Start

Get up and running with ExArrow in a few minutes. This notebook touches all three pillars: IPC (read/write Arrow streams), Arrow Flight (client/server), and ADBC (query databases and get Arrow results).

Configure adbc to use the SQLite driver (for section 5):

Application.put_env(:adbc, :drivers, [:sqlite])

db = Kino.start_child!({Adbc.Database, driver: :sqlite})
conn = Kino.start_child!({Adbc.Connection, database: db})
{:ok, result} = Adbc.Connection.query(conn, "SELECT 1 AS n, 'hello' AS msg")

1. Setup

Add ex_arrow to your Mix project and start a Livebook (or run in IEx). Here we assume the dependency is already in place.

# Verify the NIF loaded
ExArrow.native_version()

2. IPC: Read an Arrow stream

ExArrow can read Arrow IPC from a binary (e.g. from a file, socket, or HTTP body). We’ll use a small built-in fixture so you can run this without any external file.

# Get a small IPC stream binary (schema: id int64, name utf8; 2 rows)
{:ok, ipc_bytes} = ExArrow.Native.ipc_test_fixture_binary()

# Open a stream from the binary
{:ok, stream} = ExArrow.IPC.Reader.from_binary(ipc_bytes)

# Inspect schema (without consuming the stream)
{:ok, schema} = ExArrow.Stream.schema(stream)
ExArrow.Schema.fields(schema) |> Enum.map(& &1.name)

Read batches one at a time until the stream is done:

# First batch
batch1 = ExArrow.Stream.next(stream)
if batch1, do: ExArrow.RecordBatch.num_rows(batch1), else: nil

# Second call: no more batches
ExArrow.Stream.next(stream)

3. IPC: Write then read (roundtrip)

You can write the same schema and batches back to a binary or file, then read again.

# Rebuild stream from fixture and collect schema + all batches
{:ok, ipc_bytes} = ExArrow.Native.ipc_test_fixture_binary()
{:ok, stream} = ExArrow.IPC.Reader.from_binary(ipc_bytes)
{:ok, schema} = ExArrow.Stream.schema(stream)

batches =
  Stream.repeatedly(fn -> ExArrow.Stream.next(stream) end)
  |> Enum.take_while(&(is_struct(&1, ExArrow.RecordBatch)))

# Write to binary
{:ok, out_binary} = ExArrow.IPC.Writer.to_binary(schema, batches)
byte_size(out_binary)

4. Arrow Flight: Echo server and client

Arrow Flight is a gRPC-based protocol for streaming Arrow data. ExArrow includes a small echo server: you upload data with do_put, then download it with do_get using the ticket "echo".

Start the server (one cell), then use the client in the next.

## Start the built-in echo server on port 9999

{:ok, server} = ExArrow.Flight.Server.start_link(9999, [])
ExArrow.Flight.Server.port(server)

## Connect and upload our fixture data

{:ok, ipc_bytes} = ExArrow.Native.ipc_test_fixture_binary()
{:ok, stream} = ExArrow.IPC.Reader.from_binary(ipc_bytes)
{:ok, schema} = ExArrow.Stream.schema(stream)
batches = Stream.repeatedly(fn -> ExArrow.Stream.next(stream) end) |> Enum.take_while(&is_struct(&1, ExArrow.RecordBatch))

{:ok, client} = ExArrow.Flight.Client.connect("localhost", 9999, [])
:ok = ExArrow.Flight.Client.do_put(client, schema, batches)

## Download the same data via do_get

{:ok, down_stream} = ExArrow.Flight.Client.do_get(client, "echo")
{:ok, down_schema} = ExArrow.Stream.schema(down_stream)

Mix.install([ {:exarrow, path: Path.join(_DIR, “..”)}, {:rustler, “~> 0.32.0”, optional: true}, {:adbc, “~> 0.7”},

], config: [adbc: [drivers: [:sqlite]]] ) first = ExArrow.Stream.next(down_stream) ExArrow.RecordBatch.num_rows(first)

Clean up

ExArrow.Flight.Server.stop(server)

---

### 5. ADBC: Query a database, get Arrow results

With ADBC you open a database (e.g. SQLite), run SQL, and get an **Arrow stream** of result batches—same `ExArrow.Stream` API as IPC and Flight.

**Note:** ExArrow's ADBC layer uses the **C driver manager** and needs a **loadable shared library** (e.g. `libadbc_driver_sqlite.so`). The [`adbc`](https://hex.pm/packages/adbc) Hex package has its own process-based Database/Connection and native stack; its drivers are not necessarily loadable by ExArrow. So if opening with ExArrow fails below, we fall back to the adbc package's API so you still see a working query. To use ExArrow's ADBC (and get `ExArrow.Stream` batches), see [Installing an ADBC driver](INSTALL_ADBC_DRIVER.md).

Try ExArrow first (needs a driver loadable by the ADBC C driver manager)

ex_arrow_db = ExArrow.ADBC.Database.open(driver_name: “adbc_driver_sqlite”, uri: “:memory:”) |> case do

{:ok, _} = ok -> ok
{:error, _} -> ExArrow.ADBC.DriverHelper.ensure_driver_and_open(:sqlite, ":memory:")

end

case ex_arrow_db do {:ok, db} ->

# ExArrow path: Arrow stream of record batches
{:ok, conn} = ExArrow.ADBC.Connection.open(db)
{:ok, stmt} = ExArrow.ADBC.Statement.new(conn, "SELECT 1 AS n, 'hello' AS msg")
{:ok, stream} = ExArrow.ADBC.Statement.execute(stmt)
{:ok, schema} = ExArrow.Stream.schema(stream)
batch = ExArrow.Stream.next(stream)
IO.inspect(ExArrow.Schema.fields(schema), label: "Columns (ExArrow)")
IO.inspect(ExArrow.RecordBatch.num_rows(batch), label: "Rows")

->

# Fallback: adbc package's own API (works when the package is in deps)
if Code.ensure_loaded?(Kino) and Code.ensure_loaded?(Adbc.Database) do
  db = Kino.start_child!({Adbc.Database, driver: :sqlite})
  conn = Kino.start_child!({Adbc.Connection, database: db})
  {:ok, result} = Adbc.Connection.query(conn, "SELECT 1 AS n, 'hello' AS msg")
  IO.inspect(result, label: "Query result (adbc package API)")
  IO.puts("\n(ExArrow's ADBC could not load a driver. This uses the adbc package's Database/Connection. For ExArrow Arrow streams, install a standalone ADBC C driver: see livebook/INSTALL_ADBC_DRIVER.md)")
else
  IO.puts("ADBC driver not available: #{msg}")
  IO.puts("See livebook/INSTALL_ADBC_DRIVER.md for how to install a driver ExArrow can load.")
end

end

---

### Next steps

* **Notebook 01 — IPC**: Stream vs file format, reading from files, writing to files, schema and types.
* **Notebook 02 — Flight**: Full server/client API: `list_flights`, `get_flight_info`, `get_schema`, actions.
* **Notebook 03 — ADBC**: Metadata APIs, optional Explorer roundtrip, and production tips.

Docs: [hexdocs.pm/ex_arrow](https://hexdocs.pm/ex_arrow).