Working with Python

Mix.install(
  [
    {:pythonx, "~> 0.4.9"},
    {:kino_pythonx, "~> 0.1.0"},
    {:kino_db, "~> 0.4.0"},
    {:adbc, ">= 0.0.0"}
  ],
  config: [adbc: [drivers: [:duckdb]]]
)

[project]
name = "project"
version = "0.0.0"
requires-python = "==3.13.*"
dependencies = [
  "numpy==2.2.3",
  "matplotlib==3.10.1",
  "pandas==2.2.3",
  "polars==1.24.0",
  "altair==5.5.0",
  "plotly==6.0.0",
  "seaborn==0.13.2",
  "pillow==11.1.0",
  "pyarrow==23.0.0"
]

Introduction

Besides Elixir, Livebook supports running Python code. Not only that, it allows you to mix Elixir and Python code seamlessly. In this notebook we will explore various ways in which you can use Python in your notebooks.

To enable Python in a new notebook, you just need to click + Python button right below the setup cell. This does two things:

• adds :pythonx as an Elixir dependency, which embeds a fully fledged Python interpreter directly in Elixir
• inserts a pyproject.toml cell, where you can configure all the Python packages you need, as detailed in the uv package manager documentation

By explicitly listing the required dependencies, Livebook knows exactly what to install, making the environment easily reproducible, no global pip installs required!

Having done that, you can now insert Python cells and write typical Python code.

> Note that Python cells also offer intellisense such as module/function/variable completion and on-hover documentation, which you can test throughout the cells below.

Visual representation

As you can see in the setup section above, this notebook already lists a number of popular Python packages as dependencies. Livebook provides rich rendering for a variety of object, such as charts and dataframes, which we are going to see in this section.

Polars

Polars dataframes.

import polars as pl
import datetime as dt

df = pl.DataFrame(
  {
    "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
    "birthdate": [
      dt.date(1997, 1, 10),
      dt.date(1985, 2, 15),
      dt.date(1983, 3, 22),
      dt.date(1981, 4, 30),
    ],
    "weight": [57.9, 72.5, 53.6, 83.1], # (kg)
    "height": [1.56, 1.77, 1.65, 1.75], # (m)
  }
)

df

Pandas

Pandas dataframes.

import pandas as pd
import datetime as dt

> Note: the cells with import may take a while the first time you run them. That’s an expected Python behaviour when importing a large module tree. Python stores a cached module bytecode in disk, so even if you restart the notebook, subsequent imports of that module should be fast.

df = pd.DataFrame(
  {
    "name": ["Alice Archer", "Ben Brown", "Chloe Cooper", "Daniel Donovan"],
    "birthdate": [
      dt.date(1997, 1, 10),
      dt.date(1985, 2, 15),
      dt.date(1983, 3, 22),
      dt.date(1981, 4, 30),
    ],
    "weight": [57.9, 72.5, 53.6, 83.1], # (kg)
    "height": [1.56, 1.77, 1.65, 1.75], # (m)
  }
)

df

Matplotlib

Matplotlib plots.

import matplotlib.pyplot as plt

plt.plot([1, 2], [1, 2])
plt.gcf()

import numpy as np

x = np.linspace(0, 10, 100)
y = np.sin(x)

plt.plot(x, y, 'b-', label='sine wave')
plt.title('Simple Sine Wave')
plt.xlabel('x')
plt.ylabel('sin(x)')
plt.legend()
plt.grid(True)

plt.gcf()

Seaborn

seaborn plots.

import seaborn as sns

df = sns.load_dataset("penguins")

sns.pairplot(df, hue="species")

Vega-Altair

Vega-Altair charts.

import altair as alt
import pandas as pd

source = pd.DataFrame({
  "a": ["A", "B", "C", "D", "E", "F", "G", "H", "I"],
  "b": [28, 55, 43, 91, 81, 53, 19, 87, 52]
})

alt.Chart(source).mark_bar().encode(x="a", y="b")

Plotly

Plotly graphs.

import plotly.express as px

px.line(x=["a", "b", "c"], y=[1, 3, 2], title="Sample figure")

import pandas as pd

df = pd.DataFrame({"a": [1, 3, 2], "b": [3, 2, 1]})

px.bar(df)

Pillow

Pillow images.

from PIL import Image
from urllib.request import urlopen

url = "https://github.com/elixir-nx/nx/raw/v0.9/nx/nx.png"
Image.open(urlopen(url))

Elixir interoperability

In Livebook you can mix Elixir and Python code together. For example, let’s define a variable in an Elixir cell:

elixir_variable = 1

We can access it under the same name in a Python cell:

elixir_variable

> Hint: the language is shown in the bottom-right corner of each code cell. You can change the language by clicking the language icon. When inserting a new cell, you can also change the langauge by clicking on the dropdown icon.

An important implication of the variable interoperability is that you can use Kino inputs to provide data for Python code.

range_input = Kino.Input.range("How cool is that?")

number = Kino.Input.read(range_input)

if number % 3 == 0 and number % 5 == 0:
  print("FizzBuzz")
elif number % 3 == 0:
  print("Fizz")
elif number % 5 == 0:
  print("Buzz")
else:
  print(number)

Note that as you change the slider value, the cell indicator in the bottom-right changes to Stale. The value change tracking propagates from Elixir all the way to the Python cell!

One gotcha here is that sometimes the default encoding may not be what you’d expect. Specifically, this is the case with strings:

hello_from_elixir = "Hello!"

hello_from_elixir

Elixir does not distinguish strings and binaries as separate data types - strings are just binaries, with the expectation that they follow utf-8 encoding. When passed to Python, a string therefore becomes a bytes object. However, converting it to a Python string is as easy as decoding the bytes:

hello_from_elixir.decode("utf-8")

ADBC and data processing

Here we are going to have a look at another type of interoperability, by combining Elixir and Python libraries.

Elixir has a adbc package, with bindings for Arrow Database Connectivity (ADBC). ADBC provides a standard interface for querying databases and getting results in the Apache Arrow format.

In fact, the Livebook’s “Database connection” smart cell, already uses ADBC for some of the databases. Let’s connect to an in-memory DuckDB for the sake of this example:

:ok = Adbc.download_driver!(:duckdb)
{:ok, db} = Kino.start_child({Adbc.Database, driver: :duckdb})
{:ok, conn} = Kino.start_child({Adbc.Connection, database: db})

Now, we can query the database, but since there is no data, let’s use SELECT with VALUES.

In particular, because we want to further process the query result in Python, let’s use the Adbc.Connection.py_query/3 function. This will give use a Python table object that can be efficiently consumed by Python code.

{:ok, py_table} =
  Adbc.Connection.py_query(
    conn,
    """
    SELECT *
    FROM (VALUES
      (1, 'Alice',   'Engineering', 92000.00),
      (2, 'Bob',     'Marketing',   78000.00),
      (3, 'Carol',   'Engineering', 105000.00),
      (4, 'Dave',    'HR',          65000.00),
      (5, 'Eve',     'Marketing',   83000.00),
      (6, 'Frank',   'Engineering', 98000.00)
    ) AS employees(id, name, department, salary)
    ORDER BY department, salary DESC
    """,
    []
  )

The object has type pyarrow.Table and can be easily converted into a dataframe using the polars library:

df = pl.from_arrow(py_table)
df

which we can operate on as usual:

summary = (
  df
  .group_by("department")
  .agg([
    pl.col("salary").cast(pl.Float64).mean().alias("avg_salary"),
    pl.col("salary").max().alias("max_salary"),
    pl.col("name").count().alias("headcount"),
  ])
  .sort("avg_salary", descending=True)
)

summary

Finally, if we want to get the result back as Elixir data structure, we can call Adbc.Result.from_py/1.

{:ok, result} = Adbc.Result.from_py(summary)
Adbc.Result.to_map(result)

result |> Table.to_rows() |> Enum.to_list()

Distributed Python with FLAME

FLAME makes it easy to dynamically start additional machines and run code on them. Our Python integration is fully interoperable with FLAME, allowing you to run distributed Python code and keep references to Python objects across machines.

Livebook comes with a FLAME runner smart cell for configuring and starting a runner pool, however for that to work, you need to run the given notebook using either Fly or Kubernetes runtime. You can click the runtime icon in the sidebar to explore further.

The rest of this section will assume you have either a Fly or k8s runtime setup. Once that’s done, running computation on another machine is as simple as:

result =
  FLAME.call(:runner, fn ->
    1 + 1
  end)

Now, when using the FLAME runner smart cell, the runner is automatically initialized with Python environment matching the notebook’s one. This means we can also evaluate Python code in the runner, without any other setup, it would look something like this:

result =
  FLAME.call(:runner, fn ->
    ~PY"""
    1 + 1
    """
  end)

> Note that the ~PY sigil is a convenience for running Python code and it implicitly picks up any variable referenced in the code snipppet. You could also use the more explicit Pythonx.eval/3 API.

The above call would return a remote Python object. A FLAME runner typically terminates if idle for a while, but since the Python object resides on that runner, the runner is kept around as long as we have a reference to the object!

This can be used to coordinate Python work across multiple machines, You can load a model onto a runner with a GPU and get a reference to the remote model object (keeping the model itself on the runner). You could subsequently use FLAME.call/2 to run inference thorugh that model.

Wrapping up

As you can see, there are many levels of interoperability between Elixir and Python. In your notebooks, you can mix and match code from both languages. In turn, you get access to the extensive ecosystem of Python packages for enhancing your data workflows, while still being able to orchestrate it with Elixir.

Enjoy!