Axonplay

Introduction to Elixir

Elixir is a functional programming built on top of Erlang.

Elixir is successfully used in web development, embedded software, data ingestion, and multimedia processing, across a wide range of industries

Discord, Pinterest and PepsiCo have all used elixir to deploy low-latency, distributed, and fault-tolerant systems

"Hello, World!"

In Object Oriented Programming, we model the world using metaphors and analogies

In Functional Programming, we take data and endlessly transform it to what we want

Pipe Operator

name = IO.gets("Name: ")

name |> String.trim() |> String.upcase() |> String.downcase()

Immutability

Being built on top of Erlang means that Elixir will need to respect the immutability of data.

This not to say that variables cannot be reassigned it means that once a value is set its set

JAX

JAX is Autograd and XLA, brought together for high-performance numerical computing and machine learning research. It provides composable transformations of Python+NumPy programs: differentiate, vectorize, parallelize, Just-In-Time compile to GPU/TPU, and more.

JAX : Google’s Research Project on combining XLA and autograd

XLA : Accelerated Linear Algebra

autograd : Efficiently computes derivatives of numpy code.

Why talk about JAX

Functional

Immutable

Livebook

Livebook is where we will be working on!

• Shareable: notebooks are stored in the .livemd format, which is a subset of Markdown. This means your notebooks can be saved, easily shared, and play well with version control.

• Support for autocompletion, inline documentation, code formatting

• Livebook ensures your code runs in a predictable order, all the way down to package management. It also tracks your notebook state, annotating which parts are stale.

• Multiple users can work on the same notebook at once

Collaborative

Project Time 🚀

We will be working on the MNIST dataset, a large database of handwritten digits that is commonly used for training various image processing systems

We will be creating the “Hello world” of the Machine Learning world

Using the dataset, we will create a ML that predict the number written

A first look at a neural network

We will now take a look at a first concrete example of a neural network, which makes use of the Python library Keras to learn to classify hand-written digits.

The problem we are trying to solve here is to classify grayscale images of handwritten digits (28 pixels by 28 pixels), into their 10 categories (0 to 9). The dataset we will use is the MNIST dataset, a classic dataset in the machine learning community, which has been around for almost as long as the field itself and has been very intensively studied. It’s a set of 60,000 training images, plus 10,000 test images, assembled by the National Institute of Standards and Technology (the NIST in MNIST) in the 1980s.

In Depth Look at MNIST

For people who wanna dig deep into how this works

https://youtu.be/aircAruvnKk

Setting up

Mix.install([
  {:axon, "~> 0.1.0-dev", github: "elixir-nx/axon", branch: "main"},
  {:exla, "~> 0.1.0-dev", github: "elixir-nx/nx", sparse: "exla", override: true},
  {:nx, "~> 0.1.0-dev", github: "elixir-nx/nx", sparse: "nx", override: true},
  {:scidata, "~> 0.1.1"}
])

Nx (Numerical Elixir) Introduction

Nx is a multi-dimensional tensors library for Elixir with multi-staged compilation to the CPU/GPU

For Python developers, Nx currently takes its main inspirations from Numpy and JAX but packaged into a single unified library.

Nx.tensor([[1, 2], [3, 4]])

Transformers

Before training, we will preprocess our data by reshaping it into the shape that the network expects, and scaling it so that all values are in the [0, 1] interval. Previously, our training images for instance were stored in an array of shape (60000, 28, 28) of type uint8 with values in the [0, 255] interval. We transform it into a float32 array of shape (60000, 28 * 28) with values between 0 and 1.

require Axon

transform_images = fn {bin, type, shape} ->
  bin
  |> Nx.from_binary(type)
  |> Nx.reshape({elem(shape, 0), 784})
  |> Nx.divide(255.0)
  |> Nx.to_batched_list(32)
end

transform_labels = fn {bin, type, _} ->
  bin
  |> Nx.from_binary(type)
  |> Nx.new_axis(-1)
  |> Nx.equal(Nx.tensor(Enum.to_list(0..9)))
  |> Nx.to_batched_list(32)
end

Download MNIST Dataset

{train_images, train_labels} =
  Scidata.MNIST.download(transform_images: transform_images, transform_labels: transform_labels)

train_images
|> hd()
|> Nx.slice_axis(0, 1, 0)
|> Nx.reshape({1, 28, 28})
|> Nx.to_heatmap()

Modal

model =
  Axon.input({nil, 784})
  |> Axon.dense(128, activation: :relu)
  |> Axon.dropout()
  |> Axon.dense(10, activation: :softmax)

Here our network consists of a sequence of two Dense layers and a Dropout layer in between, which are densely-connected (also called “fully-connected”) neural layers. The third (and last) layer is a 10-way “softmax” layer, which means it will return an array of 10 probability scores (summing to 1). Each score will be the probability that the current digit image belongs to one of our 10 digit classes.

To make our network ready for training, we need to pick three more things, as part of “compilation” step:

• A loss function: this is how the network will be able to measure how good a job it is doing on its training data, and thus how it will be able to steer itself in the right direction.
• An optimizer: this is the mechanism through which the network will update itself based on the data it sees and its loss function.
• Metrics to monitor during training and testing. Here we will only care about accuracy (the fraction of the images that were correctly classified).

Training

final_training_state =
  model
  |> Axon.Training.step(:categorical_cross_entropy, Axon.Optimizers.adamw(0.005),
    metrics: [:accuracy]
  )
  |> Axon.Training.train(train_images, train_labels, epochs: 10, compiler: EXLA, log_every: 100)

Prediction

Pick a test image by choosing index number and take a look at how the model make the prediction.

test_images = train_images |> hd() |> Nx.slice_axis(10, 3, 0)

test_images[1]
|> Nx.reshape({1, 28, 28})
|> Nx.to_heatmap()

prediction =
  model
  |> Axon.predict(final_training_state[:params], test_images)
  |> Nx.argmax(axis: -1)

prediction[1]