Benchmarking

Mix.install([
  {:kino, github: "livebook-dev/kino", override: true},
  {:kino_lab, "~> 0.1.0-dev", github: "jonatanklosko/kino_lab"},
  {:vega_lite, "~> 0.1.4"},
  {:kino_vega_lite, "~> 0.1.1"},
  {:benchee, "~> 0.1"},
  {:ecto, "~> 3.7"},
  {:math, "~> 0.7.0"},
  {:faker, "~> 0.17.0"},
  {:utils, path: "#{__DIR__}/../utils"}
])

Navigation

Return Home Report An Issue

Setup

Ensure you type the ea keyboard shortcut to evaluate all Elixir cells before starting. Alternatively you can evaluate the Elixir cells as you read.

:timer

Out of the box, Erlang provides a :timer library which contains a tc/1 function to measure the time it takes to run a function. It returns a {time, result} tuple.

flowchart LR
Function --> timer[:timer.tc/1] --> return["{timer, result}"]

We’ll use Process.sleep/1 to simulate a slow function. Process.sleep/1 pauses the program’s execution for a number of milliseconds.

:timer.tc(fn -> Process.sleep(1000) end)

The time returned is measured in microseconds.

1000 microseconds equals 1 millisecond. 1000 milliseconds equals 1 seconds.

Therefore we can divide the number of microseconds by 1000 to get milliseconds, and the number of milliseconds by 1000 to get seconds.

You’ll notice that :timer is very close to accurate, but not perfectly accurate. This is to be expected with very small increments of time.

{microseconds, _result} = :timer.tc(fn -> Process.sleep(1000) end)

milliseconds = microseconds / 1000
seconds = milliseconds / 1000
IO.inspect(microseconds, label: "microseconds")
IO.inspect(milliseconds, label: "milliseconds")
IO.inspect(seconds, label: "seconds")

Your Turn

In the Elixir cell below, use :timer.tc/1 to measure the following slow_function. Replace nil with your answer.

slow_function = fn -> Process.sleep(1000) end

result = nil

Utils.feedback(:timer_tc, result)

Benchee

Benchee is a popular library for measuring performance and memory consumption.

External libraries need to be installed to use them.

We use Mix to install Benchee.

Mix is a build tool that provides tasks for creating, compiling, and testing Elixir projects, managing its dependencies, and more.

flowchart LR
Benchee
D[Dependencies]
I[install/2]
Mix
Mix --> I
I --> D
D --> Benchee

You’ll learn more about Mix in future lessons. For now, it’s enough to be aware that you won’t be able to use the Benchee library unless it’s installed.

We’ve installed Benchee for you for this lesson, but be aware if you try to use Benchee in other projects it likely isn’t installed.

We can use the Benchee.run/2 function to measure the performance and memory consumption of some function.

We also use Kino.nothing() to avoid cluttering your screen with additional information.

Benchee.run(%{
  "example test" => fn -> 10000 ** 10000 end
})

Kino.nothing()

Above, the most important part of the output should look like the following, but with different numbers.

Name                   ips        average  deviation         median         99th %
example test        154.65        6.47 ms    ±26.50%        6.13 ms       13.17 ms

Memory usage statistics:

Name            Memory usage
example test           600 B

The Benchee documentation explains how to read the output:

> average - average execution time/memory usage (the lower the better) > ips - iterations per second, aka how often can the given function be executed within one second (the higher the better - good for graphing), only for run times > deviation - standard deviation (how much do the results vary), given as a percentage of the average (raw absolute values also available) > median - when all measured values are sorted, this is the middle value. More stable than the average and somewhat more likely to be a typical value you see, for the most typical value see mode. (the lower the better) > * 99th % - 99th percentile, 99% of all measured values are less than this - worst case performance-ish

Using Benchee For Comparison

Let’s use Benchee to compare tuples and lists. Tuples are intended to be used as fixed-sized containers that are fast for accessing. Lists are collections intended for dynamic sized containers that get modified.

Therefore, lists should be slow to access, and tuples should be fast to access. That’s in theory, let’s verify our assumption and prove it to be true.

size = 10000
large_list = Enum.to_list(0..size)
large_tuple = List.to_tuple(large_list)

Benchee.run(%{
  "list" => fn -> Enum.at(large_list, size) end,
  "tuple" => fn -> elem(large_tuple, size) end
})

Upon running the above, you should see an output similar to the following. You’ll notice that accessing a list was far slower!

Name            ips        average  deviation         median         99th %
tuple        1.09 M        0.92 μs  ±2806.65%        0.80 μs           2 μs
list       0.0324 M       30.89 μs    ±93.31%       28.90 μs       67.80 μs

Comparison:
tuple        1.09 M
list       0.0324 M - 33.58x slower +29.97 μs

Your Turn

Use Benchee to compare accessing the first element instead of the last element as we already did above. You should notice the list is faster this time, or at least about the same speed. You’ll learn why in a future lesson.

Benchee.run(%{
  "access first element in list" => fn -> nil end,
  "access first element in tuple" => fn -> nil end
})

Commit Your Progress

Run the following in your command line from the project folder to track and save your progress in a Git commit.

$ git add .
$ git commit -m "finish benchmarking section"