Would you like to see your link here? Contact us

Notesclub

created by hec & contributors

terms privacy

ExUnit

exunit.livemd

Ricardo Trejos

@cardotrejos

learning-elixir

More notebooks

ExUnit

Mix.install([
  {:kino, github: "livebook-dev/kino", override: true},
  {:kino_lab, "~> 0.1.0-dev", github: "jonatanklosko/kino_lab"},
  {:vega_lite, "~> 0.1.4"},
  {:kino_vega_lite, "~> 0.1.1"},
  {:benchee, "~> 0.1"},
  {:ecto, "~> 3.7"},
  {:math, "~> 0.7.0"},
  {:faker, "~> 0.17.0"},
  {:utils, path: "#{__DIR__}/../utils"}
])

Navigation

Return Home Report An Issue

Setup

Ensure you type the ea keyboard shortcut to evaluate all Elixir cells before starting. Alternatively you can evaluate the Elixir cells as you read.

Testing

We test to ensure our code behaves as expected.

Automated tests automate the process of manually testing our code and verify our code continues to behave as expected.

Without tests, it’s easy to introduce breaking changes. With tests, when something breaks, the automated test suite catches issue and facilitates debugging by providing useful feedback.

> We test to increase confidence that our software does what it’s > supposed to do. Testing gives us confidence that our code works as expected. > This is true for all kinds of testing, whether for automated tests performed > by a machine or for manual tests performed by a human. > > * Testing Elixir by Andrea Leopardi and Jeffrey Matthias

Test Behavior, Not Implementation

Tests can be brittle, meaning that as you change your codebase, tests will fail frequently. Typically, tests are brittle because they test implementation rather than behavior.

Think of behavior as the input and output of your function, and the implementation is the black box within it.

flowchart
  subgraph Behavior
    subgraph BB[Black Box]
      Implementation
    end
    Input --> BB --> Output
  end

Our tests should concern themselves with the behavior of our code, not the internal details. Easier said than done, however as a general rule if refactoring your code (making improvements without changing behavior) breaks your tests, then you are likely testing implementation.

Test Cases

Generally, we can split up our test suite into test cases. Each test case may require many assertions.

Let’s consider how we would test a Math module that abstracts the operators for adding different data types together, like so.

8 = Math.add(4, 4)
9.0 = Math.add(4.5, 4.5)
"ab" = Math.add("a", "b")
[1, 2] = Math.add([1], [2])
%{one: 1, two: 2} = Math.add(%{one: 1}, %{two: 2})
[one: 1, two: 2] = Math.add([two: 2], [two: 2])

Our Happy-path test cases (where our code is used as expected) could be the following.

adding two integers
adding two floats
adding two lists
adding two maps

We also want to consider edge-case test cases, also called sad-path or unhappy path when things go wrong, or the code is misused. For example, we might consider the following cases.

adding an integer and a list
adding an integer and a float
adding a list and a map

There can be a deceptive number of edge cases to consider. For example, we could build a growing list of edge-case permutations for each data type we want the Math module to handle.

Here, we’ve colored happy path tests green and edge-case tests yellow.

By planning test cases, we can anticipate possible edge cases and ensure we understand the desired behavior of the feature.

Test Driven Development

Many developers follow a practice of TDD (Test Driven Development), where they write the test for their feature before writing the implementation.

These developers would likely claim that TDD leads to better-tested code, simpler interfaces, and other benefits. However, other developers use different approaches, and it’s possible to take TDD too far or use it for the wrong situation.

It’s for you to make up your own mind about TDD, and this course aspires to be unbiased. However, we do believe that it’s beneficial for you to understand TDD so that it can be a tool in your toolbox should you choose to use it.

TDD has three main phases, often called Red, Green, and Refactor.

Red: Write a failing test.
Green: Write the code to make the test pass.
Refactor: Improve and optimize your code without changing the behavior.

You will have the opportunity to practice TDD throughout this course. For a deeper dive on TDD and testing, you might consider watching this video by Ian Cooper. It’s over an hour and completely optional.

Kino.YouTube.new("https://www.youtube.com/watch?v=EZ05e7EMOLM")

ExUnit

ExUnit is a built-in test framework for Elixir.

Let’s look at a very minimal test.

ExUnit.start(auto_run: false)

defmodule ExampleTest do
  use ExUnit.Case

  test "truthy" do
    assert true
  end
end

ExUnit.run()

First, we start ExUnit.

Next, we define a test module, ExampleTest. The name of this module does not matter. However, it’s common to end the module name in Test.

The next line, use ExUnit.Case, generates some code from the ExUnit.Case module, which gives us access to the test and assert macros.

test is a macro that defines a single test. test accepts the name of the test as a string, in this case "truthy".

assert is a macro that makes a single assertion inside of the test. The test passes if assert is given any truthy value and fails if it receives any falsy value. Here, we assert that true is truthy.

A test can contain many assertions, and a test module can have many tests.

Assertions

ExUnit provides many different assertion macros from the ExUnit.Assertion module. We’ll focus on the following common assertions.

assert check if a value is truthy.
refute check if a value is falsy.
assert_raise check if the code raises an error.

`assert`

We can use the assert macro provided by ExUnit to assert that a value is truthy.

ExUnit.start(auto_run: false)

defmodule AssertExampleTest do
  use ExUnit.Case

  test "assert examples" do
    assert true
    assert fn -> 2 end
    assert []
    assert 1
    assert %{}
  end
end

ExUnit.run()

assert/2 accepts a message as the second argument, which will display a message if the test fails.

ExUnit.start(auto_run: false)

defmodule AssertMessageExampleTest do
  use ExUnit.Case

  test "failing example" do
    assert false, "This message will display for a failing test."
  end
end

ExUnit.run()

`refute`

The refute macro is the opposite of the assert macro. It checks that a value is falsy (nil or false).

ExUnit.start(auto_run: false)

defmodule RefuteExampleTest do
  use ExUnit.Case

  test "refute examples" do
    refute false
    refute not true
    refute !5
    refute is_map(4)
  end
end

ExUnit.run()

refute/2 also displays a message if the test fails.

ExUnit.start(auto_run: false)

defmodule RefuteMessageExampleTest do
  use ExUnit.Case

  test "failing example" do
    refute true, "This message will display for a failing test."
  end
end

ExUnit.run()

Practical Assertions

Since assert tests if a value is truthy, we often use operators to compare the value with another or functions to verify some property of the value under test.

For example, we might use comparison operators (==, ===, >=, >, <=, <) to compare our received value with the expected value.

double = fn int -> int * 2 end
double.(2) == 4

It’s also common to use the match operator = to check if the value matches a specific pattern. Alternatively, we can also use the built-in match?/2 function.

map = %{one: 1, two: 2}

%{one: _} = map
match?(%{one: 1}, map)

The test-based match operator =~ can test if a regular expression matches a string.

string = "hello world"
string =~ "hello"

string =~ ~r/\w/

There are also many built-in functions, such as is_map, is_list, is_integer, and more which we can use to assert some property of the value under test.

These operators and functions can combine with assert to make more useful assertions.

ExUnit.start(auto_run: false)

defmodule AssertionExampleTest do
  use ExUnit.Case

  test "practical assertions example" do
    # comparison operators
    assert 2 == 2.0
    assert 2.0 === 2.0
    assert 1 < 2
    assert 2 > 1
    assert 2 >= 2
    assert 2 <= 2

    # match operator
    assert %{one: _} = %{one: 1, two: 2}
    assert [one: _, two: _] = [one: 1, two: 2]
    assert {_, _, _} = {1, 2, 3}
    assert [1 | _tail] = [1, 2, 3]

    # functions
    assert is_integer(2)
    assert is_map(%{})

    # text-based match operator
    assert "hello world" =~ "hello"
    assert "hello world" =~ ~r/\w+/
  end
end

ExUnit.run()

Assertions that use an operator provide a :left and :right value for feedback. Generally, :left should be the received value for the system under test, and :right should be the expected value for the system under test.

  1) test left/right == example (LeftRightExampleTest)
     reading/exunit.livemd#cell:6
     Assertion with == failed
     code:  assert 4 == 5
     left:  4
     right: 5
     stacktrace:
       reading/exunit.livemd#cell:7: (test)

ExUnit.start(auto_run: false)

defmodule LeftRightExampleTest do
  use ExUnit.Case

  test "left/right == example" do
    assert 4 == 5
  end

  test "left/right > example" do
    assert 4 > 5
  end

  test "left/right =~ example" do
    assert "hello world" =~ "non-matching pattern"
  end
end

ExUnit.run()

Your Turn

Putting everything we’ve learned so far together, you’re going to create a suite of tests for a Math module that handles adding similar data types together.

8 = Math.add(4, 4)
9.0 = Math.add(4.5, 4.5)
"ab" = Math.add("a", "b")
[1, 2] = Math.add([1], [2])
%{one: 1, two: 2} = Math.add(%{one: 1}, %{two: 2})
[one: 1, two: 2] = Math.add([two: 2], [two: 2])

Write the following happy-path test cases for the Math module in the Elixir cell below.

add two integers
add two floats
add two strings
add two lists
add two maps
add two keyword lists.

We’ve pre-made the test case to add two integers as a demonstration of the task. Notice we can omit the do block to make a "Not implemented" test.

defmodule Math do
end

ExUnit.start(auto_run: false)

defmodule MathTest do
  use ExUnit.Case

  test "add/2 integers" do
    assert Math.add(1, 1) === 2
  end

  test "add/2 floats"
  test "add/2 strings"
  test "add/2 lists"
  test "add/2 maps"
  test "add/2 keyword lists"
end

ExUnit.run()

Your Turn (Bonus)

Implement the Math module using a module, and then implement the Math module using protocols. Do your tests pass for both solutions? Ideally, none of your tests should depend on the underlying implementation of the Math module.

Example-Based Testing

Above, we’ve demonstrated example-based testing, in that we provide a single example of input and output, and assume that the system under test continues to behave as expected.

However, there are a potentially infinite number of inputs and outputs for the Math.add/2 function. Is it safe to assume that a single example comprehensively tests all inputs?

for int1 <- 1..3, int2 <- 1..3 do
  [input: int1, input: int2, output: int1 + int2]
end
|> Kino.DataTable.new()

For example, we can make the integers test pass with a fake solution.

defmodule Math do
  def add(_int1, _int2) do
    2
  end
end

ExUnit.start(auto_run: false)

defmodule MyTest do
  use ExUnit.Case

  test "add/2 integer + integer" do
    assert Math.add(1, 1) == 2
  end
end

ExUnit.run()

A robust test ensures the code behaves correctly under many scenarios. To improve the robustness of our tests, we could write more example-based tests or more assertions in the same test.

ExUnit.start(auto_run: false)

defmodule MathTest do
  use ExUnit.Case

  test "add/2 integer + integer" do
    assert Math.add(1, 1) === 2
    assert Math.add(1, 2) === 3
  end

  # Alternatively, we could separate this into two tests.
  test "add/2 1 + 1" do
    assert Math.add(1, 1) === 2
  end

  test "add/2 1 + 2" do
    assert Math.add(1, 2) === 3
  end
end

ExUnit.run()

This is more comprehensive. However, it’s still possible to fake the solution. In addition, if we continue this pattern, we will write more tests than necessary, which makes our test suite longer and potentially harder to maintain.

Randomized Test Data

Simple example-based tests have their place, but when we want a highly robust test, we can rely on randomized data.

We can generate random data ourselves or depend on an external library like Faker.

ExUnit.start(auto_run: false)

defmodule MathTest do
  use ExUnit.Case

  test "add/2 integer + integer" do
    # you may still choose to include a few example-based assertions.
    assert Math.add(1, 1) === 2
    assert Math.add(1, 2) === 3

    # Now all integers between 1 and 99 will be covered.
    int1 = Enum.random(1..99)
    int2 = Enum.random(1..99)
    assert Math.add(int1, int2) == int1 + int2
  end
end

ExUnit.run()

Types of Test

There are three primary types of tests.

Unit Test: tests that cover a single unit in isolation.
Integration Tests tests that cover the integration of multiple components.
End to End (E2E) tests that cover the whole system from start to finish.

So far, we have only seen examples of Unit Tests. In addition, the definition of a unit is often hotly debated in the programming community, so the lines between integration and unit tests are blurry.

We’ll have the opportunity to see various types of tests. For now, it’s sufficient to be aware that there are multiple types.

For more on integration tests, consider reading Integration Tests, a blog post by Martin Fowler.

For end-to-end testing, we’ll later use the popular Wallaby project, which can test web applications by simulating realistic user interactions.

Test Suite Structure

We want to have confidence in our test suite so that as we add features, refactor, and otherwise change your codebase, we are not at risk of making breaking changes.

There are several schools of thought on how you should comprehensively test applications.

Many developers believe the Test Pyramid creates a fast and comprehensive test suite with many unit tests, some integration tests, and fewer end-to-end tests.

Alternatively, you might consider the Test Trophy, which prioritizes integration tests.

Rather than recommend a specific pattern, we hope to introduce you to them so that you can develop your own opinions and do further research.

Test Naming

Naming tests is important, but also highly subjective. There are some common patterns such as Given, When, Then.

In general, a good test name should capture:

the system under test
the state under test
the desired behavior

All of the following would capture the desired information.

test "Given two strings, when calling add/2 with both strings, it concatenates the strings."
test "add/2 _ two strings _ concatenates the strings."

We might omit the desired behavior from the test name, as it is also implicitly stated by the assertions in the test. By omitting the desired behavior, we prevent the test name from becoming outdated when the desired behavior changes, at the cost of possibly reducing the immediate clarity of our test.

So the following test name could be appropriate:

test "add/2 strings"

Rather than dogmatically recommend a pattern, we’ll say that a good test name makes it easier to understand the goal of the test. Of course, how you do this is up to you, and many teams develop different patterns.

Test Organization

We can group tests into a describe block. A describe block organizes tests by some commonality. For example, it’s common to group tests for a single function in the module under test together.

ExUnit.start(auto_run: false)

defmodule DescribeExampleTest do
  use ExUnit.Case

  describe "Module.function1/1" do
    test "test 1" do
    end

    test "test 2" do
    end
  end

  describe "Module.function2/1" do
    test "test 1" do
    end

    test "test 2" do
    end
  end
end

ExUnit.run()

Your Turn

In the Elixir cell below, write the happy-path test cases for the Math.subtract/2 function in it’s own describe block.

subtract two integers
subtract two floats
subtract two strings
subtract two lists
subtract two maps
subtract two keyword lists.

In addition, add your previous happy-path test cases for the Math.add/2 function in it’s own describe block.

Math.subtract/2 should work as follows.

4 = Math.subtract(8, 4)
2.0 = Math.subtract(4.5, 2.5)
"b" = Math.subtract("ab", "a")
[1] = Math.subtract([1, 2], [2])
%{two: 2} = Math.subtract(%{one: 1, two: 2}, %{one: 1})
[one: 1] = Math.add([one: 2, two: 2], [two: 2])

defmodule Math do
end

ExUnit.start(auto_run: false)

defmodule MathTest do
  use ExUnit.Case

  describe "Math.add/2" do
    test "integers"
    test "floats"
    test "strings"
    test "lists"
    test "maps"
    test "keyword lists"
  end

  describe "Math.subtract/2" do
    test "integers"
    test "floats"
    test "strings"
    test "lists"
    test "maps"
    test "keyword lists"
  end
end

ExUnit.run()

Your Turn (Bonus)

Testing Mix Projects

Mix projects provide some conveniences when working with ExUnit.

Let’s see how we would test the Math module if it were a Mix project. First, create a new mix project called math.

$ mix new math

You should see an output similar to the following.

* creating README.md
* creating .formatter.exs
* creating .gitignore
* creating mix.exs
* creating lib
* creating lib/math.ex
* creating test
* creating test/test_helper.exs
* creating test/math_test.exs

Your Mix project was created successfully.
You can use "mix" to compile it, test it, and more:

    cd math
    mix test

Run "mix help" for more commands.

Tests in a mix project are in the test/ folder. Generally there is an equivalent file in the test/ folder for each tested file in the lib folder.

ExUnit is started for us in the test/test_helper.exs file, and files in the test/ folder automatically compile, evaluate, and execute when we run tests.

Run Tests

We can execute all tests by running the following from the command line in the project folder.

$ mix test

Run Tests in A File

We can run a single test file by providing its path.

$ mix test path/to/test/file.exs

For example, we can run the math_test.exs file.

$ mix test test/math_test.exs

Run Tests By Line Number

We can execute a specific test or several tests under a describe block by providing the line number.

$ mix test test/math_test.exs:5

Visual Studio Code Elixir Testing Extension

We highly recommend using the Elixir Test extension if you are using Visual Studio Code.

Elixir Test provides several commands which facilitate more straightforward test running.

Your Turn

Move the Math.add/2 and Math.subtract/2 test suite into the math_test.exs file. Then use the command line to experiment with running tests.

Run all tests from the command line.

$ mix test

Run tests in the math_test.exs file.

$ mix test test/math_test.exs

Run tests in the the Math.add/2 describe block. Replace 1 with the correct line number.

$ mix test test/math_test.exs:1

Run a single test. Replace 1 with the correct line number.

$ mix test test/math_test.exs:1

Test Tags

We can use @moduletag, @describetag, and @tag module attributes to tag our tests. Once tagged, we can configure ExUnit to exclude, include, or only run tests with specific tags.

See the documentation for more on ExUnit Configuration.

Notice below that the test suite passes because the failing test never runs.

ExUnit.start(auto_run: false)
# We can configure ExUnit to exclude, include, or only run certain tests.

ExUnit.configure(exclude: :my_module_tag)

defmodule TagTest do
  use ExUnit.Case

  @moduletag :my_module_tag
  test "failing test" do
    assert false
  end
end

ExUnit.run()

Your Turn

@moduletag tags all tests in a module, @descibetag tags all tests in a describe block, and @tag tags a single test.

We’ve provided several examples of configuring tests with tags in the cell below. Uncomment each configuration one by one and execute the tests to see which tests are included and excluded.

Feel free to write your own configure/1 function to experiment with configuration.

ExUnit.start(auto_run: false)

ExUnit.configure(exclude: :my_module_tag, include: :my_descibe_tag, include: :my_tag)
# ExUnit.configure(exclude: :my_descibe_tag, exclude: :my_tag)
# ExUnit.configure(only: :my_descibe_tag)
# ExUnit.configure(only: :my_tag)

defmodule TagTest do
  use ExUnit.Case

  @moduletag :my_module_tag

  test "failing module test" do
    assert false
  end

  describe "failing describe" do
    @describetag :my_describe_tag
    test "failing test" do
      assert false
    end
  end

  @tag :my_tag
  test "failing test" do
    assert false
  end
end

ExUnit.run()

Mix Project Tags

Once tagged, we can use the flags --only, --exclude, and --include to run specific tests.

$ mix test --exclude my_tag

Alternatively, we can place the ExUnit.configure/1 function in test_helpers.exs.

ExUnit.start()
ExUnit.configure(exclude: :my_tag)

Your Turn

Use the @moduletag, @describetag, and @tag module attributes in the math_test.exs module. Then use the --exclude, --include, and --only flags for the mix test command to only run certain tests.

Finally, configure the test_helper.exs file to exclude each tag individually and run all tests. Ensure tests with your skipped test do not run.

Documentation and Doc Testing

We use the @moduledoc and @doc module attributes to document our code in a mix project. By documenting our project, we explain to other developers the purpose of our code.

@moduledoc contains documentation for the entire module, and @doc documents the function below it.

When we created the Math mix project, it used the @doc and @moduledoc module attribute to document the main math.ex module.

defmodule Math do
  @moduledoc """
  Documentation for `Math`.
  """

  @doc """
  Hello world.

  ## Examples

      iex> Math.hello()
      :world

  """
  def hello do
    :world
  end
end

Notice the Examples section, which simulates an IEx shell. This is a Doctest. Doctests are run by the corresponding test file math_test.exs using the doctest macro from ExUnit.

ExUnit.start(auto_run: false)

defmodule MathTest do
  use ExUnit.Case
  doctest Math
  ...
end

ExUnit.run()

Doctests are generally not as comprehensive as typical testing and are not a full replacement. However, they can be a great way to TDD your code and ensure the documentation remains up to date.

Your Turn

Add Doctests for the math.ex the add/2 and subtract/2 function. Include an example for each data type (integers, floats, strings, lists, maps, keyword lists).

Generally, Doctests go in an Examples section. You can write multiple Doctests like so.

  @doc """
  add data types.

  ## Examples

      iex> Math.add(1, 1)
      2

      iex> Math.add(2.1, 2.1)
      4.2
  """

Commit Your Progress

Run the following in your command line from the project folder to track and save your progress in a Git commit.

$ git add .
$ git commit -m "finish exunit section"

Other notebooks:

@andyl

elix_util

Examples

vegalite.livemd

vega_lite jason

2022-8-18
Chris Martin
@trbngr

elixir_cqrs_tools

Using cqrs_tools with Commanded

commanded.livemd

commanded cqrs_tools ecto etso jason

2022-8-18
@TomBers

livebookNotes

Attractors

attractors.livemd

decimal vega_lite kino

2022-8-18
Kevin Pan
@feng19

spider_man

ElixirJobs

elixirjobs.livemd

spider_man floki nimble_csv kino

2022-8-18
@aforward-oss

empex_recommend

Talk

talk.livemd

nx scholar optimus explorer exla req kino_vega_lite kino kino_explorer scidata

2023-12-4
mrdotb
@mrdotb

resvg_nif

Resvg example

example.livemd

resvg req kino

2023-7-13
Digital Identity Labs
@Digital-Identity-Labs

smee_live

SmeeView Workbook

smee_view.livemd

smee_view rambo kino

2023-6-7

Back

ExUnit

exunit.livemd

ExUnit

Navigation

Setup

Testing

Test Behavior, Not Implementation

Test Cases

Test Driven Development

ExUnit

Assertions

assert

refute

Practical Assertions

Your Turn

Your Turn (Bonus)

Example-Based Testing

Randomized Test Data

Types of Test

Test Suite Structure

Test Naming

Test Organization

Your Turn

Your Turn (Bonus)

Testing Mix Projects

Run Tests

Run Tests in A File

Run Tests By Line Number

Visual Studio Code Elixir Testing Extension

Your Turn

Test Tags

Your Turn

Mix Project Tags

Your Turn

Documentation and Doc Testing

Your Turn

Further Reading

Commit Your Progress

Other notebooks:

`assert`

`refute`