Notesclub

created by hec & contributors

terms privacy

Supervisors

reading/supervisors.livemd

@ammar-mohamed-massoud

Elixir-DockYard

Share to X

Share to Bluesky

More notebooks

Supervisors

Mix.install([
  {:jason, "~> 1.4"},
  {:kino, "~> 0.8.0", override: true},
  {:youtube, github: "brooklinjazz/youtube"},
  {:hidden_cell, github: "brooklinjazz/hidden_cell"}
])

Navigation

Return Home Report An Issue

Review Questions

What kinds of problems do supervisors solve?
How do we use supervisors to make our application fault-tolerant?
Why might we use each supervisor restart strategy (:one_for_one, :one_for_all, :rest_for_one)?

Setup

Ensure you type the ea keyboard shortcut to evaluate all Elixir cells before starting. Alternatively, you can evaluate the Elixir cells as you read.

Fault Tolerance

Elixir and Open Telecom Platform (OTP) are fantastic for building highly concurrent and fault-tolerant systems which can handle runtime errors.

Errors happen. We often implement specific error handling for the errors we expect. However, it’s impossible to anticipate every possible error. For unexpected errors, Elixir follows a let it crash philosophy.

To achieve a fault-tolerant system in Elixir, we leverage concurrency and isolate our system into different processes. These processes share no memory, and a crash in one process cannot crash another.

Therefore, we can let a process crash, and it will not affect the rest of our system.

> it’s somewhat surprising that the core tool for error handling is concurrency. > In the BEAM world, two concurrent processes are > completely separated; they share no memory, and a crash in one process can’t by default compromise the execution flow of another. > Process isolation allows you to confine the negative effects of an error to a single process or a small group of related processes, > which keeps most of the system functioning normally. > > * Sasa Juric, Elixir in Action.

For example, We can spawn a process and raise an error. Unless we explicitly link the process with spawn_link/1 The process will die and leave the parent process unaffected.

pid =
  spawn(fn ->
    raise "error"
  end)

# allow the process time to crash
Process.sleep(100)

"I still run."

When a linked process crashes, it will also crashed the process that spawned it. Uncomment the code below, and you’ll see it crashes the current Livebook process. Re-comment the code when finished.

# spawn_link(fn -> raise "error" end)

That means if we start a GenServer (or other process) unsupervised it will raise an error if it crashes.

defmodule UnsupervisedWorker do
  use GenServer

  def init(state) do
    {:ok, state}
  end

  def handle_info(:crash, _state) do
    raise "crash!"
  end
end

Uncomment and run the following code to see it crashes the current Livebook process, then re-comment the code.

# {:ok, pid} = GenServer.start_link(UnsupervisedWorker, [])
# Process.send(pid, :crash, [])

The same is true if we manually exit a process using Process.exit/2 with the :exit reason. Uncomment and run the following code to see it crashes the current Livebook process, then re-comment the code.

# {:ok, pid} = GenServer.start_link(UnsupervisedWorker, [])
# Process.exit(pid, :exit)

Supervisors

Instead of crashing the parent process, we can use Supervisors to monitor processes and restart them when they die.

The term restart is a bit misleading. To be clear, we cannot restart a process. Restart in this case means that we kill the process and start a new one in its place. However, it’s conceptually easier to think of it as restarting.

Why would we want to restart the process? Well, most technical issues are the result of state. Our system gets into a bad state and cannot recover. Turning a system on and off again clears the state and often resolves the issue. Using a Supervisor is like adding an automatic on/off switch to your application.

A Supervisor monitors one or more child processes. We generally refer to these processes as workers. The Supervisor can use different strategies for restarting its child workers. The :one_for_one strategy individually restarts any worker that dies.

flowchart
S[Supervisor]
W1[Worker]
W2[Worker]
W3[Worker]

S --> W1
S --> W2
S --> W3

Here, we define a simple Worker process which we’ll start under a supervisor.

defmodule Worker do
  use GenServer

  def start_link(_opts) do
    GenServer.start_link(__MODULE__, [], [])
  end

  def init(state) do
    {:ok, state}
  end
end

Every supervisor is also a process. The Supervisor.start_link/2 function accepts a list of child processes and starts the supervisor process. We provide each child as a map with an :id and a :start signature.

children = [
  %{
    id: :worker1,
    start: {Worker, :start_link, [1]}
  },
  %{
    id: :worker2,
    start: {Worker, :start_link, [2]}
  },
  %{
    id: :worker3,
    start: {Worker, :start_link, [3]}
  }
]

{:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_one)

Your Turn

Here, we’ve used the livebook-exclusive Kino.Process.sup_tree/2 function to create a visualization of the above supervision tree.

Kino.Process.sup_tree(supervisor_pid)

When we kill one of the processes under the supervision tree, it will be restarted. For demonstration purposes only, you can use the :c.pid/3 function from erlang to manually create the pid from the diagram above where 727 should be a pid from the diagram above.

pid = :c.pid(0, 727, 0)

Then, kill the process using Process.exit/2.

Process.exit(pid, :exit)

Re-evaluate the diagram above and you should see a new Worker process started with a different pid. Repeat this process to kill each process under the supervisor and see them restarted.

Restart Strategies

Supervisors can have different restart strategies that change the behavior of how we handle restarting child processes when one of them dies.

We’ve seen the above :one_for_one strategy. This is the most common strategy, where we restart the child process that died under the supervisor.

However, sometimes we want different restart strategies such as :one_for_all and :rest_for_one.

One-For-One

Restart only the worker that crashed.

flowchart TD 
    Supervisor
    Supervisor --> P1
    Supervisor --> P2
    Supervisor --> P3
    Supervisor --> ..
    Supervisor --> Pn

    classDef crashed fill:#fe8888;
    classDef restarted stroke:#0cac08,stroke-width:4px

    class P2 crashed
    class P2 restarted

In the diagram above, only P2 crashed, so only P2 will be restarted by the supervisor.

One-For-All

Restart all of the child workers.

flowchart TD 
    Supervisor
    Supervisor --> P1
    Supervisor --> P2
    Supervisor --> P3
    Supervisor --> ..
    Supervisor --> Pn

    classDef crashed fill:#fe8888;
    classDef terminated fill:#fbab04;
    classDef restarted stroke:#0cac08,stroke-width:4px

    class P2 crashed
    class P1,P3,..,Pn terminated
    class P1,P2,P3,..,Pn restarted

P2 crashed, which led to all the other child processes to be terminated, then all the child processes are restarted.

Rest-For-One

Restart child workers in order after the crashed process.

flowchart TD 
    Supervisor
    Supervisor --> P1
    Supervisor --> P2
    Supervisor --> P3
    Supervisor --> ..
    Supervisor --> Pn

    classDef crashed fill:#fe8888;
    classDef terminated fill:#fbab04;
    classDef restarted stroke:#0cac08,stroke-width:4px

    class P2 crashed
    class P3,..,Pn terminated
    class P2,P3,..,Pn restarted

P2 crashed, the child processes after it in the start order are terminated, then P2 and the other terminated child processes are are restarted.

Restart Strategy Examples

To better understand how we can use a supervisor with different restart strategies, we’re going to supervise multiple Bomb processes that will crash after a specified amount of time.

Here, we’ve created a Bomb process that will send itself a message to cause it to crash after a specified :bomb_time.

defmodule Bomb do
  use GenServer

  def start_link(opts) do
    GenServer.start_link(__MODULE__, opts)
  end

  @impl true
  def init(opts) do
    bomb_time = Keyword.get(opts, :bomb_time, 2000)
    name = Keyword.get(opts, :name, 2000)

    Process.send_after(self(), :explode, bomb_time)
    {:ok, name}
  end

  @impl true
  def handle_info(:explode, name) do
    raise "#{name} Exploded!"
  end
end

Each Bomb will exist for a specified time and then crash (explode).

Bomb 1 will explode after 2 seconds.
Bomb 2 will explode after 3 seconds.
Bomb 3 will explode after 5 seconds.

We’ll reuse these same children in all of our examples.

children = [
  %{
    id: :bomb1,
    start: {Bomb, :start_link, [[name: "Bomb 1", bomb_time: 2000]]}
  },
  %{
    id: :bomb2,
    start: {Bomb, :start_link, [[name: "Bomb 2", bomb_time: 3000]]}
  },
  %{
    id: :bomb3,
    start: {Bomb, :start_link, [[name: "Bomb 3", bomb_time: 5000]]}
  }
]

One For One

We’ll start the three bombs using the :one_for_one strategy.

For the sake of example, we’ve increased :max_restarts to increase the limit of three crashes per five seconds to five crashes per five seconds.

You can go to Supervisor Options on HexDocs for a full list of configuration options.

Each bomb restarts individually without affecting the others, so the timeline of crashes should be:

2 seconds: Bomb 1 crashes.
3 seconds: Bomb 2 crashes.
4 seconds: Bomb 1 crashes.
5 seconds: Bomb 3 crashes.

Uncomment and run the cell below to see the timeline of crashes, then re-comment it.

# {:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_one, max_restarts: 5)

One For All

If we change the restart strategy to :one_for_all, then all processes will be terminated and restarted when a single process crashes.

So the timeline of crashes will be:

2 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
4 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
6 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
8 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
10 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)

Notice that Bomb2 and Bomb3 will never explode because their timers restart.

Uncomment the line below to observe the timeline of crashes, then re-comment it.

# {:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :one_for_all, max_restarts: 5)

Rest For One

Rest for one works like a row of dominoes, where only children defined after the crashed process will be restarted.

If we change the restart strategy to :rest_for_one, the processes will restart any processes ordered after them in the supervisor.

At first, we won’t notice any change from the :one_for_all strategy. The timeline will still be:

2 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
4 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
6 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
8 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)
10 seconds: Bomb 1 crashes (Bomb 2 and Bomb 3 restarted)

Uncomment the line below to observe the timeline of crashes, then re-comment it.

# {:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :rest_for_one, max_restarts: 5)

However, if we change the order of children in the supervisor, only children after the crashed process will restart.

children = [
  %{
    id: :bomb3,
    start: {Bomb, :start_link, [[name: "Bomb 3", bomb_time: 5000]]}
  },
  %{
    id: :bomb1,
    start: {Bomb, :start_link, [[name: "Bomb 1", bomb_time: 2000]]}
  },
  %{
    id: :bomb2,
    start: {Bomb, :start_link, [[name: "Bomb 2", bomb_time: 3000]]}
  }
]

So the timeline of crashes would be:

2 seconds: Bomb1 crashes (Bomb2 restarted)
4 seconds: Bomb1 crashes (Bomb2 restarted)
5 seconds: Bomb3 crashes (Bomb1 and Bomb2 restarted)
7 seconds: Bomb1 crashes (Bomb2 restarted)
9 seconds: Bomb1 crashes (Bomb2 restarted)

# {:ok, supervisor_pid} = Supervisor.start_link(children, strategy: :rest_for_one, max_restarts: 5)

Your Turn

Start four Bomb processes under a supervisor with different bomb times. Try each restart strategy :one_for_one, one_for_all, and :rest_for_one. Predict the crash timeline then evaluate your code. Did the result match your prediction?

Syntax Sugar

Instead of providing a map with the :id and :start keys, we can instead provide the child as a tuple. The name of the module will also be the :id, and the second value will be the argument passed to start_link/1.

children = [
  {Bomb, [name: "Syntax Sugar Bomb", bomb_time: 1000]}
]

Uncomment the following code to see the crash timeline, then re-comment it.

# Supervisor.start_link(children, strategy: :one_for_one)

Mark As Completed

file_name = Path.basename(Regex.replace(~r/#.+/, __ENV__.file, ""), ".livemd")

save_name =
  case Path.basename(__DIR__) do
    "reading" -> "supervisors_reading"
    "exercises" -> "supervisors_exercise"
  end

progress_path = __DIR__ <> "/../progress.json"
existing_progress = File.read!(progress_path) |> Jason.decode!()

default = Map.get(existing_progress, save_name, false)

form =
  Kino.Control.form(
    [
      completed: input = Kino.Input.checkbox("Mark As Completed", default: default)
    ],
    report_changes: true
  )

Task.async(fn ->
  for %{data: %{completed: completed}} <- Kino.Control.stream(form) do
    File.write!(
      progress_path,
      Jason.encode!(Map.put(existing_progress, save_name, completed), pretty: true)
    )
  end
end)

form

Commit Your Progress

Run the following in your command line from the curriculum folder to track and save your progress in a Git commit. Ensure that you do not already have undesired or unrelated changes by running git status or by checking the source control tab in Visual Studio Code.

$ git checkout -b supervisors-reading
$ git add .
$ git commit -m "finish supervisors reading"
$ git push origin supervisors-reading

Create a pull request from your supervisors-reading branch to your solutions branch. Please do not create a pull request to the DockYard Academy repository as this will spam our PR tracker.

DockYard Academy Students Only:

Notify your teacher by including @BrooklinJazz in your PR description to get feedback. You (or your teacher) may merge your PR into your solutions branch after review.

If you are interested in joining the next academy cohort, sign up here to receive more news when it is available.