Persistence

Mix.install([
  {:kino, github: "livebook-dev/kino", override: true},
  {:kino_lab, "~> 0.1.0-dev", github: "jonatanklosko/kino_lab"},
  {:vega_lite, "~> 0.1.4"},
  {:kino_vega_lite, "~> 0.1.1"},
  {:benchee, "~> 0.1"},
  {:ecto, "~> 3.7"},
  {:math, "~> 0.7.0"},
  {:faker, "~> 0.17.0"},
  {:utils, path: "#{__DIR__}/../utils"}
])

Navigation

Return Home Report An Issue

Setup

Ensure you type the ea keyboard shortcut to evaluate all Elixir cells before starting. Alternatively you can evaluate the Elixir cells as you read.

Overview

This lesson serves as a brief primer on the many topics related to persistence. You will have the opportunity to practice these concepts throughout the course and develop a deeper knowledge.

Persistence

To persist means to continue to exist.

We often want to store information. For example, we might store some user information, or the particular state of the application.

For example, it would be unpleasant if you couldn’t save your emails.

Persistence is necessary for creating almost every application, though some purposely avoid persistence. Snapchat, for example, intentionally avoids persisting messages.

Short Term Persistence Vs. Long Term Persistence

We have a variety of methods to achieve persistence. We also have short term persistence and long term persistence.

Short-term persistence is usually in memory. We might have some value that persists during the runtime of our program, but as soon as the program ends, that value is lost.

For example, we can bind a variable in memory then use it in other Elixir cells.

i = 4

i + 2

But when the program ends and you turn off your computer and open a different livebook file, i no longer exists.

However, the .livemd file that powers this notebook persists. You could That’s because these notebooks use long-term persistence with a file system. write some code in the following Elixir cell, and it will still be there the next time you open this lesson.

State

Simple variable values are stored in memory, but they have no state.

The variable i never changes value during the execution of the program. We can rebind it in a particular scope, but we’re not mutating the value, so it’s not truly a persisting state, it’s more like binding a new variable value with the same name.

i = 4

When talking about short-term persistence, we generally think in terms of state.

For example, take a road light.

Utils.animate(:road_light)

It cycles from green to yellow to red

In other words, it has some state of the light. This state changes and the lights use that state to determine if they should be on or off.

stateDiagram
  [*] --> Green
  Green --> Yellow
  Yellow --> Red
  Red --> Green

Now that we have persistent data, we’re going to encounter issues with the state of our program. Let’s say we’re storing some state.

stateDiagram
    state "Initial State" as init
    [*] --> init

Then some other operation changes the state. So now, when we read the state, we receive a different value.

sequenceDiagram
    participant R as External
    participant S as State
    S->>S: Init State
    R->>S: Get State
    S->>R: Return State (Init State)

State introduces an entire class of bugs. What if we set the state at the wrong time? or read the state too fast or slow?

For example, you might set the state, then attempt to retrieve the state. However, the internal state hasn’t finished updating, so you retrieve the unchanged state rather than the new state you were expecting.

sequenceDiagram
    participant R as External
    participant S as State
    S->>S: Init State
    R->>S: Set State
    R->>S: Get State
    S->>R: Return State (Init State)
    S->>S: Second State

Have You Tried Turning It Off and On Again?

Have you noticed you can fix most technology issues by turning the device off and on again?

That’s because most bugs occur when the program gets into a bad state that it can’t get out of. You reset the state and solve the issue by turning it off and on again.

Your Turn

Let’s get practical. Here we have some File module functions that create files on your computer. You’re going to learn more about the File module in a future lesson. For now, trust that write/2 creates a file with some content, and read/1 reads that content.

First, we create a file. You can think of this file as storing the first state.

File.write("data/some_file", "first")

Then we read that file. We return the first state.

File.read("data/some_file")

Uncomment this File.write/1 line below and evaluate the Elixir cell.

Notice that the return value of File.read("data/some_file") has changed. The same is true if you re-evaluate the Elixir cell above.

# File.write("data/some_file", "second")
File.read("data/some_file")

File System Based Persistence

We can use a file system to persist information. For example, this entire course is saved in a series of .livemd files.

When working on real-world projects, it’s not recommended to use a file system for persisting long-term information unless you are hacking together a small personal project, or know what you are doing.

That’s because many issues arise with a file system that different tools could better handle.

Of course, there are plenty of times to use files. Most often, you’ll read information from them. For example, your company might be working on an email newsletter. Your sales team has entered a list of emails into an excel spreadsheet document.

If there’s only a handful of emails, you might manually copy paste these into your program, but if there are thousands, you’ll have to write some code to convert them into data that you can persist in your program.

DBMS (Database Management System)

Generally speaking, we use a Database to store long-term information in a program. The Database is a separate layer in the program that handles the complexity of storing information.

flowchart
    Program --> Database --> P[Persisted Data]

There are several different types of databases, here’s a great primer by Linux Academy.

Kino.YouTube.new("https://www.youtube.com/watch?v=Tk1t3WKK-ZY")

This course will focus on RDBMS (Relational Database Management Systems). We’ll provide a brief primer highlighting the core concepts you’ll need to learn to work with DataBases in Elixir. In later lessons, you’ll get hands-on experience to reinforce this overview.

Elixir applications generally default to using PostgreSQL which is an open-source relational database.

flowchart
Program --> PostgreSQL --> P[Persisted Data]

Relational databases store data as a series of tables with relationships to each other.

Think of a table as a spreadsheet with rows and columns. For example, when building out a photo app, we might have a table of users and a table of photos. Each tables stores relevant information for the user or photo.

Utils.table(:users_and_photos)

flowchart
P[PostgreSQL]
U[Users]
PH[Photos]
P --> U
P --> PH

There is a relationship between the data because some photos may belong to a user. Typically, the photo will store a foreign key which links to some user on the users table.

The foreign key is typically the id of the entry on another table.

flowchart
P[PostgreSQL]
U[Users]
PH[Photos]
PH1[Picture Of The Daily Bugle]
U1[Peter Parker]

P --> U --> U1
P --> PH --> PH1 --foreign_key--> U1

You can imagine those tables stored in the Database like so.

Utils.table(:user_photo_foreign_key)

SQL (Structured Query Language)

We can then query the Database for information using SQL. SQL is a Structured Query Language specifically for working with SQL based databases. We use SQL to communicate with PostgreSQL.

flowchart
E[Elixir Program] --> SQL --> PostgreSQL --> P[Persisted Data]

For a primer on SQL, check out Danielle Thé’s excellent video.

Kino.YouTube.new("https://www.youtube.com/watch?v=27axs9dO7AE")

Your Turn

Ensure you have PostgreSQL installed and run psql to start your local Database. Enter the following to create a users table, insert a user, and retrieve all users.

CREATE TABLE users (
    name varChar(255) NOT NULL
);

INSERT INTO users ('Peter Parker');

SELECT * from users;

Ecto

Ecto is a database library for Elixir. It comes with Phoenix, the framework for web development in Elixir.

Rather than deal directly with SQL in Elixir we generally use Ecto to interact with SQL databases.

flowchart
E[Elixir Program] --> Ecto --> SQL --> PostgreSQL --> P[Persisted Data]

You will have the opportunity to learn more about Ecto and get hands-on experience with databases throughout this course.

Commit Your Progress

Run the following in your command line from the project folder to track and save your progress in a Git commit.

$ git add .
$ git commit -m "finish persistence section"