Powered by AppSignal & Oban Pro
Would you like to see your link here? Contact us
Notesclub
Sign in with GitHub
created by hec & contributors
terms privacy

ElixirZone: Metaprogramming Elixir - James Lavin

metaprogramming_part_1.livemd

avatar
James Lavin
avatar
avatar
More notebooks
0

ElixirZone: Metaprogramming Elixir - James Lavin

No videos for 7 1/2 months?!?! Are you dead?

Fortunately, I’m NOT dead.

My job keeps me very busy

I’ve slowed down on ElixirZone to spend more of my nights and weekends building a product I’ve wanted to build for a decade.

I love preparing and sharing ElixirZone videos with you (though recording & editing is less fun) and really appreciate your appreciative, positive feedback, but my side project is even more exciting.

You promised us software scalability insights from biology!

This will also be slower than planned due to pouring most of my free time into my side project.

I have continued to learn from and be amazed by biology and know it holds many lessons for scaling computational systems.

If you’re interested, here are some great resources I have enjoyed:

RESEARCHERS:

BOOKS:

  • Donna Jackson Nakazawa, The Angel and the Assassin
  • Peter M. Hoffman, Life’s Ratchet (recommended by an ElixirZone subscriber. Thanks!)
  • Paul G. Falkowski, Life’s Engines
  • Beronda L. Montgomery, Lessons from Plants
  • Carl Zimmer, Life’s Edge
  • Lee Know, Mitochondria and the Future of Medicine
  • Steven Strogatz, Sync
  • Nick Lane, Power, Sex, Suicide (this prof is amazing… has authored a handful of great books)
  • Mark Humphries, The Spike
  • Athena Aktipis, The Cheating Cell
  • Benjamin Bikman, Why We Get Sick
  • Merlin Sheldrake, Entangled Life
  • Antonio Damasio, The Strange Order of Things
  • Jeremy DeSilva, First Steps
  • Walter Isaacson, The Code Breaker
  • Dean Buonomano, Your Brain Is a Time Machine
  • Alex Bezzerides, Evolution Gone Wrong
  • Giulia Enders, Gut
  • Michael Pollan, Cooked
  • Frans de Waal, Are We Smart Enough to Know How Smart Animals Are? (another amazing prof with multiple fabulous books)
  • Oliver Sacks, The Mind’s Eye (another prolific researcher/author)
  • Alex Korb, The Upward Spiral
  • Matthew Walker, Why We Sleep
  • Vivek H. Murthy, Together
  • Daniel E. Lieberman, Exercised

Elixir metaprogramming… Words of encouragement

  • Metaprogramming: “abus[ing] Elixir syntax for fun and profit” (Chris McCord, Metaprogramming Elixir, p. 104)

  • Metaprogramming is powerful but hard/confusing/frustrating to learn.

  • You won’t understand it all the first (or second or third) time through.

  • Learning metaprogramming is like seeing “Magic Eye” (https://www.magiceye.com/) the first time. Just keep staring and you’ll eventually get it!

  • Just keep watching videos and reading blog posts, Chris McCord’s book, etc., and you’ll slowly get there!

Elixir metaprogramming… Words of warning

  • You may not need this!

    • You can go very far with Elixir without deep knowledge of metaprogramming/macros (or OTP/concurrency)
    • The benefits of metaproogramming & concurrency * are baked into popular and powerful libraries and frameworks. E.g.:
      • Phoenix lets you spin up millions of simultaneous & completely isolated web socket connections by leveraging OTP
      • Phoenix lets you create routes dynamically through its easy-to-use DSL built with metaprogramming
      • Phoenix lets you generate powerful functions into your controllers with a single call to use
    • Phoenix’s HEEx templates are another powerful DSL
    • Ecto’s power and simplicity derive from its DSL, possible through metaprogramming

  • Learning metaprogramming requires contorting your brain in unfamiliar ways.

    • Learning (for me) requires seeing a bunch of examples
    • “Use it or lose it”

https://en.wikipedia.org/wiki/Woozle_effect

woozle

Image below from Mark Lewis Tweet: https://twitter.com/marklewismd/status/1576677683408228353?s=20&t=PaaLKiq_z2NxWGVehdSTUw

Learn & forget the Krebs cycle

  • I’m NOT an expert. I’ve done metaprogramming but only a tiny fraction as often as the true experts have.

  • This talk will RAMBLE ON AND ON because:

    • Metaprogramming is hard to grasp, so covering the same ground multiple times from multiple perspectives should help you grasp the concepts more fully

    • I’m too lazy to organize this rambling material better

  • EXPERTS are folks like these, who have done 10,000x as much metaprogramming as I have:

jose

McCord

wojtek

[DELETED: Photo of Wojtek & my son engaged in a serious ping pong battle at our house]

Maybe one of these true experts will record an expert-level metaprogramming video???

Would be great, but they’ve already given us SO much. Just an idea… NOT a request!

If I’m no expert, why am I presenting this?

We make excuses for not taking chances and putting ourselves out there, but we should…

My Excuse Why It’s Lame Implication For You
I don’t know everything and will waste people’s time Having just SOME knowledge can be an ADVANTAGE in teaching… Curse of Knowledge Each of YOU knows things well enough to teach those who know less
I don’t know everything and don’t want to embarrass myself “Imposter Syndrome” is very real. I know enough to help non-expert metaprogrammers Most of us suffer from “imposter syndrome” and perfectionism. Don’t let them win. Share what you DO know
The world is crumbling all around us Rising fascism, global warming/climate collapse, and Russia’s invasion & crimes against humanity in Ukraine are mostly beyond my control Each of us should try to make the world better and happier however we can, especially when things are bleak. Don’t cower at the size of the challenge. Do what you can and others will do what they can
I’ve been working on something super cool and de-prioritized this Okay, but it has been 7 1/2 months Prioritize & balance your life. Avoid going 100% on any one area

The Tweets that got me off my butt to finish this after 7+ months in mothballs: Solnica Joshi

Additional resources

decompile (Michał Muskała)

decompile_1 decompile_2

This works from the top-level of a Phoenix app, creating a decompiled Elixir.SiliconBrainWeb.Router.ex file in my top-level directory:

mix decompile Elixir.SiliconBrainWeb.Router --to expanded

My compiled code is much larger than my source code:

routes.ex: 118 lines, including comments: 118 lines in ex file compiled routes.ex file: 2,180 lines: 2,180 lines in ex file consumed.ex: 711 lines, including comments, docs, and examples: 711 lines in ex file compiled consumed.ex: 1,463 lines: 1,463 lines in ex file

Compile two simple macro-calling files

defmodule UnlessTrue do
  defmacro unless(boolean_exp, do: false_case) do
    quote do
      if !unquote(boolean_exp), do: unquote(false_case)
    end
  end
end

Calling this macro from another module modifies the compiled code of the calling module: obviously_true_ex.png Even though the compiled module’s function body could be replaced with nil, the code is not executed at compilation time: Elixir.ObviouslyTrue_ex.png possibly_true_ex.png Here, the return value of the function cannot be known until runtime because it depends on a runtime argument value: Elixir.PossiblyTrue_ex.png possibly_true_execute.png

Examples from Elixir’s Kernel module (slightly simplifed)

Kernel and Kernel.SpecialForms contain many macros

Kernel.__info__(:macros)
Kernel.SpecialForms.__info__(:macros)
Elixir.Kernel == :"Elixir.Kernel" &&
  Kernel == Elixir.Kernel
defmacro !value do

  quote do

    case unquote(value) do
      x when :"Elixir.Kernel".in(x, [false, nil]) -> true
      _ -> false
    end
    
  end

end
defmacro !{:!, _, [value]} do

  quote do

    case unquote(value) do
      x when :"Elixir.Kernel".in(x, [false, nil]) -> false
      _ -> true
    end
    
  end

end
defmacro if(condition, clauses) do  # clauses is a keyword list, e.g. [do: ..., else: ...]
  build_if(condition, clauses)
end

defp build_if(condition, do: do_clause) do
  build_if(condition, do: do_clause, else: nil)
end

defp build_if(condition, do: do_clause, else: else_clause) do
  quote do
    case unquote(condition) do
      x when :"Elixir.Kernel".in(x, [false, nil]) -> unquote(else_clause)
      _ -> unquote(do_clause)
    end
  end
end

AST literals

33 ==
  quote do
    33
  end
"thirty-three" == quote do: "thirty-three"
:thirty_three == quote do: :thirty_three
[33, 29, 11] == quote do: [33, 29, 11]
{:ok, 33} == quote do: {:ok, 33}

Some Elixir types that aren’t passed through to AST unmodified

quote do: %{cat: "bad", dog: "good"}
# tuples with > 2 elements
quote do: {1, 2, 3}
quote do: x
quote do: Pizza
quote do: Food.Pizza
quote do
  defmodule Food.Cake do
  end
end
quote do
  def my_fun do
    7
  end
end
quote do: true and false
quote do: !true
quote do: x in [1, 2]
x = 8
quote do: x in [1, 2]
x = 8

quote do
  x in [1, 2]
end
|> Code.eval_quoted(x: 2)
# macro hygiene: No pollution of namespaces/environments/contexts

# unquote/1 allows reaching outside the AST's scope

x = 8

quote do
  unquote(x) in [1, 2]
end
|> Code.eval_quoted()
# macro hygiene: No pollution of namespaces/environments/contexts

# Kernel.var!/2 "marks that a variable should not be hygienized"

# @spec eval_quoted(Macro.t(), binding(), Macro.Env.t() | keyword())
#   :: {term(), binding()}

x = 8

quote do
  var!(x) in [1, 2]
end
|> Code.eval_quoted(x: 2)
# special atom: Module name

defmodule SampleModule do
end

SampleModule == :"Elixir.SampleModule"

quote do: SampleModule
:"Elixir.SampleModule"
quote(do: if(true, do: 1, else: 0))
|> Macro.expand(__ENV__)
quote do
  defmodule MyMod do
    def my_fun do
      7
    end
  end
end
|> Macro.expand(__ENV__)

Why aren’t 2-element tuples modified?

Keeping 2-element tuples unchanged in their AST representations makes keyword lists much easier to read:

quote do: [{:red_sox, :good}, {:yankees, :evil}]
quote do: [red_sox: :good, yankees: :evil]

The above is much cleaner AST than this:

quote do: [{:red_sox, :good, :fenway_park}, {:yankees, :evil, :yankees_stadium}]

Code.eval_quoted(quoted, binding, opts)

quote(do: 17 + 18) |> Code.eval_quoted()
quote do
  17 + 18
end
|> Code.eval_quoted()
|> elem(0)

Code.string_to_quoted/2

Code.string_to_quoted/2 converts a string into AST (the “abstract syntax tree” representation of that same Elixir code).

The string representation “17” converts into the INTEGER 17, not the STRING “17”:

"17" |> Code.string_to_quoted()
"17" |> Code.string_to_quoted() |> elem(1)

To represent the STRING “17” within a string, we must either pass it into Integer.to_string() or wrap it within (escaped) quotation marks:

"17 |> Integer.to_string()" |> Code.string_to_quoted() |> elem(1)
"17 |> Integer.to_string()"
|> Code.string_to_quoted()
|> elem(1)
|> Code.eval_quoted()
|> elem(0)
"\"17\"" |> Code.string_to_quoted() |> elem(1)

Macro.to_string()

We can REVERSE the operation Code.string_to_quoted() using Macro.to_string():

"17 |> Integer.to_string()"
|> Code.string_to_quoted()
|> elem(1)
|> Macro.to_string()
"\"17\""
|> Code.string_to_quoted()
|> elem(1)
|> Macro.to_string()

quote(opts, block)

You can transform Elixir code into its AST representation using quote(), (i.e., Kernel.SpecialForms.quote(opts, block)):

quote do: 3 + 4
quote(do: 3 + 4) |> Macro.to_string()
quote(do: 3 + 4) |> Macro.to_string() |> Code.string_to_quoted!()

Metaprogramming is writing code that creates other code.

quote/2 transforms Elixir code into an Elixir AST (abstract syntax tree).

Manipulating Elixir AST lets us interact with the code-generation process in a manner impossible in most languages.

For example, Logger can inspect code at compilation time and completely remove debugging code intended only to run in dev mode or above a certain log level.

Another example: we can use macros to create DSLs (domain-specific languages), as Ecto and Phoenix do.

We can also create new “language primitives ‘missing’ from Elixir,” like while. Elixir primitives like defmodule, def, and if are all macros, not ordinary functions.

quote do: 17 + 18

Structure of this 3-element tuple: {function call, context, [arguments]}

All AST expressions have this shape

17 + 18 is syntactic sugar for Kernel.+(17,18), which is why the AST has :+ as the function, [context: Elixir, import: Kernel] as the context, and [17, 18] as arguments

quote do: 7 + 8
quote do
  7 + 8
end
|> Code.eval_quoted()
|> elem(0)

(Aside) Weird display of charlists (character lists)

[67, 97, 116]
[67, 97, 116] |> is_list()
[67, 97, 116] |> List.Chars.to_charlist()
[67, 97, 116] |> List.to_string()
# See https://elixir-lang.org/getting-started/binaries-strings-and-char-lists.html#charlists
# for why this is so weird

# Erlang & Elixir try to help us by displaying lists of numbers as characters
# whenever possible, even if we intend the numbers to just be numbers.

# Character lists != Strings

{?\a, ?\b, ?\r, ?\s, ?a, ?b, ?c, ?A, ?B, ?C}
[7, 8] |> List.Chars.to_charlist()
'\a\b' == [7, 8]
# Character lists != Strings

'\a\b' != "\a\b"
{:+, [context: Elixir, import: Kernel], [7, 8]} ==
  {:+, [context: Elixir, import: Kernel], '\a\b'}
quote do: [1000, 1001, 1002]
{?P, ?i, ?z, ?z, ?a}
quote do: [80, 105, 122, 122, 97]
[?P, ?i, ?z, ?z, ?a] |> List.to_string()
'efg' == [101, 102, 103]
# character list (from Erlang) != string (from Elixir)
'efg' != "efg"
"efg" |> IEx.Info.info()
'efg' |> IEx.Info.info()
[1000, 1001, 1002] |> IEx.Info.info()

unquote_splicing/1

z = [4, 5, 6]
quote do: [1, 2, 3, unquote(z), 7, 8]
z = [4, 5, 6]
quote do: [1, 2, 3, unquote_splicing(z), 7, 8]

Macro.to_string()

quote do: 17 + 18
quote(do: 17 + 18)
|> Macro.to_string()
quote do
  [1, 2, 3]
  |> Enum.map(&(&1 + 1))
end
quote do
  [1, 2, 3]
  |> Enum.map(&(&1 + 1))
end
|> Macro.to_string()
# The above represented a calculation without triggering the actual calculation.
# This is "lazy evaluation."
# The code for the calculation need not even be calculatable at compile time:
quote do
  [user_input_a, user_input_b, user_input_c]
  |> Enum.map(&(&1 + 1))
end
|> Macro.to_string()
quote do: x * y * z
quote(do: x * y * z)
|> Macro.to_string()

Deeply nested AST

quote do
  1 + 2 + 3 + 4 + 5 + 6
end
|> IO.inspect()

An Elixir program in its AST representation is a large (but structurally simple) tree of nested 3-element tuples

Variables & macro hygiene

quote do
  x
end

Whatever happened outside quote is invisible inside quote

x = 3

quote do
  x
end

To explicitly pull values from outside quote into it (at compile time), you can use unquote/1:

x = 3

quote do
  unquote(x)
end
y = 7
quote do: unquote(y)

To explicitly pull values from outside quote into it from its variable bindings at runtime, you can use var!/1.

Here I’m also passing in an optional keyword list for use only when evaluating this particular quoted expression:

Code.eval_quoted(
  quote do
    var!(x)
  end,
  x: 17
)

The above has NOT changed the value of x outside of the quoted expression:

x

The following grabs the values of x and y at compilation but does not run the calculation defined in the AST returned by quote:

x = 3
y = 4

quote do
  unquote(x) + unquote(y)
end
x = 3
y = 4

quote do
  unquote(x) + unquote(y)
end
|> Code.eval_quoted()
|> elem(0)
x = 3
y = 4

quote do
  x = unquote(x)
  y = unquote(y)
  x = x * x
  y = y * y
  x + y
end
|> Code.eval_quoted()
|> elem(0)

What happens inside quote stays inside quote.

x and y are still 3 and 4, not 9 and 16

%{x: x, y: y}

Break macro hygiene at runtime with var!/1

If you want your macro to modify a variable’s value outside the macro, you can do so by assigning to var!(x):

x = 3
y = 4

defmodule MyMultiply do
  defmacro mult(x, y) do
    quote do
      var!(x) = unquote(x) * unquote(x)
      var!(y) = unquote(y) * unquote(y)
      var!(x) + var!(y)
    end
  end
end

The return value is still 25:

require MyMultiply

MyMultiply.mult(x, y)
|> Code.eval_quoted()

But we have now permanently changed the values bound to the variables x and y in the scope outside the macro:

%{x: x, y: y}

Use bind_quoted instead of quote

In the above example, we called unquote(x) twice and unquote(y) twice.

This is a bad practice. One reason why is inefficiency. More importantly, if the parameter passed into unquote/1 is an impure function, you will be evaluating it multiple times.

This would be highly inefficient if the expression were, say, a database lookup.

It could be disastrous if the expression included a side effect like fire_the_missiles()!

The preferred way to bind variables is using bind_quoted.

We can (and should) rewrite the module above as follows:

x = 3
y = 4

defmodule MyMultiply2 do
  defmacro mult(x, y) do
    quote bind_quoted: [x: x, y: y] do
      var!(x) = x * x
      var!(y) = y * y
      var!(x) + var!(y)
    end
  end
end
require MyMultiply2

MyMultiply2.mult(x, y)
|> Code.eval_quoted()

Because we chose to assign x * x to var!(x), rather than to x and y * y to var!(y), rather than to y, we permanently modified these values outside the context of the macro. We could easily avoid var!/1 to preserve macro hygiene.

%{x: x, y: y}

Macro.escape for values

quote do
  {:a, :b, :c}
end
this_val = {:a, :b, :c}

quote do
  this_val
end
# Macro.escape() tranforms Elixir values --> AST values

this_val |> Macro.escape()
this_val_ast = this_val |> Macro.escape()

quote do
  unquote(this_val_ast)
end
# This fails to compile because a 3-tuple is not valid AST:

# defmodule BestTeam do
#   tuple = {:man_city, :haaland, :guardiola}
#   def football do
#     unquote(tuple)
#   end
# end
defmodule BestTeam do
  tuple = {:man_city, :haaland, :guardiola}
  tuple_ast = tuple |> Macro.escape()

  def football do
    unquote(tuple_ast)
  end
end

BestTeam.football()

Kernel.is_struct/2

defmodule Human do
  @enforce_keys [:name, :age]
  defstruct name: nil, age: 0
  @type t :: %__MODULE__{name: String.t(), age: non_neg_integer}
end
mary = %Human{name: "Mary", age: 28}
IO.inspect(mary, structs: false)
quote do
  %Human{name: "Mary", age: 28}
end
quote do
  mary
end
mary |> Macro.escape()
quote do
  %{__struct__: Human, age: 28, name: "Mary"}
end
escaped = mary |> Macro.escape()

quote do
  unquote(escaped)
end
defmacro is_struct(term) do
  quote do
    is_map(unquote(term)) and
      :erlang.is_map_key(:__struct__, unquote(term)) and
      is_atom(:erlang.map_get(:__struct__, unquote(term)))
  end
end
# ELIXIR CODE

# %{__struct__: Human, age: 28, name: "Mary"}
term = %Human{age: 28, name: "Mary"}

is_map(term) and
  :erlang.is_map_key(:__struct__, term) and
  is_atom(:erlang.map_get(:__struct__, term))
# AST (ABSTRACT SYNTAX TREE)

term_as_ast =
  {:%{}, [], [__struct__: {:__aliases__, [alias: false], [:Human]}, age: 28, name: "Mary"]}

quoted =
  quote do
    is_map(unquote(term_as_ast)) and
      :erlang.is_map_key(:__struct__, unquote(term_as_ast)) and
      is_atom(:erlang.map_get(:__struct__, unquote(term_as_ast)))
  end

quoted |> Code.eval_quoted() |> elem(0)
{is_struct(mary), is_struct(mary, Human), is_struct(mary, Ecto.Changeset)}

Kernel.tap/2

%{a: 1}
|> Map.update!(:a, &(&1 + 2))
|> tap(&IO.inspect(&1.a))
|> Map.update!(:a, &(&1 * 2))
  defmacro tap(value, fun) do
    quote bind_quoted: [fun: fun, value: value] do
      _ = fun.(value)
      value
    end
  end

unquote fragments

images/def_block_wrapped_in_unquote.png

binding_and_unquote_fragments.png 

defmodule FooBar do
  kv = [foo: 1, bar: 2]

  Enum.each(kv, fn {k, v} ->
    def unquote(k)(), do: unquote(v)
  end)
end
FooBar.foo() + FooBar.bar()
defmodule FooBar2 do
  kv = [foo: 1, bar: 2]

  Enum.each(kv, fn {k, v} ->
    def unquote(k)(extra \\ 0), do: unquote(v) + extra
  end)
end
FooBar2.foo() + FooBar2.bar()
FooBar2.foo(2) + FooBar2.bar(-4)

What is a metaprogramming… and why should I care?

Why learn metaprogramming?

  • Understand Elixir itself, most of which is created using Elixir macros
    • Examples: def, defmodule, if, unless, etc.
    • Even defmacro is defined using defmacro! (Don’t ask me how!)
  • Extend the Elixir syntax with declarative functions that keep your code clean/sparse
    • Example: then was only recently added, but you could have added it long ago
  • Debug with greater confidence
    • When debugging, we’ll often wander into macros and other metaprogramming. The more familiar you are with these techniques, the better equipped you’ll be to debug your problem.
  • Create functions dynamically, perhaps from data in a file
    • Example: functions to generate Unicode functions (e.g., capitalize) dynamically from Unicode file
  • Create your own DSLs (domain-specific languages):
  • Create reusable compile-time code by extracting code you wish to use in multiple modules into a code block dynamically injectable via the defmacro __using__(opts) macro
  • Optimize your code:
    • The compiler can simplify or optimize your code based on opts passed in at compliation time
      • If you specify a log level of :warning, Logger.warning calls will be compiled into your code, but Logger.info calls will not

Compilation-time code generation from external files

See Chris McCord’s “Metaprogramming Elixir” book, pages 43-58, for additional explanation.

UnicodeData.txt: UnicodeData.txt

Unicode.ex generates String.Unicode: unicode_ex.png

String.Unicode constituted from functions generated from .txt files: String.Unicode.png

@external_resource

See Chris McCord’s “Metaprogramming Elixir” pp. 48-49 for more.

Compilation-time code generation from API responses

In Chris McCord’s “Metaprogramming Elixir” book, pages 59-61, he shows a 20-line module that dynamically creates an Elixir function for displaying all Github repos owned by a user and dynamically creates two functions per repo, one for displaying all details of the repo and the other for opening the repo in your browser.

Compilation-time code elimination

logger_0 logger_1 logger_2 maybe_log logger_3

Before invoking a macro, you must require it

require require

Back