ElixirZone: Metaprogramming Elixir: Part 2 - James Lavin

Metaprogramming Costs

• Introduces an additional layer of indirection/abstraction
• Harder to debug code
• Harder to understand code
• Requires thinking about both compilation-time code execution and runtime-code execution

Jay Hayes - Elixir in Elixir

https://www.youtube.com/watch?v=p8MGNw045AE

:lol = :wat

# This does NOT throw a MatchError because it's AST describing an
# assignment, but the assignment is NOT EXECUTED

quote do
  :lol = :wat
end

quote do
  :lol = :wat
end == {:=, [], [:lol, :wat]}

If we attempt to execute/evaluate it, it will blow up:

{:=, [], [:lol, :wat]} |> Code.eval_quoted()

Jay then uses pattern-matching to define an Expression.match?/1 function returning a Boolean representing whether the argument represents a match/assignment:

num = 42
quote do: 1 + num

num = 42
quote do: 1 + unquote(num)

quote do
  1 + unquote(num)
end
|> Code.eval_quoted()

Tyler Pachal - Metaprogramming for Dummies

https://www.youtube.com/watch?v=DFa1bC95wxA

Sasha Fondeca - Metaprogramming With Elixir

https://www.youtube.com/watch?v=uBWFBU97Qkw

Marlus Saraiva - Surface, HEEX and the Power of Choice

https://www.youtube.com/watch?v=ChGSLUVe5Gs

Benefits of LiveView Surface’s Macro Components:

• Extensibility via compile-time AST manipulation
• Static evaluation of body/content (warn about errors, etc.)
• Can embed other languages
• Performance optimizations

Qing Wu: A Gentle Introduction to Elixir Macros

https://www.youtube.com/watch?v=CHHQ_xzv4pw (doesn’t indicate the speaker’s name) https://www.youtube.com/watch?v=X8QfT7BNE44 is from Qing Wu, who I believe is the same presenter)

Billy Ceskavich: Is Elixir Just Lisp?

https://www.youtube.com/watch?v=F7qaIqcaTDc

Bryan Weber: Code Generation in Elixir

https://www.youtube.com/watch?v=-mgwW3RVI50

# Code.compile_quoted(quoted_code)

# File.write(...)

Code compilation process

Jesse Anderson: Don’t Write Macros But Do Learn How They Work

• https://www.youtube.com/watch?v=Bo48sQDb-hk

if_ast = quote do: if(var!(x) == var!(y), do: "yeah", else: "nah")

if_ast |> Macro.to_string() |> IO.puts()

if_ast |> Code.eval_quoted(x: 7, y: 9)

if_ast |> Code.eval_quoted(x: 19, y: 19)

Andrew Summers - Domain Specific Languages and Metaprogramming in Elixir

Let’s Play With Metaprogramming!

# runtime evaluation

r = 2
:math.pi() * :math.pow(r, 2)

# ordinary function created at compile time

defmodule Area do
  def circle(radius) do
    :math.pi() * :math.pow(radius, 2)
  end
end

# runtime evaluation

Area.circle(2)

# Macro function

defmodule AreaMacro do
  require Logger

  defmacro circle(radius) do
    Logger.info("compiling...")

    quote do
      :math.pi() * :math.pow(unquote(radius), 2)
    end
  end
end

# runtime evaluation

require AreaMacro
# x = 3
# {AreaMacro.circle(2), AreaMacro.circle(x)}
AreaMacro.circle(2)

use && using/1

defmodule AreaMacro2 do
  defmacro __using__(opts) do
    case opts do
      :circle ->
        quote location: :keep, bind_quoted: [opts: opts] do
          def area(radius) do
            :math.pi() * :math.pow(radius, 2)
          end
        end

      :square ->
        quote location: :keep, bind_quoted: [opts: opts] do
          def area(side) do
            :math.pow(side, 2)
          end
        end

      other when is_binary(other) ->
        quote location: :keep, bind_quoted: [opts: opts] do
          other = other |> String.to_atom()
          IO.inspect("Do not recognize shape #{other}")
        end
    end
  end
end

defmodule Circle do
  use AreaMacro2, :circle
end

defmodule Square do
  use AreaMacro2, :square
end

# runtime evaluation

{Circle.area(2), Square.area(2)}

On p. 99 of “Metaprogramming Elixir,” Chris tells us to avoid using use just for mix-ins.

alias or import is fine for just pulling in functions to DRY up your code. Or you can just call functions using their fully qualified names.

Circle

# runtime evaluation

if 1 + 2 == 3, do: "this", else: "that"

# runtime evaluation

if 1 + 2 == 3 do
  "this"
else
  "that"
end

# runtime evaluation

if(Kernel.==(Kernel.+(1, 2), 3), [{:do, "this"}, {:else, "that"}])

# compile without evaluation

quote do
  if(1 + 2 == 3, do: "this", else: "that")
end

# compile, then evaluate

quote do
  if(1 + 2 == 3, do: "this", else: "that")
end
|> Code.eval_quoted()

one = 1
two = 2
three = 3
if(one + two == three, do: "this", else: "that")

quote do
  if(one + two == three, do: "this", else: "that")
end

quote do
  if(var!(one) + var!(two) == var!(three), do: "this", else: "that")
end
|> Code.eval_quoted(one: 1, two: 2, three: 3)

quote do
  if(var!(one) + var!(two) == var!(three), do: "this", else: "that")
end
|> Code.eval_quoted(one: 1, two: 2, three: 4)

quote do
  if(one + two == three, do: "this", else: "that")
end

# Fails with ** (CompileError) nofile:1: undefined variable "three"

quote do
  if(var!(one) + var!(two) == var!(three), do: "this", else: "that")
end
|> Code.eval_quoted(one: 1, two: 2)

Delegate to Erlang functions whenever possible

Macro.postwalk() …and Macro.prewalk()

This is a MUCH simplified version of just one part of Argentinian Lucas San Román’s 3-part series at https://dorgan.netlify.app/posts/2021/04/the_elixir_ast/. I recommend the whole series to you!

filepath =
  "/Users/jameslavin/Git/ElixirZone/Elixir/LiveBooks/Elixir_104_Metaprogramming/AreaMacro2.ex"

areamacro2_ast =
  filepath
  |> Path.expand()
  |> File.read!()
  |> Code.string_to_quoted!(columns: true)

defmodule Checker do
  @warning "You can create only a finite number of atoms. You should not be generating them dynamically from strings."

  def check(ast) do
    {_ast, issues} = Macro.postwalk(ast, [], &handle_node/2)
    issues
  end

  defp handle_node({{:., _, [{:__aliases__, _, [:String]}, :to_atom]}, meta, _args} = node, acc) do
    issue = %{
      warning: @warning,
      line: meta[:line],
      column: meta[:column]
    }

    {node, [issue | acc]}
  end

  defp handle_node(node, acc), do: {node, acc}
end

areamacro2_ast |> Checker.check()

Macro.prewalk and Macro.postwalk can be used not only to report on AST but to MODIFY AST, as the documentation explains:

> Returns a new AST where each node is the result of invoking fun on each corresponding node of ast. > Example: >

bind_quoted & execution context of unquote()

The following is one small portion of Daniel Xu’s excellent article, “The Minimum Knowledge You Need to Start Metaprogramming in Elixir,” https://dockyard.com/blog/2016/08/16/the-minumum-knowledge-you-need-to-start-metaprogramming-in-elixir

defmodule M do
  defmacro my_macro(name) do
    IO.puts(1)

    quote do
      IO.puts(4)
      unquote(IO.puts(2))
      IO.puts("hello, #{unquote(name)}")
    end
  end
end

defmodule Create do
  import M

  IO.puts(3)
  my_macro("hello")
end

defmodule M do
  defmacro my_macro(name) do
    IO.puts(1)

    quote bind_quoted: [name: name] do
      IO.puts(4)

      def unquote(name)() do
        unquote(IO.puts(2))
        IO.puts("hello, #{unquote(name)}")
      end
    end
  end
end

defmodule Create do
  import M

  IO.puts(3)
  my_macro(:hello)
end

defmodule M do
  defmacro my_macro(name) do
    IO.puts(1)

    quote bind_quoted: [name: name] do
      IO.puts(4)

      def unquote(name)() do
        unquote(IO.puts(2))
        IO.puts("hello, #{unquote(name)}")
      end
    end
  end
end

defmodule Create do
  import M

  IO.puts(3)

  [:foo, :bar]
  |> Enum.each(&my_macro(&1))
end

{Create.foo(), Create.bar()}

Macro.escape()

long_tuple = {:x, :y, :z}

inspect_long_tuple_ast = quote do: IO.inspect(unquote(long_tuple))

# throws an ArgumentError because {:x, :y, :z} is not valid AST

# inspect_long_tuple_ast |> Code.eval_quoted()

long_tuple = {:x, :y, :z} |> Macro.escape()
# returns {:{}, [], [:x, :y, :z]}

inspect_long_tuple_ast = quote do: IO.inspect(unquote(long_tuple))

inspect_long_tuple_ast |> Code.eval_quoted()

&Kernel.+/2

quote do: 1 + 2

quote do
  Kernel.+(1, 2)
end

Environments (ENV & CALLER)

IO.inspect(__ENV__, limit: :infinity, printable_limit: :infinity, structs: false)

Why not just use functions?

defmodule Our do
  def if_fun(condition, do: this, else: that) do
    if(condition) do
      this
    else
      that
    end
  end
end

Our.if_fun 1 == 1 do
  IO.inspect("true")
else
  IO.inspect("false")
end

Our.if_fun 1 == 3 do
  IO.inspect("true")
else
  IO.inspect("false")
end

Function args are always evaluated when the function gets called!

To avoid that, we use macros, which receive AST, which represents unevaluatable data structures that have not (yet) been evaluated.

Nicholas J. Henry - The Upside Down Dimension of Elixir

• https://www.youtube.com/watch?v=xj6yNzcGlEE
• https://www.youtube.com/watch?v=EFAgc7YqDP8

Adi Iyengar: The Pillars of Metaprogramming in Elixir

• https://www.youtube.com/watch?v=oikF1Ze2Vao

Lizzie Paquette: Responsible Code Generation

https://www.youtube.com/watch?v=55-X7rSw8M0

• Presentation video & slides
• Slides (PDF)

defmodule MyMod do
  defmacro who_am_i() do
    IO.inspect(__MODULE__, label: "Macro Context")
    IO.inspect(__CALLER__.module, label: "Caller Env")

    quote do
      IO.inspect(__MODULE__, label: "Caller Context")
      IO.inspect(unquote(__MODULE__), label: "Value from Macro Context")
    end
  end
end

defmodule MyCaller do
  require MyMod
  MyMod.who_am_i()
end

Dynamic function names (using “unquote fragments”)

For more on unquote fragments, see:

• https://hexdocs.pm/elixir/1.14.0/Kernel.SpecialForms.html#quote/2
• Chris McCord’s “Metaprogramming Elixir” pp. 46-48

defmodule TalkingHeads do
  @funs [ask_yourself: "How did I get here?", no_party: "No disco!"]
  # @funs [{:ask_yourself, "How did I get here?"}, {:no_party, "No disco!"}]

  for {k, v} <- @funs do
    def unquote(k)(answer), do: answer
    def unquote(k)(), do: unquote(v)
  end
end

TalkingHeads.ask_yourself()

TalkingHeads.no_party()

TalkingHeads.ask_yourself("By taxi")

After I run:

elixirc talking_heads.ex
mix decompile Elixir.TalkingHeads.beam --to expanded
vi Elixir.TalkingHeads.ex

I see basically what the compiled BEAM file logic looks like:

defmodule TalkingHeads do
  def no_party(answer) do
    answer
  end

  def no_party() do
    "No disco!"
  end

  def ask_yourself(answer) do
    answer
  end

  def ask_yourself() do
    "How did I get here?"
  end
end

quote/2 transforms Elixir code –> an AST (“abstract syntax tree”) data structure

# The input `1` is ordinary Elixir code. The output `1` is that value represented as AST
quote do: 1

# Elixir -> AST
# `sum(1,2)` has a different representation in AST than in Elixir code
quote do: sum(1, 2)

# The AST this generates spells out -- in a nested data structure -- 
#       the exact operations to be run.
# This is a nested data structure representing operations to be run.
# This runnable specification does NOT execute those operations.
# In fact, this is not runnable as is because `sum` is not even defined.
# Executing the AST this generates will require a definition for `sum/2`:
quote do: sum(sum(sum(1, 3), sum(2, 4)), sum(5, 6))

# The following cannot be evaluated because we have not provided a definition of `sum`
quote do
  sum(1, 2)
end
|> Code.eval_quoted()

defmodule MyMath do
  def sum(a, b), do: a + b
end

quote do
  MyMath.sum(1, 2)
end
|> Code.eval_quoted()

quote do
  MyMath.sum(MyMath.sum(MyMath.sum(1, 3), MyMath.sum(2, 4)), MyMath.sum(5, 6))
end
|> Code.eval_quoted()

quote do
  import MyMath
  sum(sum(sum(1, 3), sum(2, 4)), sum(5, 6))
end

quote do
  import MyMath
  sum(sum(sum(1, 3), sum(2, 4)), sum(5, 6))
end
|> Code.eval_quoted()

quote do
  1 + 2
end
|> Code.eval_quoted()

quote do: sum(1, 2) |> Code.eval_quoted()

quote do: random_variable_name

random_variable_name = 7

quote do: random_variable_name

random_variable_name = 9

quote do: unquote(random_variable_name)

# AST cannot be further converted
{:sum, [], [1, 2]}

# Elixir -> AST -> String
quote do
  sum(1, 2)
end
|> Macro.to_string()

# Using `quote do: ...`, you'll want to add parentheses or the compiler will likely
# misinterpret your code
quote(do: sum(1, 2))
|> Macro.to_string()

# AST -> String
{:sum, [], [1, 2]} |> Macro.to_string()

# AST -> String -> AST
{:sum, [], [1, 2]} |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

# AST -> String -> AST
{:%{}, [], [{:a, 1}, {:b, 2}]} |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

# AST -> String -> AST
{:{}, [], [1, 2, 3, 4]} |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

# AST -> String -> AST
{:{}, [], [1, 2]} |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

# AST -> String -> AST
[1, 2, 3, 4, 5] |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

# AST -> String -> AST
[{:a, 1}, {:b, 2}, {:c, 3}, {:d, 4}, {:e, 5}]
|> Macro.to_string()
|> Code.string_to_quoted()
|> elem(1)

# AST -> String -> AST
[a: 1, b: 2, c: 3, d: 4, e: 5] |> Macro.to_string() |> Code.string_to_quoted() |> elem(1)

fields = [
  %{name: "id", type: "integer"},
  %{name: "name", type: "String.t()"},
  %{name: "status", type: "atom()"}
]

{:%{}, [], fields |> Enum.map(fn x -> {x.name, x.type} end)}

fields = [
  %{name: "id", type: "integer"},
  %{name: "name", type: "String.t()"},
  %{name: "status", type: "atom()"}
]

{:%, [],
 [{:__MODULE__, [], Elixir}, {:%{}, [], fields |> Enum.map(fn x -> {x.name, x.type} end)}]}
|> Macro.to_string()

Understanding DSL “magic” (including Elixir itself)

A blessing of powerful languages like Ruby and Elixir – which provide users the ability to create domain-specific languages (DSLs) – is that you can greatly simplify tasks via DSLs.

But DSLs come with a curse… they add layers of indirection, which can make it harder to understand and debug code that uses those DSLs.

DSLs are “magic.” Magical spells are powerful but also dangerous.

Given that much of Elixir is created using the “magic” of macros and metaprogramming, you can’t fully understand Elixir, leverage its full power, or help extend it without understanding macros and metaprogramming.

92% of Elixir was written in Elixir… via the magic of metaprogramming:

Kernel.SpecialForms

^^^ We have touched the metal / hit the event horizon, beyond which we cannot proceed.

Kernel.then/2

%{}
|> then(&Map.put(&1, :baseball, "Red Sox"))
|> then(&Map.put(&1, :football, "Patriots"))
|> then(&Map.put(&1, :basketball, "Celtics"))
|> then(&Map.put(&1, :hockey, "Bruins"))
|> then(&Map.put(&1, :real_football, "Revolution"))

Macro.expand_once/2

# This is compiled and then immediately evaluated
unless true, do: :a, else: :b

quote do
  unless true, do: :a, else: :b
end
|> Macro.expand(__ENV__)

# This is compiled and then immediately evaluated, but it throws an error
# because the variable `unset_value` has no value
unless unset_value, do: "yay", else: "boo"

# This is transformed into AST but NOT immediately evaluated:
quote do
  unless unset_value, do: :a, else: :b
end

quote do
  unless unset_value, do: :a, else: :b
end
|> Macro.to_string()
|> String.split("\n")
|> Enum.each(&IO.puts(&1))

# This is transformed into AST but NOT immediately evaluated,
# but one round of "macro expansion" is applied:
quote do
  unless unset_value, do: :a, else: :b
end
|> Macro.expand_once(__ENV__)

# We can see this more clearly:
quote do
  unless unset_value, do: :a, else: :b
end
|> Macro.expand_once(__ENV__)
|> Macro.to_string()
|> String.split("\n")
|> Enum.each(&IO.puts(&1))

# This is transformed into AST but NOT immediately evaluated,
# but two rounds of "macro expansion" are applied:
quote do
  unless unset_value, do: :a, else: :b
end
|> Macro.expand_once(__ENV__)
|> Macro.expand_once(__ENV__)
|> Macro.to_string()
|> String.split("\n")
|> Enum.each(&IO.puts(&1))

defmacro

• defmacro takes its arguments in as AST, not as ordinary Elixir values
• unquote/1 (inside quote do ... end) transforms an AST argument into ordinary Elixir code by evaluating its current Elixir value, allowing you to pull in a value during code compilation (i.e., “at compile time”) from the environment outside the quote do ... end block.
• Everything inside the quote do ... end gets transformed from ordinary Elixir code into AST

defmodule OurThen do
  defmacro our_then(value, fun) do
    fun |> Macro.to_string() |> IO.inspect(label: "Compile time fun")
    value |> Macro.to_string() |> IO.inspect(label: "Compile time value")

    quote do
      # becomes (at compilation time below): (fn x -> x * 2 end).(1)
      unquote(fun).(unquote(value))
    end
  end
end

require OurThen

1 |> OurThen.our_then(fn x -> x * 2 end)
# becomes (at compilation time):
# (fn x -> x * 2 end).(1)

Kernel.tap/2

%{a: 1}
|> tap(&IO.inspect(&1.a))
|> Map.update!(:a, &(&1 + 100))

defmodule OurMod do
  defmacro our_tap(value, fun) do
    fun |> Macro.to_string() |> IO.inspect(label: "Compile time fun")
    value |> Macro.to_string() |> IO.inspect(label: "Compile time value")

    quote bind_quoted: [fun: fun, value: value] do
      # after bind_quoted:
      #   * `fun` will represent the uninvoked function itself
      #   * `value` will be the evaluated value
      _ = fun.(value)
      value
    end
  end
end

%{a: 1} |> Map.update!(:a, &(&1 * 20))

&IO.inspect(&1.a)

&List.first/1

quote do
  &List.first/1
end
|> Macro.to_string()

require OurMod

%{a: 1}
|> Map.update!(:a, &(&1 * 20))
|> OurMod.our_tap(&IO.inspect(&1.a))
|> Map.update!(:a, &(&1 + 100))

Kernel.SpecialForms

In a previous episode, I wasn’t clear on the difference between Kernel & Kernel.SpecialForms.

I’ve since learned that macros in Kernel are expanded while macros in Kernel.SpecialForms are, well, special, and don’t have documented implementations.

Here’s an example, the for comprehension. The documentation provides a LONG description and good examples:

The documentation is about 150 lines long, but here’s the entire documented “implementation”:

It has a secret (?) implementation.

Macros

Very heavily inspired by p. 6 of Chris McCord’s “Metaprogramming Elixir”

Chris writes “Rule 1: Don’t Write Macros” because “macros can make programs difficult to debug and reason about.”

Macros receive AST and return AST. In other words, macros transform ASTs

defmodule LogMath do
  defmacro log({:+, ctx, [left_arg, right_arg]}) do
    quote do
      require Logger
      left = unquote(left_arg)
      right = unquote(right_arg)
      Logger.debug("received '#{left} + #{right}'")
      unquote(ctx) |> IO.inspect(label: "ctx")
      left + right
    end
  end

  defmacro log({op, ctx, args}) do
    quote bind_quoted: [op: op, ctx: ctx, args: args] do
      IO.inspect(op, label: "op")
      IO.inspect(ctx, label: "ctx")
      IO.inspect(args, label: "args")
    end
  end

  defmacro log2({op, ctx, [left_arg, right_arg]}) when op in [:+, :-, :*, :/] do
    quote bind_quoted: [op: op, left: left_arg, right: right_arg, ctx: ctx] do
      require Logger
      # left = unquote(left_arg)
      # right = unquote(right_arg)
      Logger.debug("received '#{left} #{op} #{right}'")
      ctx |> IO.inspect(label: "ctx")
      # unquote(op)(left, right)
      {op, ctx, [left, right]} |> Code.eval_quoted() |> elem(0)
    end
  end

  defmacro log2({op, ctx, args}) do
    quote bind_quoted: [op: op, ctx: ctx, args: args] do
      IO.inspect(op, label: "op")
      IO.inspect(ctx, label: "ctx")
      IO.inspect(args, label: "args")
    end
  end
end

Kernel.+(3, 4)

When this code is compiled, 111 + 222 will be converted to AST – {:+, [context: Elixir, import: Kernel], [111, 222]} – and passed into LogMath.log():

require LogMath

(111 + 222)
|> LogMath.log()

(333 + 444)
|> LogMath.log()

# I added second defmacro clause to debug why this wasn't working:
quote do
  555 + 444
end
|> LogMath.log()

quote do
  Kernel.+(555, 444)
end
|> LogMath.log()

# |> Macro.escape()

Why can’t I pattern match against [import: Kernel, context: Elixir]???

(666 - 555)
|> LogMath.log2()

(666 / 3)
|> LogMath.log2()

Unless & navigating Elixir documentation

Macros receive (and can pattern-match against) the AST representations of their arguments

quote do: %{a: 1, b: 2}

defmodule MyMerge do
  defmacro display({:%{}, [], _kwlist}) do
    quote do
      17
    end
  end

  # defmacro merge(first, _second) do
  #  quote do
  #    unquote(first)
  #  end
  # end
  defmacro merge({:%{}, [], first_kv_pairs}, {:%{}, [], second_kv_pairs}) do
    quote do
      first_map = unquote(first_kv_pairs) |> Enum.into(%{})
      second_map = unquote(second_kv_pairs) |> Enum.into(%{})
      Map.merge(first_map, second_map)
    end
  end
end

require MyMerge

MyMerge.display(%{})
# (quote do: %{}) |> MyMerge.display()

require MyMerge

first = quote do: %{a: 1, b: 2}

second = quote do: %{c: 3, d: 4}

MyMerge.merge(first, second)

Module.create/3

ast =
  quote do
    def answer_to_life, do: 42

    def unladen_swallow_airspeed, do: "African or European?"
  end

Module.create(Culture, ast, Macro.Env.location(__ENV__))

Culture.answer_to_life()

Culture.unladen_swallow_airspeed()

use & using(opts)

register_attribute(…, accumulate: true) & @before_compile

Source: Jia Hao Woo, https://blog.appsignal.com/2021/10/26/how-to-use-macros-in-elixir.html

Chris McCord’s “Metaprogramming Elixir” book has a very similar example on pages 39-42.

Jia Hao Woo published four metaprogramming articles at: https://blog.appsignal.com/authors/jia-hao-woo

Testing macros

In “Metaprogramming Elixir” (page 69), Chris McCord advises you to test your generated code, not the macro code generation process directly.

For more, you might find this talk useful:

Debugging Elixir Code - The Tools, The Mindset - Michal Buszkiewicz (ElixirConf EU 2021)