Mnesia for “hot” telemetry

Loading sample data

I already downloaded 100k records from last week and stored them in a format that will work for us:

{account, monitor, check, instance} -> [{ts, value}, {ts, value}, ...]

ordered by timestamp. For fun, I stashed it in a DETS file.

{:ok, dets_table} = :dets.open_file('/home/eng/Downloads/one_week_telemetry.dets')
:dets.info(dets_table)

We want this table to be always available on all nodes. It is actually not very big (currently should be around 250MB for the whole dataset). So we do some Mnesia setup and copy the data over.

First ensure that we have everything clustered.

Node.list()

We are running on “first”, so seeing the other two nodes, “second” and “livebook” is correct.

:mnesia.start()
:mnesia.create_table(:last_week, [])

Next, we fill that table with what’s in DETS. We should end up with a bunch of records living only on the local node. Note that Mnesia wants Erlang records, which are tuples where the first field is the type and, in this case, the table name.

:mnesia.transaction(fn ->
  :dets.traverse(dets_table, fn {k, v} ->
    IO.puts("Inserting #{inspect(k)} -> #{inspect(v)}")
    :mnesia.write({:last_week, k, v})
    :continue
  end)
end)

:mnesia.table_info(:last_week, :size)

Next, we copy it to the second node. This is probably where the Livebook ends being useful given that now you need to kill nodes and copy back and forth.

Node.list()
|> Enum.each(fn node -> :mnesia.add_table_copy(:last_week, node, :ram_copies) end)

:mnesia.table_info(:last_week, :ram_copies)