Overview

Intro - Background and Purpose

Welcome to the livebook demo for form_validator, an sample project intending to show how to use Ash framework in a Phoenix LiveView project to create, retrieve, update, and delete a parent/child data relationship in a single data entry form.

Database model is users and tweets. There are many tweets per user but only one user per tweet.

Access to User and Tweet structures is mediated through Ash framework, which encapsulates database data at a table level through an Ecto.Schema like module called a “resource”. Ash.Resources define both the structure of a database table, (e.g., “attributes”, “validations”, or “virtual fields” called “aggregates” or “calculations”) as well as the actions or methods that can be used to retrieve, manipulate and persist that data through to the database.

Remainder of this livebook flows along the following path - first we connect livebook to the database and add in some sample data. Then, we’ll read, update, and delete that data dealing with each table as a standalone entity. After that, we’ll figure out how to manage the parent/child relationship between users and tweets, then we’ll try and expose that backend up through a single, parent/child data entry form, using Ash.PhoenixForm.

Data Model - Users & Tweets

Ash.Resources are by convention placed in the ./lib/resource/ directory, with one “schema” module defined for each table. In our example, that would be ./lib/resource/user.ex for Users, or ./lib/resource/tweet.ex for Tweets.

If you pause for a moment and go take a look at thes files, you’ll notice that they specify both the structure of the database, as well as the validations and update actions used to maintain it. The syntax for specifying resources is fully laid out in the documentation under Ash.Resource.Dsl but highlights include attribute for specifying physical database columns, calculations and aggregates for specifying virtual columns, validations for ensuring good data, and actions that control how to persist data to the databse. Resources also handle other things like access control, multi-tennancy, etc. The documentation for Ash.Resource.Dsl contains full details.

Introspection: If you want to interactively explore your resources, Ash.Resource.Info has some introspection functions you can use to poke around, so you can type stuff into an IEx command prompt like Ash.Resource.Info.attributes(User) to show list of fields defined for Resource.User, or Ash.Resource.Info.actions(User) to see the actions instead.

For our livebook sample applications there are two resources defined: Ash.Resource.User considers both :id and :email to be two separate unique identifiers, while Ash.Resource.Tweet is defined with just one unique (primary) key of :id.

Startup Database, take a look around.

Launch the Ecto.Repo via start_link(), setup some useful aliases, and take a look at the structure.

# setup require and aliases
require Ash.Query
alias FormValidator.{User, Tweet, Api, Repo}

# launch repo process
FormValidator.Repo.start_link()

# see what resources are available 
Ash.Api.resources(Api)

# iterate through User's "attributes" and "actions" 
Enum.each(Ash.Resource.Info.attributes(User), fn x -> IO.inspect(x.name, label: "attribute: ") end)

Enum.each(Ash.Resource.Info.actions(User), fn x -> IO.inspect(x.name, label: "action: ") end)

# do the same for Tweet 
Enum.each(Ash.Resource.Info.attributes(Tweet), fn x ->
  IO.inspect(x.name, label: "attribute: ")
end)

Enum.each(Ash.Resource.Info.actions(Tweet), fn x -> IO.inspect(x.name, label: "action: ") end)

# truncate databse so rest of examples start from clean baseline. 
Repo.query("truncate users cascade")

CRUD Operations

In this section, we’ll go through the create/read/update/destroy actions common to business applications. First we add some users and tweets, then we’ll read to get them back from the database, then we’ll spend a few moments showing how to relate these two tables together, including adding a new tweet to existing user, removing a tweet, linking orphaned tweets to a user and changing the owner of a tweet.

Add some new Users

Make a list of new user emails and then add them, one at a time. To do this, we take an empty User struct, feed that into Ash.Changeset.for_create with an “action” of :create, and then persist to database via Api.create()

# --  NOTE: Throughout this livebook, you may notice bindings being assigned to expressions pipelined 
# -- across multiple lines of text. If you try to cut and paste multi-line expressions from a pipeline into IEx, 
# --  the binding often won't work because IEx evaluates expressions one line at a time. 
# -- the solution is to either place the |> token at end of each line or join all of the lines
# -- together into one, big line and then paste that into IEx.

# create list of users
list_of_users = [
  "alice@wonderland.com",
  "joe@volcano.com",
  "sam@wainwright.com",
  "james@marfugi.com",
  "mary@littlelamb.com"
]

# iterate through list, adding one user at a time.
users_added =
  Enum.reduce(list_of_users, 0, fn item, acc ->
    # Struct -> Changeset -> Resource
    User
    |> Ash.Changeset.for_create(:create, %{email: item})
    |> Api.create()

    acc + 1
  end)

IO.inspect(users_added, label: "Users added: ")

Add some new Tweets

Do the same for Tweets, considering only :body as a required attribute.

# list
list_of_tweets = [
  "Phoenix LiveView + OTP means scalable real-time goodness.",
  "Tweet-2",
  "Tweet-3",
  "Tweet-4",
  "Tweet-5"
]

# iterate through list, adding one tweet at a time.
tweets_added =
  Enum.reduce(list_of_tweets, 0, fn tweet_body, acc ->
    # Struct -> Changeset -> Resource
    Tweet
    |> Ash.Changeset.for_create(:create, %{body: tweet_body})
    |> Api.create()

    acc + 1
  end)

IO.inspect(tweets_added, label: "Tweets added: ")

Reading Data

The api layer handles interactions with the database, so just like Api.create() is used to save new records to the database, Api.read() is used to retrieve them instead. Before we can understand how to read, though, we must understand Ash.Query first.

Querying Data

Ash.Query contains a bunch of functions that facilitate data retrieval. The full list is available in the documentation at Ash.Query or you can interactively explore it in IEx by typing “Ash.Query.” (with the “dot”) and then hitting TAB to expand to show all of its functions. (For fun, try typing in h Ash.Query.filter into IEx to see the command line docs on filter). There are many query functions, but the main ones to get started include filter, sort and load. Filter includes or excludes certain rows from the result set, while load pre-loads related tables, sort sorts, and offset and limit are useful for pagination.

Filtering: Query.filter takes a resource and a filter expression, which can be specified in either keyword or expression style syntax. Here is example below:

# No query needed to "select all", just push the Ash.Resource into Api.read and you'll get everything.
Api.read(User)

# Query.filter to retrieve a subset of users, using keyword style syntax in this case.
User
|> Ash.Query.filter(email: "james@marfugi.com")
|> Api.read()

# Same query but with expression style filtering syntax. 
User
|> Ash.Query.filter(email == "james@marfugi.com")
|> Api.read()

Loading Related Data

Query.load is used to preload related tables. Note that it doesn’t have to have a Query.filter on it first, but can.

# Query.load using keyword style syntax. This result will show `[]` for associated tweets because there haven't been any tweets linked to a user yet.
User
|> Ash.Query.load(:tweets)
|> Api.read()

# Query.filter using expression style syntax.
User
|> Ash.Query.filter(email == "james@marfugi.com")
|> Ash.Query.load(:tweets)
|> Api.read()

Sorting & Select

Query.sort is used for sorting and Query.select is used to include or remove columns from the result set. IEx will still show all of the attributes via IO.inspect() to the screen (because the User %Struct{} defines them), but those attributes not “selected” in the Ash.Query.select() result, will be be set to nil.

# Query.load using keyword style syntax. Won't show any related tweets yet because they haven't been linked to users yet.
User
|> Ash.Query.sort([:email])
|> Ash.Query.select([:email, :username])
|> Api.read()

Relationships

Relationships are typically the most complicated part of database application, so there’s a lot to cover here. This is in many ways the entire point of this livebook so we’ll go through various examples slowly, to try and cover the most common use cases. First, we’ll discuss how relationships are defined, how children are added and removed from parents, and how parents are changed from children.

Defining Relationships

Relationships are defined in Ash.Resource (e.g., in user.ex or tweet.ex) and “managed” through functions expecting an Ash.Changeset. When defining the child side of a parent/child relation, you don’t need to specify the foriegn keys explicitly, since Ash will create and use one for you automatically, using the convention of _id, (e.g., tweet.user_id).

# .. from tweet.ex (foreign key "user_id" is implicitly defined by convention)

defmodule FormValidator.Tweet do
  # ... snip ...
  # note -> :user_id is implicity defined
  relationships do
    belongs_to :user, FormValidator.User
  end
end

Of course, it’s still possible to define your foriegn keys explicitly if you prefer, though this is not generally recommended, since there’s a lot less noise in the code once you get comfortable with the idea that the foreign keys are still actually there, just working for you behind the scenes. Here’s how to define foreign key explicitly:

# From tweet.ex 
# - foreign key is defined explicitly via "attribute" 
# - automatic foreign key generation is "turned off" in relationships via "define_field? false" 

defmodule FormValidator.Tweet do
  # ... snip ...
  
  attribute :user_id, :uuid
  
  relationships do
    belongs_to :user, FormValidator.User
    define_field? false
  end
end

Managing Relationships - Configuration.

Relationships are manipulated primarily through the manage_relationship function that takes in an Ash.Changeset along with a set of potential changes that are passed in as either a single map, a list of maps or an Ash.Identity value. (Collectively, this is called “input”). Changes are applied from both sides, first by comparing the “input” to the “changeset” and then comparing the “changeset” to the proposed “input” changes.
Processing on the “input-to-changeset” side begins first and is configured through :on_lookup, :on_match, and :on_no_match options that specify the control logic to use depending upon whether the input item was found in the changeset or not. Matches are found based upon the primary key, so it or some other Ash.Identity should be present in the input. If an input item is found in the changeset, then the :on_match behavior is invoked, but if not found, then :on_lookup behavior is triggered to go and retrieve it from the database instead. At that point, either :on_match or :on_no_match is invoked depending upon whether the item was found or not found in the databsae. (:on_no_match is also invoked if :on_lookup was not specified or set to :ignore).

Next, after “input-to-changeset” processing has completed, the “changeset-to-input” process begins, examining each changeset item for a corresponding match on the input side. In this case, :on_missing logic triggered for any changeset item not found in the corresponding input map. The end result of these two processes can be visualized declaratively in the diagram below:

Managing Relationships - Operation Types

Rather than have to remember and specify each of these control options individually, we can instead group them together into an “operation type” that makes common manipulation settings easier. Operation types are specified via the :type option and correspond to the specifications below:

:replace --> [on_lookup: :relate, on_no_match: :error, on_match: :ignore, on_missing: :unrelate]
:append --> [on_lookup: :relate, on_no_match: :error, on_match: :ignore, on_missing: :ignore]
:remove  --> [on_no_match: :error, on_match: :unrelate, on_missing: :ignore]
:direct_control --> [on_lookup: :ignore, on_no_match: :create, on_match: :update, on_missing: :destroy]
:create --> [on_no_match: :create, on_match: :ignore]

Managing Relationships - Aliases

Finally, as an alternative to specifying operation types or setting relationship configuration options granularly, three function aliases are also available that encapsulate common settings for adding, replacing or removing relations in code. The grandular approach is commonly used when defining “actions” inside of your “resources” while the operation types and aliased functions are more commonly used in code.

append_to_relationship/4 is an alias for manage_relationship with operation type of :append. 
replace_relationship/4 is an alias for manage_relationship with operation type of :replace. 
remove_from_relationship/4 is an alias for manage_relationship with operation type of :remove

Relationships Examples

Example 1: Create a new child and save it (from Parent’s point of view)

First, we demonstrate how to :create a new child linked to an existing parent. We retrieve “james@marfugi.com” as sample user, define some new content to serve as the tweet, insert that tweet into the changeset via manage_relationship and finally, persist it to the database via Api.update()

# retrieve "james@marfugi.com" as parent into changeset. 
james_changeset =
  User
  |> Ash.Query.filter(email: "james@marfugi.com")
  |> Api.read_one!()
  |> Ash.Changeset.for_update(:update)

# specify input data to serve as content for the new tweet. 
input_tweet_data = %{body: "This is a new tweet from James"}

# insert `input_tweet_data` into the changeset via Ash.Changeset.manage_relationship
# note {tweet.user_id} will be added in by `manage_relationship` implicitly 
james_with_tweet_changeset =
  Ash.Changeset.manage_relationship(james_changeset, :tweets, input_tweet_data, type: :create)

# persist to the database. 
# tweet.user_id will be saved in in the SQL INSERT statement generated for us. 
# note: from Api point of view, the change is an update, since "updating" or changing the set of 
#        tweets tied to user.  We are adding Tweets, but chainging User.
Api.update(james_with_tweet_changeset)

Pipeline Through in One Step: We also join each step through together through elixir pipeline as in the same, but more compact example, below.

User
|> Ash.Query.filter(email: "james@marfugi.com")
|> Api.read_one!()
|> Ash.Changeset.for_update(:update)
|> Ash.Changeset.manage_relationship(:tweets, %{body: "another new tweet from james"},
  type: :create
)
|> Api.update()

Example 2: Update children’s foreign key pointer to change its relationship to parent (from Child’s point of view).

This example takes all of the tweets we created so far and assigns them to user “james@marfugi.com”, which shows us how to update a child’s foreign key pointer to its parent. First, we retrieve the :id of James from the database and save it as a variable. Next, we retrieve all of the tweets from the database and save them in a list. From there, we iterate across each item in the list, transforming each tweet into an Ash.Changeseet, changing it’s user_id foreign key to james.id and saving back to the database. Note that we have to use Ash.Changeset.force_change_attribute rather than regular change_attribute because remember, the user_id foreign key is being managed for us by Ash and is therefore set to “not writable” by default. force_change_attribute allows us to override this. We could also override by specifying attribute_writable? in the Resource definition, which will allow us to write to user_id manually if we want to as well.

# get james@marfugi's :id from the database. 
james_id =
  User
  |> Ash.Query.filter(email: "james@marfugi.com")
  |> Ash.Query.load(:tweets)
  |> Api.read_one!()
  |> Map.get(:id)

# get all the tweets, and save into list of %Tweets{} 
tweet_list = Api.read!(Tweet)

# iterate through each one, set it's user_id to `james_id` and persist to the database. 
Enum.each(tweet_list, fn tweet ->
  tweet
  |> Ash.Changeset.for_update(:update, %{})
  |> Ash.Changeset.force_change_attribute(:user_id, james_id)
  |> Api.update!()
end)

We would also handle this in a cleaner way via manage_relationship and not have to worry about force_changing the user_id attribute explicitly at all:

Enum.each(tweet_list, fn tweet ->
  tweet
  |> Ash.Changeset.for_update(:update)
  |> Ash.Changeset.manage_relationship(:user, james_id, type: :replace)
  |> Api.update()
end)

Example 3: Remove Tweet from User

To remove a tweet from a user, we need a way of identifying which one we want to delete, so we’ll need an identity value of some kind, which we’ll obtain via Enum from the tweets participating as children in the parent user changeset. This might apply when a “delete” button is clicked on a HTML form, for example. The form sends in a value that is used to drive the match in comparision to the changeset’s enum of tweets. In this example, we’ll peel off the first tweet from the top of the enum and use it’s primary key (tweet.id) as the identifier to be used as input to drive behavior of the manage_relationship call, which will match against that value and call on_match: with the :destroy instruction for the database.

Note that we’re using :destroy here, rather than :remove for deleting. Remove only disassociates the link or relationship (effectively setting the foreign_key to nil), which would not actually remove it from the databse. Destroy does that for us.

It’s also important to note that when we’re trying to delete or remove children from the parent, we doing so from the perspective of the parent, not the child. This means when we specify on_match: :destroy in the example, we’re talking about “tweets” that we want to destroy, but when we want to save to the databse, we’re doing so from the perspective of the parent User, which essentially amounts to an “update” to a set of tweets that is now smaller than before. That’s why we call Api.update() rather than Api.destroy() at the end of the expression.

# retrieve james preloaded with all of his tweets 
james_changeset =
  User
  |> Ash.Query.filter(email: "james@marfugi.com")
  |> Ash.Query.load(:tweets)
  |> Api.read_one!()
  |> Ash.Changeset.for_update(:update)

# pick-out first associated tweet and use the tweet.id from that as the one to delete. 
# to find a particular tweet.body, we could try `Enum.find(james_changeset.data.tweets, fn x -> x.body == "Tweet-4"end )`

input = %{id: Enum.at(james_changeset.data.tweets, 0).id}

Ash.Changeset.manage_relationship(james_changeset, :tweets, input, on_match: :destroy)
|> Api.update()

Example 4: Update Existing Tweet from User

Updating an existing tweet from parent side is essentially the same as example 3 above, except in this case, we specify an operation type of :update instead of :remove, and we use Enum.find/3 to target “Tweet-4” to change.

# retrieve james preloaded with all of his tweets 
james_changeset =
  User
  |> Ash.Query.filter(email: "james@marfugi.com")
  |> Ash.Query.load(:tweets)
  |> Api.read_one!()
  |> Ash.Changeset.for_update(:update)

# find 'Tweet-4' and grab the tweet.id from that as the one to change. 
tweet4 = Enum.find(james_changeset.data.tweets, fn x -> x.body == "Tweet-4" end)

# change the body attribute to simulate an update to the tweet content from parent user.
input = %{id: tweet4.id, body: "I just changed you."}

Ash.Changeset.manage_relationship(james_changeset, :tweets, input, on_match: :update)
|> Api.update()

Example 5: Change Parent via :on_lookup

This next example is pretty cool. We don’t actually have to get the user_id field from the databse first like we’ve been doing so far. Instead, since :email is defined as an Ash.Identity in Ash.Resource, we can reply upon :on_lookup behavior to do that for us automatically. In this example, we’ll change the parent user from the first Tweet in our database.

Tweet
|> Ash.Query.limit(1)
|> Api.read_one!()
|> Ash.Changeset.for_update(:update)
|> Ash.Changeset.manage_relationship(:user, %{email: "alice@wonderland.com"},
  on_lookup: :relate,
  on_no_match: :error
)
|> Api.update()

Actions

Now that we know how to manage_relationships manually, let’s figure out how to bake them into reusable actions that we can call more easily from our Api. We’ll refactor the examples above to use actions defined within Ash.Resource, instead of managing it outside manually. Actions have a name (like :add_tweets), they can accept parameters (defined as “arguments”), and while a single change or update behavior is usually all that’s needed, actions also allow multiple preparation or change steps, to handle more complex data manipulations. For :read type actions, we use Ash.Resource.Preparation steps allow us to build up filter and relation-loading operations before reading from the database, while create/update/destroy actions rely upon Ash.Resource.Change steps for manipulate data before persisting back to the database via Api.update(), Api.create() or Api.destroy().

Action :add_tweets

The User.add_tweets action takes in a tweet and adds it to the user. There are actually three “update” style actions defined for user - one for default, plus ones for add and remove tweets respectively. Also note that “change manage_relationship/3” call actually accepts three parameters, (the argument_name, relationship_name and processing keywords like :on_match, :on_lookup, etc). In this case, since the argument name (tweets) is the same as the relationship name (also tweets), we don’t have to type it in again twice.

Arguments: (definition versus use): You may notice the syntax used when defining arguments in the resource (e.g., argument :tweets, {:array, :map}), is a bit different than the syntax used to supply those values as parameters in the Api. That’s because the Api can handle not only the arguments you’ve pre-defined in a particular action, but all of the other attributes in the resource as well. They all come through together in a single, consolidated keyword list. So, the argument definition for :tweets as {:array, :map} simply means “there is an argument called ‘:tweets’ that will supply a list of maps”. When those tweets are eventually supplied in a for_update call, they will be packaged inside a keyword list with the key-handle of “tweets:” (plus a list of tweet values) along with any other attributes that might be needed, like :email or :username.

With that background in mind, here is the definition and use of the :add_tweets action, below.

  # DEFINITION of :add_tweets (from User.ex)
  #...snip...
  actions do
    defaults [:create, :read, :update, :destroy]

    update :add_tweets do
      argument :tweets, {:array, :map}
      change manage_relationship(:tweets, type: :create)
    end

    update :remove_tweets do
      argument :tweets, {:array, :string}
      change manage_relationship(:tweets, on_match: :destroy)
    end
  end

Based upon that definition, we can now invoke it to add one more or tweets via the following expression.

User
|> Ash.Query.limit(1)
|> Api.read_one!()
|> Ash.Changeset.for_update(:add_tweets, %{
  tweets: [%{body: "first new tweet"}, %{body: "Can add multiple new tweets"}],
  email: "james@marfugi.com",
  username: "james"
})
|> Api.update()

Using actions, we can remove them in much the same way. We pull first user from the database as a sample, preload the tweets and pick a random one from within the list to target for deletion. We get it’s id and then call the :remote_tweets action to destroy it from the database.

first_user =
  User
  |> Ash.Query.limit(1)
  |> Ash.Query.load(:tweets)
  |> Api.read_one!()

first_user_random_tweet =
  first_user.tweets
  |> Enum.take_random(1)
  |> hd()
  |> Map.get(:id)

first_user
|> Ash.Changeset.for_update(:remove_tweets, %{tweets: [first_user_random_tweet]})
|> Api.update()

Forms

Now we come to a lot of cool stuff. While Ash certainly provides a lot of functionality for the back end (like declaratively implementing data structure, validation, access and persistance of related tables to the database), it really begins to shine once you surface all of that functionality up to the front-end for use by clients as a “service”. In many cases, you would be done with just that, configuing the Ash.Resource to expose a JSON or GraphQL endpoint that can be consumed by various clients on the front end directly, using virtually any toolchain as desired (e.g. iOS, Android, web service, etc.) This, like most things in Ash, would be defined declaratively in the resource using either a json_api do or graphql do directive.

In our case, we’re going to be using Phoenix Liveview as the front-end so we can demonstrate how AshPhoenix.LiveView and AshPhoenix.Form make things easier for liveview developers. AshPhoenix.LiveView contains functions that make querying and updating ash resources easier (e.g. for “reading” or “listing” on the front end), while AshPhoenix.Form is used in “edit” or “add” forms to manipulate and persist that information back to the databse. Both modules are Ash.Resource aware and thus tied into the “actions” defined there, including child/parent relationships to other tables/resources as applicable.

AshPhoenix.Form

If you’re already familiar with Phoenix.HTML forms, the same concepts generally apply in AshPhoenix.Form as well, except for a few configuration options needed in the initialization call up front. To initialize a form, the liveview must know what values to use for original data and it must know the Ash resource Api it’s connected to in order to find the actions and validations to use to run the form.

To do this, it’s pretty much the same as in Phoenix Liveview. First we define a mount/3 function that locates or creates the data structure used to initialize the form, which will be packaged and presented for use in the form as an Ash.Changeset. Then, we connect the changeset to the form and specify the ash action that should be used to manipulate any edits. Finally, we define a template or render method for the front-end, and in that we bind back to the form that was created in mount/3 through LiveView’s standard syntax for forms (e.g., <.form let={f} for={@form}). It may sound and look a bit complicated (see image), but it’s pretty much all “normal” Phoenix LiveView stuff.

Line Items: Note the use of inputs_for to generate the line items. This helper is relationship aware and knows that there are “tweets” associated with this user. inputs_for will iterate through each tweet and render an input or other markup for each one of them. In this way, the entire changeset (both parent and children) are editable as a single unit and the entire result can be written back to the databse exactly as it appears on screen.

Pagination

TO COME

# todo 

Changesets

Changesets.new is used rarely, when you want a “light touch” on validations as when working with internal data that you know is already sound. It’s almost always better to use “Changeset.” instead (e.g. Changeset.for_create/2, Changeset.for_destroy/3, Changeset.for_update/4), since the validations applied there will align with the type of “for_action” being applied. This is made more clear in the examples below.

# internal data, will be valid
loose_validation = Ash.Changeset.new(User)
IO.inspect(loose_validation.valid?, label: "Is Changeset.new() valid?")

# validation is more rigorous using "for_create" than just "new"
strict_validation_missing_email = Ash.Changeset.for_create(User, :create, %{})
# false
IO.inspect(strict_validation_missing_email.valid?, label: "Is Changeset.for_create() valid?: ")

# passes validations, true
strict_validation_has_email =
  Ash.Changeset.for_create(User, :create, %{email: "super@de-duper.com"})

IO.inspect(strict_validation_has_email.valid?, label: "Is Changeset.for_create valid now?: ")

HEEX Template Notes

when you want the render the output in HTML, then you need the <%= %> when you want to pass an expression from within HEEX or HTML tag to another component, you use {} when calling a component, like <.tweet tweet_form = {foo} />, the “assigns” parameter in your components’s rendering method will have the properties you passed into it attached to that assignes (e.g. assigns.tweet_form). Therefore, to access its values you can just reference the property prefixed by the “assigns” operator (e.g. @tweet_form)

<%= %> means to run elixir code dynamically within a HEEX template. {} means to run elixir code dynamically within a tag (e.g. ‘<>’ like live_component or html_element)

or <.table id={"hello"} >

@ means to reference an assigns variable from within a HEEX template.