Diffo TMF Service and Resource Manager

Mix.install(
  [
    {:diffo, "~> 0.1"}
  ],
  consolidate_protocols: false
)

Overview

Diffo is a Telecommunications Management Forum (TMF) Service and Resource Manager, built for autonomous networks.

It is implemented using the Ash Framework leveraging core and community extensions including some created and maintained by diffo-dev. As such it is highly customizable using Spark DSL and as necessary Elixir. If you are not already familiar with Ash then please explore Ash Get Started

In this livebook tutorial you will learn:

• Installing Neo4j and Configuring Boltx
• Implementing a Minimal Service
• Adding Initial Parties and Places
• Defining Service Features and Characteristics
• Setting Service Expectations
• Resolving the Outstanding Service
• Advancing Service State
• Relating a Service with other Services and Resources

Installing Neo4j and Configuring Boltx

Diffo uses the Ash Neo4j DataLayer, which requires Neo4j to be installed

While Neo4j community edition is open source and you can build from source it is likely that you’ll use an installation.

AshNeo4j uses neo4j which must be installed and running. You can install latest major Neo4j versions from the community tab at Neo4j Deployment Center, or use the 5.26.8 direct link

When you install neo4j you’ll typically have a default username and password. Take note of this and any other non-standard config.

Update the configuration below as necessary and evaluate.

config = [
  uri: "bolt://localhost:7687",
  auth: [username: "neo4j", password: "password"],
  user_agent: "diffoLivebook/1",
  pool_size: 15,
  max_overflow: 3,
  prefix: :default,
  name: Bolt,
  log: false,
  log_hex: false
]

Boltx needs a process in your supervision tree, this will start one with the config if not already running:

AshNeo4j.BoltxHelper.start(config)

Now you should be able to verify that Neo4j is running:

AshNeo4j.BoltxHelper.is_connected()

You can get all nodes related to other nodes the following query:

AshNeo4j.Cypher.run("MATCH (n1)-[r]->(n2) RETURN r, n1, n2 LIMIT 25")

It is helpful to have a Neo4j browser open locally, typically:

http://localhost:7474/browser/

Once you connect and issue a query like the one above you’ll be able to explore the results interactively.

OPTIONAL If you want to clear your database you can evaluate:

AshNeo4j.Neo4jHelper.delete_all()

Implementing a Minimal Service

The key resources we will use here are here are Diffo.Provider.Instance and Diffo.Provider.Specification. Logically a Specification instance SPECIFIES an Instance. The specification has a uuid (which you can set so is consistent in catalog/dev/prod). Convention is to issue a new uuid for each major version. The specification supports actions for incrementing minor and patch versions, and calculates the version as v…. It also calculates a specification href based on the specification type and the uuid.

Firstly we require Ash.Query and set some aliases for convenience

require Ash.Query
alias Diffo.Provider
alias Diffo.Provider.Specification
alias Diffo.Provider.Instance
alias Diffo.Provider.Feature
alias Diffo.Provider.Characteristic
alias Diffo.Provider.Reference
alias Diffo.Provider.Party
alias Diffo.Provider.PartyRef
alias Diffo.Provider.Place
alias Diffo.Provider.PartyRef
alias Diffo.Uuid
import Jason, only: [encode: 2]
use Outstand

We can either create specification instances with Ash directly, or use the Diffo.Provider code interface.

broadband_v1 = Provider.create_specification!(%{name: "broadband"})
nbn_ethernet_v1 = Provider.create_specification!(%{name: "nbnEthernet", type: :resourceSpecification})
mobile_backup_v2 = Provider.create_specification!(%{name: "mobileBackup", major_version: 2})
esim_v1 = Provider.create_specification!(%{name: "esim", type: :resourceSpecification})
specifications = [broadband_v1, mobile_backup_v2, nbn_ethernet_v1, esim_v1]

Now we create some service instances using these specifications:

We can also then create some service and resource instances

broadband_0001 = Provider.create_instance!(%{name: "broadband_0001", specified_by: broadband_v1.id})
mobile_backup_0001 = Provider.create_instance!(%{name: "mobileBackup_0001", specified_by: mobile_backup_v2.id})
services = [broadband_0001, mobile_backup_0001]

esim_0001 = Provider.create_instance!(%{name: "esim_0001", type: :resource, specified_by: esim_v1.id})
nbn_ethernet_0001 = Provider.create_instance!(%{name: "nbnEthernet_0001", type: :resource, specified_by: nbn_ethernet_v1.id})
resources = [esim_0001, nbn_ethernet_0001]

Look in your Neo4j browser you should see 4 specification nodes each SPECIFIES a single instance node. The nodes will have properties, but not properties that relate to relationships or calculations.

Each Diffo.Provider resource supports Elixir Jason protocol, so we can simply encode the specifications, services or resources (try them all), even while they are in collections:

Jason.encode!(specifications, pretty: true) |> IO.puts

We’ll now perform an action on a specification, updating it’s minor version. We’d typically do this when we’ve enhanced the service in a back compatible way, perhaps by adding an optional feature. When we reload the specified service instance we should see that it has an incremented minor version in the middle of the version string (this resets the patch version)

broadband_v1 = broadband_v1 |> Diffo.Provider.next_minor_specification!()
Jason.encode!(broadband_v1, pretty: true) |> IO.puts
broadband_0001 = Ash.reload!(broadband_0001)
Jason.encode!(broadband_0001, pretty: true) |> IO.puts

Adding Parties and Places

Typically a Service instance will have various relationships with both Party and Place instances.

Diffo models PartyRef and PlaceRef which relate Instances and Party/Place resources. The PartyRef and PlaceRef each serve as a connector resource connecting a single Instance with a single Party/Place, and holding the role (of the Party/Place to the service).

For our broadband service instance we’ll add both Customer and Reseller parties to our service, and we’ll add a CustomerSite place where the service is to be delivered to the customer.

individual =
  Diffo.Provider.create_party!(%{
    id: "IND000000897354",
    name: :individualId,
    referredType: :Individual
  })
org =
  Diffo.Provider.create_party!(%{
    id: "ORG000000123456",
    name: :organizationId,
    referredType: :Organization
  })
parties = [individual, org]
Jason.encode!(parties, pretty: true) |> IO.puts

We’ll then relate these parties to the service in their roles using PartyRef, which we’ll also encode to json in the TMF way, where the PlaceRef references rather than embeds the Place.

site_contact = Diffo.Provider.create_party_ref!(%{instance_id: broadband_0001.id, role: :SiteContact, party_id: individual.id})
reseller = Diffo.Provider.create_party_ref!(%{instance_id: broadband_0001.id, role: :Reseller, party_id: org.id})
parties = [site_contact, reseller]
Jason.encode!(parties, pretty: true) |> IO.puts

We’ll also add a CustomerSite place where the service is to be delivered to the customer. Historically this is the Z-end.

z_end = Provider.create_place!(%{id: "1657363", name: :addressId, href: "place/telstra/1657363", referredType: :GeographicAddress})
Jason.encode!(z_end, pretty: true) |> IO.puts

And we’ll relate this place to our broadband service using a PlaceRef

z_end_place_ref = Diffo.Provider.create_place_ref(%{instance_id: broadband_0001.id, role: :CustomerSite, place_id: z_end.id})
Jason.encode!(z_end_place_ref, pretty: true) |> IO.puts

Now we should be able to refresh the broadband service and see it’s related parties and places.

broadband_0001 = Diffo.Provider.get_instance_by_id!(broadband_0001.id)
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

Defining Service Features and Characteristics

Services are defined by their Features and Characteristics. Service Features can be enabled or disabled, and can themselves be defined by their own Feature Characteristics, typically used as configuration of the feature. Service Characteristics are directly on the service and often contain a mix of configuration and status attributes.

Generally when a service is being provisioned Feature and Characteristic attributes are injected/defaulted/derived, and while a service is being operated Characteristic attributes are ‘live’, reflecting aspects of the running services.

We’ll initially update our broadband service with some high level Features and Characteristics refecting the customer’s intent, which we’ll later use in feasibility checking the service.

First we’ll create a backup feature to indicate that we want mobile backup, and a device management feature to say we want our customer gateway remote managed.

We’ll create an e2e ‘instance’ characteristic to detail our bandwidth and latency requirements:

e2e = Provider.create_characteristic!(%{type: :instance, name: :e2e, value: %{bandwidth: %{downstream: 250, upstream: 25, units: :Mbps}}})
broadband_0001 = broadband_0001 
  |> Provider.relate_instance_characteristics!(%{characteristics: [e2e.id]})
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

We’ll create mobile backup and device management features. The device management feature will also have a characteristic defining some required options:

backup = Provider.create_feature!(%{name: :backup, isEnabled: true})
options = Provider.create_characteristic!(%{type: :feature, name: :options, value: [:updates, :monitoring]})
device_management = Provider.create_feature!(%{name: :deviceManagement, isEnabled: true, characteristics: [options.id]})
broadband_0001 = broadband_0001 
  |> Provider.relate_instance_features!(%{features: [backup.id, device_management.id]}) 
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

Setting Service Expectations

We will set service expectations as goals. These can be generic goals for a particular stage of a service lifecycle (feasiblityChecked for example), or could be specific to a particular instance. Expectations can be high level, representing consumer or provider intent, and also can be low level, such as to how the service should be composed, configured and operated. The low level expectations can be derived from a combination of intent, policy, current expectations and current actual.

We can write this in elixir as:

expected = fun(intended, expected, actual)

We can construct expectations using the struct Ash generates for our resources. Expectations may include functions and/or concrete values. We need to maintain compatibility with our Ash Resource attribute types, which does limit our ability to validate/persist resources using Ash.

We’ll construct a simple set of expectations for a feasible service:

expected_specification = %{name: "broadband", major_version: 1}
expected_features = [%{name: :backup, isEnabled: true}, %{name: :deviceManagement, isEnabled: true}]
expected_characteristics = [
    %{name: :e2e},
    %{name: :technology, value: %{access: {&Outstand.one_of/2, [:nbnEthernet, :fixed4g, :fixed5g]}}}
  ]
expected_places = [
  %{role: :AccessNNI},
  %{role: :CustomerSite, place: %{name: "locationId"}},
  %{role: :CustomerSite, place: %{name: "addressId"}},
  %{role: :ServingArea}
]
expected_parties = [%{role: :Provider}, %{role: :Reseller}, %{role: :SiteContact}]
expected_instance = %{
  id: &Uuid.expect_uuid4/1,
  specification: expected_specification, 
  service_state: :feasibilityChecked,
  service_operating_status: :feasible, 
  characteristics: expected_characteristics, 
  places: expected_places,
  parties: expected_parties}

The reason we set expectations is so that we can check whether our expectations are met (or exceeded) by the actuality. Unmet expectations represent ‘outstanding’ work. We’ll calculate outstanding by comparing our expected_instance with the (actual) broadband_0001 service from the database:

broadband_0001 = Diffo.Provider.get_instance_by_id!(broadband_0001.id)
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts
outstanding = expected_instance --- broadband_0001

Resolving the Outstanding Service

We’ll have a few things outstanding which we would normally find out during service qualification. We expect a technology Characteristic, a provider Party and some network Places relating to service hinterland and edge location.

We recommend using outstanding to drive next task logic, so that the orchestration is directed by the difference engine. This could look like an address lookup (where we learn the provider) followed by a provider service qualification (where we learn the technology) and the related network places.

nbn = Provider.create_party!(%{id: :nbn, name: "NBNCo", referredType: :Organization})
nbn_party_ref = Provider.create_party_ref!(%{instance_id: broadband_0001.id, party_id: nbn.id, role: :Provider})
provider_z_end = Provider.create_place!(%{id: "LOC000000899353", name: "locationId", href: "place/nbnco/LOC000000899353",referredType: :GeographicAddress})
provider_z_end_place_ref = Provider.create_place_ref!(%{instance_id: broadband_0001.id, role: :CustomerSite, place_id: provider_z_end.id})
csa = Provider.create_place!(%{id: "CSA200000000685", name: "csaId", href: "place/nbnco/CSA200000000685", referredType: :GeographicLocation})
csa_place_ref = Provider.create_place_ref!(%{instance_id: broadband_0001.id, role: :ServingArea, place_id: csa.id})
poi = Provider.create_place!(%{id: "2CAR", name: "poiId", href: "place/nbnco/2CAR",referredType: :GeographicSite})
poi_place_ref = Provider.create_place_ref!(%{instance_id: broadband_0001.id, role: :AccessNNI, place_id: poi.id})

places = Diffo.Provider.list_place_refs!()
places |> Jason.encode!(pretty: true) |> IO.puts

This should have resolved the Parties and Places in the outstanding service:

broadband_0001 = Diffo.Provider.get_instance_by_id!(broadband_0001.id)
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts
outstanding = expected_instance --- broadband_0001

We create and add the technology characteristic to the actual service. This should resolve the characteristic expectation.

technology = Provider.create_characteristic!(%{type: :instance, name: :technology, value: %{access: :nbnEthernet}})
broadband_0001 |> Provider.relate_instance_characteristics!(%{characteristics: [technology.id]})


broadband_0001 = Diffo.Provider.get_instance_by_id!(broadband_0001.id)

outstanding = expected_instance --- broadband_0001

We simply need to advance the services state to end the feasiblity

Advancing Service State

TMF638 Services implement a service_state and also have an service_operating_status. The service we created earlier was in the :initial service state.

Diffo uses the AshStateMachine Extension. By default we allow all TMF638 state transitions, but these are customisable by manipulating the Instance DSL.

From the initial state we can cancel, activate, feasibilityCheck or reserve the service. From feasibilityChecked state we can cancel, activate or reserve the service. From active state we can deactivate, suspend or terminate the service From inactive or suspended state we can activate or terminate the service. No transitions are possible from terminated or cancelled.

Given that our feasibilityCheck above was complete we want to set the :feasibilityChecked service_state, and because it was successful we’ll set the :feasible service_operating_status:

broadband_0001 = broadband_0001 |> Provider.feasibilityCheck_service!(%{service_operating_status: :feasible})
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

This should have resolved the last part of the outstanding service:

broadband_0001 = Diffo.Provider.get_instance_by_id!(broadband_0001.id)

outstanding = expected_instance --- broadband_0001

AshStateMachine will also error on invalid transitions.

Lets try and terminate the feasibilityChecked service (this is not allowed, instead cancel)

{:error, error} = broadband_0001 |> Provider.terminate_service()
error

Relating a Service with other Services and Resources

The power Diffo has to simplify orchestration comes through combination of the outstanding protocol and the relationships a parent Service has with other Services and Resources. We will relate the services and resource instances we made earlier with our broadband service.

This is done with a Relationship resource, which links the source and target Instances. Relationship with source_id are stored on the source in the forward_relationships[] and be listed in TMF service serviceRelationship[] or resourceRelationship[] json accordingly.

Each Relationship may have an alias, which must be unique for the source Instance. Alias’s are important for reasoning about a Relationship independently of the target id, which intially may not be known or may change over the life of the composite service. The targets of aliased Relationships are considered TMF service supportingService[] or supportingResource[] accordingly.

If we went on to activate our broadband service, we’d create/acquire child services and resources, then configure and manage them over the life of the service. Both the composition and the orchestration can be dynamic.

Let’s create a ‘parent->child’ relationship of the broadband service to its mobileBackup service:

relationship = Provider.create_relationship!(%{
  alias: "backup", 
  source_id: broadband_0001.id,
  target_id: mobile_backup_0001.id,
  type: :bestows
})
relationship |> Jason.encode!(pretty: true) |> IO.puts

Now when we refresh the broadband_0001 service it should have both a serviceRelationship[] and supportingService[]:

broadband_0001 = Provider.get_instance_by_id!(broadband_0001.id)
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

We’ll now relate sim resource with the mobile backup service

relationship = Provider.create_relationship!(%{
  alias: :sim, 
  source_id: mobile_backup_0001.id,
  target_id: esim_0001.id,
  type: :isAssigned
})
relationship |> Jason.encode!(pretty: true) |> IO.puts

Now when we refresh the mobile_backup_0001 service it should have both a resourceRelationship[] and supportingResource[]:

mobile_backup_0001 = Provider.get_instance_by_id!(mobile_backup_0001.id)
mobile_backup_0001 |> Jason.encode!(pretty: true) |> IO.puts

We’ll also add the nbnEthernet resource directly to the broadband service:

relationship = Provider.create_relationship!(%{
  alias: :access, 
  source_id: broadband_0001.id,
  target_id: nbn_ethernet_0001.id,
  type: :isAssigned
})
relationship |> Jason.encode!(pretty: true) |> IO.puts

broadband_0001 = Provider.get_instance_by_id!(broadband_0001.id)
broadband_0001 |> Jason.encode!(pretty: true) |> IO.puts

What Next?

In this tutorial you’ve used Diffo to create, relate and update some TMF Service and Resources, simulating activities over the service and resource lifecycle, and you’ve learned how this functionality is underpinned by open source Neo4j, Ash Framework and Elixir.

If you find Diffo useful please visit and star on github. Feel free to join discussions and raise issues to discuss PR’s.