# Ch 3: Harness the Power of Math

```
Mix.install([
{:nx, "~> 0.7.2"},
{:exla, "~> 0.7.2"},
{:kino, "~> 0.12.3"},
{:stb_image, "~> 0.6.8"},
{:vega_lite, "~> 0.1.9"},
{:kino_vega_lite, "~> 0.1.11"}
])
```

## Setup

`Nx.default_backend(EXLA.Backend)`

`{Nx.BinaryBackend, []}`

## The Building Blocks of Linear Algebra

The fundamental object in Linear Algebra is the *vector*. `Nx`

doesn’t explicitly differentiate between scalars, vectors and matrices. Everythin in `Nx`

is a tensor

```
a = Nx.tensor([1, 2, 3])
b = Nx.tensor([4.0, 5.0, 6.0])
c = Nx.tensor([1, 0, 1], type: {:u, 8})
IO.inspect(a, label: :a)
IO.inspect(b, label: :b)
IO.inspect(c, label: :c)
```

```
a: #Nx.Tensor<
s64[3]
EXLA.Backend
[1, 2, 3]
>
b: #Nx.Tensor<
f32[3]
EXLA.Backend
[4.0, 5.0, 6.0]
>
c: #Nx.Tensor<
u8[3]
EXLA.Backend
[1, 0, 1]
>
```

```
#Nx.Tensor<
u8[3]
EXLA.Backend
[1, 0, 1]
>
```

### Important Operations in Linear Algrebra

Some of the most important and fundamental operations in linear algebra are:

- Vector addition
- Scalar multiplication
- Transpose
- Linear Transformations

```
sales_day_1 = Nx.tensor([32, 10, 14])
sales_day_2 = Nx.tensor([10, 24, 21])
```

```
#Nx.Tensor<
s64[3]
EXLA.Backend
[10, 24, 21]
>
```

`total_sales = Nx.add(sales_day_1, sales_day_2)`

```
#Nx.Tensor<
s64[3]
EXLA.Backend
[42, 34, 35]
>
```

```
keep_rate = 0.9
unreturned_sales = Nx.multiply(keep_rate, total_sales)
```

```
#Nx.Tensor<
f32[3]
EXLA.Backend
[37.79999923706055, 30.599998474121094, 31.5]
>
```

```
price_per_product = Nx.tensor([9.95, 10.95, 5.99])
revenue_per_product = Nx.multiply(unreturned_sales, price_per_product)
```

```
#Nx.Tensor<
f32[3]
EXLA.Backend
[376.1099853515625, 335.0699768066406, 188.68499755859375]
>
```

```
sales_matrix =
Nx.tensor([
[32, 10, 14],
[10, 24, 21]
])
Nx.transpose(sales_matrix)
```

```
#Nx.Tensor<
s64[3][2]
EXLA.Backend
[
[32, 10],
[10, 24],
[14, 21]
]
>
```

```
Kino.FS.file_path("Cat.jpg")
|> StbImage.read_file!()
|> StbImage.resize(256, 256)
|> StbImage.to_nx()
|> Nx.as_type({:u, 8})
|> Kino.Image.new()
```

```
invert_color_channels =
Nx.tensor([
[-1, 0, 0],
[0, -1, 0],
[0, 0, 1]
])
Kino.FS.file_path("Cat.jpg")
|> StbImage.read_file!()
|> StbImage.resize(256, 256)
|> StbImage.to_nx()
|> Nx.dot(invert_color_channels)
|> Nx.as_type({:u, 8})
|> Kino.Image.new()
```

## Thinking Probablistically

Probability helps you think about and make predictions from data

The difference between modern approaches and previous generations of ML, largely based on logic, is the acceptance of probabilistic methods over logic-based approaches

There are three primary tools used in ML to reason about data and make predictions: *probability theory*, *decision theory* and *information theory*

### Reasoning about Uncertainty

Probability theory is a framework for understanding uncertainty. Uncertainty is a fact of life therefore it is best to embrace it rather than ignore it. ML shines in uncertain situations because uncertainty is a built-in assumption

### All about Bayes

Bayes Theorem describes the *conditional probability* of an event based on available information or the occurence of another event. Bayes Theorem is a rule that describes how probabilities should be updated in the face of new evidence

As a ML model is trained it is making *inferences* or predictions about it’s surroundings (data). After a prediction is made, a decision must be made

### Making Decisions

Decision Theory provides a framework for acting optimally in the presence of uncertainty. When an ML model is making decisions in the presence of uncertainty, the model must consider the objective in order to make decisions after inference

### Learning from Observations

Information theory provides a framework for reasoning about information systems. Information theory is a mathematical study of coding information in sequences and symbols and the study of how much information can be stored and transmitted in these mediums.

In Information theory, the main concern is the degree of surprise of a particular observation

## Tracking Changes

While exploring the field of ML, the terms *derivative*, *gradient* and *automatic differentiation* come up often. All of these terms come from Vector calculus

Vector calculus is concerned with *differentiation* and *integration* of *vector fields* or functions

### Understanding Differentiation

The most important concept from calculus to understand in ML is the *derivative*. A derivative is a measure of the instantaneous rate of change of a function

For example, to maximize profits for the year, you could compute the derivative of your profit function to determine the amount of a product which maximizes profit

```
defmodule BerryFarm do
import Nx.Defn
defn profits(trees) do
trees
|> Nx.subtract(1)
|> Nx.pow(4)
|> Nx.negate()
|> Nx.add(Nx.pow(trees, 3))
|> Nx.add(Nx.pow(trees, 2))
end
defn profits_derivative(trees) do
grad(trees, &profits/1)
end
end
```

`{:module, BerryFarm, <<70, 79, 82, 49, 0, 0, 12, ...>>, true}`

`grad/2`

is an `Nx`

function which is capable of taking the *gradient* of a function

The gradient is the direction of greatest change of a scalar function. The gradient is like a heat-seeking missle attracted to change

An analogy, you are on a boat in a lake and you want to find the location of the deepest point in the lake as quickly as possible using a depth finder, how do you solve this problem? One method is to compute the depth-change in every direction and always travel in the direction of steepest descent, that way you always travel in the direction of steepest descent. That way you can guarentee you’re always following a path towards a deeper depth in the lake. If the bottom of the lake resembles a valley, you’re basically guarenteed to find the steepest point

From the analogy, the optimization process of objective functions in ML models is the same process using gradient descent. Gradient descent is useful because often times you don’t have access to the complete gradient of the objective function. Instead the model needs to look at a few samples at a time and move accordingly

*Gradient-based optimization* is a powerful form of optimization and is widely used in ML, especially DL. With automatic differentiation, all you need is an objective function and you can optimize it with respect to some model parameters

```
trees = Nx.linspace(0, 4, n: 100)
profits = BerryFarm.profits(trees)
profits_derivative = BerryFarm.profits_derivative(trees)
VegaLite.new(title: "Berry Profits", width: 600, height: 400)
|> VegaLite.data_from_values(%{
trees: Nx.to_flat_list(trees),
profits: Nx.to_flat_list(profits),
profits_derivative: Nx.to_flat_list(profits_derivative)
})
|> VegaLite.layers([
VegaLite.new()
|> VegaLite.mark(:line, interpolate: :basis)
|> VegaLite.encode_field(:x, "trees", type: :quantitative)
|> VegaLite.encode_field(:y, "profits", type: :quantitative),
VegaLite.new()
|> VegaLite.mark(:line, interpolate: :basis)
|> VegaLite.encode_field(:x, "trees", type: :quantitative)
|> VegaLite.encode_field(:y, "profits_derivative", type: :quantitative)
|> VegaLite.encode(:color, value: "#ff0000")
])
```

`{"$schema":"https://vega.github.io/schema/vega-lite/v5.json","data":{"values":[{"profits":-1.0,"profits_derivative":4.0,"trees":0.0},{"profits":-0.8462191820144653,"profits_derivative":3.620183229446411,"trees":0.04040404036641121},{"profits":-0.706821620464325,"profits_derivative":3.287757396697998,"trees":0.08080808073282242},{"profits":-0.5799248218536377,"profits_derivative":3.001140832901001,"trees":0.12121212482452393},{"profits":-0.46370965242385864,"profits_derivative":2.7587499618530273,"trees":0.16161616146564484},{"profits":-0.3564212918281555,"profits_derivative":2.5590012073516846,"trees":0.20202019810676575},{"profits":-0.2563686668872833,"profits_derivative":2.4003114700317383,"trees":0.24242424964904785},{"profits":-0.16192500293254852,"profits_derivative":2.2810988426208496,"trees":0.28282827138900757},{"profits":-0.0715271383523941,"profits_derivative":2.199779748916626,"trees":0.3232323229312897},{"profits":0.01632404327392578,"profits_derivative":2.154770851135254,"trees":0.3636363744735718},{"profits":0.10306348651647568,"profits_derivative":2.144489288330078,"trees":0.4040403962135315},{"profits":0.19006246328353882,"profits_derivative":2.1673526763916016,"trees":0.4444444477558136},{"profits":0.2786282002925873,"profits_derivative":2.2217769622802734,"trees":0.4848484992980957},{"profits":0.37000375986099243,"profits_derivative":2.3061797618865967,"trees":0.5252525210380554},{"profits":0.46536850929260254,"profits_derivative":2.418977975845337,"trees":0.5656565427780151},{"profits":0.5658379197120667,"profits_derivative":2.558588743209839,"trees":0.6060606241226196},{"profits":0.6724629998207092,"profits_derivative":2.723428726196289,"trees":0.6464646458625793},{"profits":0.786231279373169,"profits_derivative":2.911914825439453,"trees":0.6868686676025391},{"profits":0.90806645154953,"profits_derivative":3.122464418411255,"trees":0.7272727489471436},{"profits":1.0388275384902954,"profits_derivative":3.353494167327881,"trees":0.7676767706871033},{"profits":1.1793103218078613,"profits_derivative":3.603421211242676,"trees":0.808080792427063},{"profits":1.3302464485168457,"profits_derivative":3.8706626892089844,"trees":0.8484848737716675},{"profits":1.4923030138015747,"profits_derivative":4.153635025024414,"trees":0.8888888955116272},{"profits":1.6660840511322021,"profits_derivative":4.450756072998047,"trees":0.9292929172515869},{"profits":1.8521294593811035,"profits_derivative":4.760441780090332,"trees":0.9696969985961914},{"profits":2.0509138107299805,"profits_derivative":5.081110000610352,"trees":1.0101009607315063},{"profits":2.262850522994995,"profits_derivative":5.411177158355713,"trees":1.0505050420761108},{"profits":2.4882864952087402,"profits_derivative":5.749061107635498,"trees":1.0909091234207153},{"profits":2.7275047302246094,"profits_derivative":6.09317684173584,"trees":1.1313130855560303},{"profits":2.980726718902588,"profits_derivative":6.441944122314453,"trees":1.1717171669006348},{"profits":3.24810791015625,"profits_derivative":6.793778419494629,"trees":1.2121212482452393},{"profits":3.5297389030456543,"profits_derivative":7.147095680236816,"trees":1.2525252103805542},{"profits":3.8256494998931885,"profits_derivative":7.500314712524414,"trees":1.2929292917251587},{"profits":4.135802745819092,"profits_derivative":7.8518524169921875,"trees":1.3333333730697632},{"profits":4.460097789764404,"profits_derivative":8.200122833251953,"trees":1.3737373352050781},{"profits":4.798373222351074,"profits_derivative":8.543547630310059,"trees":1.4141414165496826},{"profits":5.150400161743164,"profits_derivative":8.88054084777832,"trees":1.454545497894287},{"profits":5.5158843994140625,"profits_derivative":9.209519386291504,"trees":1.494949460029602},{"profits":5.894474029541016,"profits_derivative":9.528902053833008,"trees":1.5353535413742065},{"profits":6.285747528076172,"profits_derivative":9.837104797363281,"trees":1.575757622718811},{"profits":6.6892194747924805,"profits_derivative":10.13254165649414,"trees":1.616161584854126},{"profits":7.104344844818115,"profits_derivative":10.41363525390625,"trees":1.6565656661987305},{"profits":7.530511379241943,"profits_derivative":10.678799629211426,"trees":1.696969747543335},{"profits":7.967041969299316,"profits_derivative":10.926450729370117,"trees":1.73737370967865},{"profits":8.413199424743652,"profits_derivative":11.155006408691406,"trees":1.7777777910232544},{"profits":8.868179321289062,"profits_derivative":11.362886428833008,"trees":1.8181818723678589},{"profits":9.331111907958984,"profits_derivative":11.548501968383789,"trees":1.8585858345031738},{"profits":9.801070213317871,"profits_derivative":11.710275650024414,"trees":1.8989899158477783},{"profits":10.277055740356445,"profits_derivative":11.846620559692383,"trees":1.9393939971923828},{"profits":10.75800895690918,"profits_derivative":11.95595645904541,"trees":1.9797979593276978},{"profits":11.242805480957031,"profits_derivative":12.036697387695312,"trees":2.0202019214630127},{"profits":11.730262756347656,"profits_derivative":12.087263107299805,"trees":2.060606002807617},{"profits":12.219127655029297,"profits_derivative":12.106069564819336,"trees":2.1010100841522217},{"profits":12.708084106445312,"profits_derivative":12.091534614562988,"trees":2.141414165496826},{"profits":13.19575309753418,"profits_derivative":12.042073249816895,"trees":2.1818182468414307},{"profits":13.68069076538086,"profits_derivative":11.956103324890137,"trees":2.222222328186035},{"profits":14.16138744354248,"profits_derivative":11.832043647766113,"trees":2.2626261711120605},{"profits":14.636279106140137,"profits_derivative":11.66830825805664,"trees":2.303030252456665},{"profits":15.10372543334961,"profits_derivative":11.463315963745117,"trees":2.3434343338012695},{"profits":15.562030792236328,"profits_derivative":11.21548080444336,"trees":2.383838415145874},{"profits":16.009429931640625,"profits_derivative":10.923226356506348,"trees":2.4242424964904785},{"profits":16.444095611572266,"profits_derivative":10.584964752197266,"trees":2.464646577835083},{"profits":16.864139556884766,"profits_derivative":10.199111938476562,"trees":2.5050504207611084},{"profits":17.26760482788086,"profits_derivative":9.764086723327637,"trees":2.545454502105713},{"profits":17.65247344970703,"profits_derivative":9.27830696105957,"trees":2.5858585834503174},{"profits":18.016660690307617,"profits_derivative":8.740188598632812,"trees":2.626262664794922},{"profits":18.35802459716797,"profits_derivative":8.148149490356445,"trees":2.6666667461395264},{"profits":18.67435073852539,"profits_derivative":7.50060510635376,"trees":2.7070705890655518},{"profits":18.963363647460938,"profits_derivative":6.79597282409668,"trees":2.7474746704101562},{"profits":19.222728729248047,"profits_derivative":6.032668590545654,"trees":2.7878787517547607},{"profits":19.450042724609375,"profits_derivative":5.209111213684082,"trees":2.8282828330993652},{"profits":19.642837524414062,"profits_derivative":4.3237175941467285,"trees":2.8686869144439697},{"profits":19.798580169677734,"profits_derivative":3.3749046325683594,"trees":2.909090995788574},{"profits":19.91468048095703,"profits_derivative":2.3610944747924805,"trees":2.9494948387145996},{"profits":19.988475799560547,"profits_derivative":1.2806897163391113,"trees":2.989898920059204},{"profits":20.017250061035156,"profits_derivative":0.1321239471435547,"trees":3.0303030014038086},{"profits":19.998214721679688,"profits_derivative":-1.0862011909484863,"trees":3.070707082748413},{"profits":19.928516387939453,"profits_derivative":-2.375861167907715,"trees":3.1111111640930176},{"profits":19.80524444580078,"profits_derivative":-3.738431453704834,"trees":3.151515245437622},{"profits":19.62541961669922,"profits_derivative":-5.175499439239502,"trees":3.1919190883636475},{"profits":19.385997772216797,"profits_derivative":-6.688662528991699,"trees":3.232323169708252},{"profits":19.083873748779297,"profits_derivative":-8.279485702514648,"trees":3.2727272510528564},{"profits":18.715879440307617,"profits_derivative":-9.949562072753906,"trees":3.313131332397461},{"profits":18.27878189086914,"profits_derivative":-11.700475692749023,"trees":3.3535354137420654},{"profits":17.769275665283203,"profits_derivative":-13.533799171447754,"trees":3.39393949508667},{"profits":17.184011459350586,"profits_derivative":-15.451114654541016,"trees":3.4343433380126953},{"profits":16.51955223083496,"profits_derivative":-17.454015731811523,"trees":3.4747474193573},{"profits":15.772408485412598,"profits_derivative":-19.544090270996094,"trees":3.5151515007019043},{"profits":14.939033508300781,"profits_derivative":-21.722911834716797,"trees":3.555555582046509},{"profits":14.015803337097168,"profits_derivative":-23.992055892944336,"trees":3.5959596633911133},{"profits":12.999043464660645,"profits_derivative":-26.353118896484375,"trees":3.6363637447357178},{"profits":11.885002136230469,"profits_derivative":-28.80767059326172,"trees":3.676767587661743},{"profits":10.66987419128418,"profits_derivative":-31.35731315612793,"trees":3.7171716690063477},{"profits":9.349775314331055,"profits_derivative":-34.00361633300781,"trees":3.757575750350952},{"profits":7.920779228210449,"profits_derivative":-36.7481689453125,"trees":3.7979798316955566},{"profits":6.37888240814209,"profits_derivative":-39.59254455566406,"trees":3.838383913040161},{"profits":4.720010757446289,"profits_derivative":-42.53834533691406,"trees":3.8787879943847656},{"profits":2.9400548934936523,"profits_derivative":-45.58710861206055,"trees":3.919191837310791},{"profits":1.0348033905029297,"profits_derivative":-48.7404670715332,"trees":3.9595959186553955},{"profits":-1.0,"profits_derivative":-52.0,"trees":4.0}]},"height":400,"layer":[{"encoding":{"x":{"field":"trees","type":"quantitative"},"y":{"field":"profits","type":"quantitative"}},"mark":{"interpolate":"basis","type":"line"}},{"encoding":{"color":{"value":"#ff0000"},"x":{"field":"trees","type":"quantitative"},"y":{"field":"profits_derivative","type":"quantitative"}},"mark":{"interpolate":"basis","type":"line"}}],"title":"Berry Profits","width":600}`

From the above graph, notice that the *derivative of the profit function* is `0`

at *the same time the profit function maxes out*