EvalViz tour

Mix.install([
  {:evalviz, path: Path.expand("..", __DIR__)},
  # kino on its own does not know how to draw a Vega-Lite spec: the
  # Kino.Render implementation for it ships in kino_vega_lite, and without
  # that Livebook prints the struct instead of the chart
  {:kino_vega_lite, "~> 0.1"}
])

Data

Three well separated blobs, plus a binary target for the classification plots.

key = Nx.Random.key(42)

{a, key} = Nx.Random.normal(key, 0.0, 0.6, shape: {30, 2})
{b, key} = Nx.Random.normal(key, 4.0, 0.6, shape: {30, 2})
{c, key} = Nx.Random.normal(key, 8.0, 0.6, shape: {24, 2})

x = Nx.concatenate([a, b, c])
Nx.shape(x)

Confusion matrix

y_true = Nx.tensor([0, 0, 1, 1, 2, 2, 0, 1, 2, 2, 1, 0])
y_pred = Nx.tensor([0, 1, 0, 2, 2, 2, 0, 1, 2, 1, 1, 0])

EvalViz.confusion_matrix(y_true, y_pred, num_classes: 3)

Normalized by true class, with names. Each row now sums to one, so the diagonal reads as per-class recall.

EvalViz.confusion_matrix(y_true, y_pred,
  num_classes: 3,
  class_names: ["cat", "dog", "bird"],
  normalize: :true_class
)

ROC, precision-recall and DET

{scores, key} = Nx.Random.uniform(key, shape: {300})
{noise, key} = Nx.Random.uniform(key, shape: {300})

labels = Nx.select(Nx.greater(scores, 0.5), 1, 0)
weak = Nx.add(Nx.multiply(scores, 0.4), Nx.multiply(noise, 0.6))

EvalViz.roc_curve(
  [
    {"Strong", labels, scores},
    {"Weak", labels, weak}
  ],
  title: "ROC"
)

EvalViz.precision_recall_curve(
  [
    {"Strong", labels, scores},
    {"Weak", labels, weak}
  ],
  title: "Precision-Recall"
)

EvalViz.det_curve(labels, weak, title: "DET")

Multiclass

Pass a score matrix with one column per class and you get one one-vs-rest curve each. These three blobs overlap on purpose, so the middle one is genuinely harder to separate than the two on the outside.

mc_key = Nx.Random.key(7)

{ma, mc_key} = Nx.Random.normal(mc_key, 0.0, 1.6, shape: {60, 2})
{mb, mc_key} = Nx.Random.normal(mc_key, 1.8, 1.6, shape: {60, 2})
{mc, _mc_key} = Nx.Random.normal(mc_key, 3.4, 1.6, shape: {60, 2})

mx = Nx.concatenate([ma, mb, mc])

my =
  Nx.concatenate([
    Nx.broadcast(0, {60}),
    Nx.broadcast(1, {60}),
    Nx.broadcast(2, {60})
  ])

model = Scholar.Linear.LogisticRegression.fit(mx, my, num_classes: 3)
probabilities = Scholar.Linear.LogisticRegression.predict_probability(model, mx)

EvalViz.roc_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  title: "One-vs-rest ROC"
)

middle scores well below the other two, which is what being boxed in on both sides looks like.

:average adds a curve over the classes. :micro pools every (sample, class) pair into one binary problem, so every sample counts the same; :macro averages the per-class curves, so every class does, however few samples it has.

EvalViz.roc_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  average: [:micro, :macro],
  title: "With both averages"
)

Six classes overlaid would be six curves you have to trace back to a legend. facet: true gives each its own panel instead, titled with its name and AUC:

EvalViz.roc_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  facet: true,
  width: 200,
  height: 180,
  title: "One panel per class"
)

Faceting pays for itself twice over on the other two. Overlaid, the precision-recall baseline has to be left out because the classes disagree on their share of positives. A panel holds one class, so each draws its own:

EvalViz.precision_recall_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  facet: true,
  width: 200,
  height: 180,
  title: "Each with its own baseline"
)

And the threshold curve gets colour back for the metric, so the dash pattern it was pushed onto is not needed at all:

EvalViz.threshold_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  facet: true,
  width: 210,
  height: 190,
  title: "Three metrics, one class each"
)

Calibration takes the same matrix, one curve per class:

EvalViz.calibration_curve(my, probabilities,
  class_names: ["left", "middle", "right"],
  bins: 5,
  title: "Per-class calibration"
)

Threshold

Every other classification plot says how good the ranking is. This one answers what follows: given that ranking, where do you cut?

EvalViz.threshold_curve(labels, weak, title: "Where to cut")

And here is why that cut works or does not: the scores the model gave, split by what the answer actually was.

EvalViz.score_distribution(labels, weak,
  class_names: ["negative", "positive"],
  threshold: 0.5,
  title: "The weak model"
)

The humps overlap, and everything in the overlap is a mistake some threshold has to make. The strong model has the same data with almost no overlap:

EvalViz.score_distribution(labels, scores,
  class_names: ["negative", "positive"],
  threshold: 0.5,
  title: "The strong model"
)

Each class sums to one rather than showing raw counts, so a rare class stays visible next to a common one. Pass normalize: :none for the counts.

Learning curve

Nothing is trained here: these are scores you would have measured yourself, one per fold, so the mean comes with a band one standard deviation wide.

EvalViz.learning_curve(
  [20, 50, 100, 200, 400, 800],
  [
    [1.00, 0.99, 1.00],
    [0.98, 0.97, 0.98],
    [0.95, 0.96, 0.94],
    [0.93, 0.92, 0.93],
    [0.91, 0.92, 0.91],
    [0.90, 0.90, 0.91]
  ],
  [
    [0.62, 0.58, 0.65],
    [0.72, 0.70, 0.75],
    [0.80, 0.78, 0.82],
    [0.85, 0.84, 0.86],
    [0.88, 0.87, 0.88],
    [0.89, 0.89, 0.89]
  ],
  title: "Learning curve"
)

Comparing models

One learning curve says whether more data would help. Two say which model is still getting better as the data grows, which is a different question and often the one that decides.

EvalViz.learning_curve(
  [20, 50, 100, 200, 400],
  [
    {"Linear", [[0.72, 0.70], [0.74, 0.73], [0.75, 0.74], [0.75, 0.75], [0.76, 0.75]],
     [[0.60, 0.58], [0.68, 0.66], [0.72, 0.71], [0.74, 0.73], [0.75, 0.74]]},
    {"Forest", [[1.00, 0.99], [0.99, 0.99], [0.98, 0.98], [0.97, 0.97], [0.96, 0.96]],
     [[0.55, 0.52], [0.68, 0.65], [0.78, 0.76], [0.84, 0.83], [0.88, 0.87]]}
  ],
  title: "Which one wants more data?"
)

Colour carries the model and the dash pattern carries training against validation, so both readings survive at once. Linear flattens by 100 examples and Forest is still climbing at 400.

The regression plots take a list too, shown further down.

The threshold curve is the one plot that takes no list of models. Colour and dash are already spent there on class and metric, and it exists to tune one model's cut-off rather than to compare models: for that, ROC and precision-recall already overlay as many as you like.

Model selection

The learning curve asks whether more data would help. These work with the data you already have, and they run against Scholar.ModelSelection directly.

{ms_x, ms_key} = Nx.Random.normal(Nx.Random.key(9), 0.0, 1.0, shape: {80, 4})

ms_y =
  Nx.add(
    Nx.add(Nx.multiply(ms_x[[.., 0]], 2.0), Nx.multiply(ms_x[[.., 1]], -1.0)),
    elem(Nx.Random.normal(ms_key, 0.0, 0.5, shape: {80}), 0)
  )

folding = fn t -> Scholar.ModelSelection.k_fold_split(t, 5) end

scoring = fn xs, ys ->
  {x_train, x_test} = xs
  {y_train, y_test} = ys
  model = Scholar.Linear.RidgeRegression.fit(x_train, y_train, alpha: 1.0)
  pred = Scholar.Linear.RidgeRegression.predict(model, x_test)

  [
    Scholar.Metrics.Regression.mean_square_error(y_test, pred),
    Scholar.Metrics.Regression.mean_absolute_error(y_test, pred)
  ]
end

Scholar.ModelSelection.cross_validate(ms_x, ms_y, folding, scoring)
|> EvalViz.fold_scores(metric_names: ["MSE", "MAE"], title: "Five folds")

One panel per metric, one point per fold, with the mean they scatter around. The folds are not joined: they are interchangeable, so a line would draw a trend across an order that means nothing. Each subtitle states the spread that a mean on its own would hide.

Now the same model over a grid of settings:

grid_scoring = fn xs, ys, opts ->
  {x_train, x_test} = xs
  {y_train, y_test} = ys
  model = Scholar.Linear.RidgeRegression.fit(x_train, y_train, opts)
  pred = Scholar.Linear.RidgeRegression.predict(model, x_test)

  [Scholar.Metrics.Regression.mean_square_error(y_test, pred)]
end

Scholar.ModelSelection.grid_search(ms_x, ms_y, folding, grid_scoring,
  alpha: [0.0, 0.5, 1.0, 5.0, 20.0],
  fit_intercept?: [true, false]
)
|> EvalViz.grid_search(best: :min, metric_name: "MSE", title: "Grid search")

MSE is best at its smallest, so best: :min both outlines the winning cell and reverses the colour ramp: darker stays better rather than asking you to invert it by eye.

And one knob on its own, swept over orders of magnitude:

EvalViz.validation_curve(
  [0.001, 0.01, 0.1, 1.0, 10.0, 100.0],
  [
    [0.99, 0.99],
    [0.98, 0.98],
    [0.96, 0.95],
    [0.90, 0.89],
    [0.71, 0.70],
    [0.40, 0.38]
  ],
  [
    [0.82, 0.80],
    [0.87, 0.85],
    [0.91, 0.89],
    [0.88, 0.86],
    [0.69, 0.67],
    [0.39, 0.37]
  ],
  param_name: "alpha",
  scale: :log,
  title: "Ridge regularisation"
)

The whole story in one picture: too little regularisation on the left, where training beats validation, too much on the right, where both collapse, and the best setting marked between them. A linear axis would crush every small value against the origin, which is why the sweep is drawn on a log scale.

Calibration

A model whose probabilities mean what they say tracks the diagonal.

{u, key} = Nx.Random.uniform(key, shape: {600})
{r, key} = Nx.Random.uniform(key, shape: {600})

calibrated_y = Nx.select(Nx.less(r, u), 1, 0)
overconfident = Nx.clip(Nx.add(Nx.multiply(Nx.subtract(u, 0.5), 1.8), 0.5), 0.001, 0.999)

EvalViz.calibration_curve(
  [
    {"Calibrated", calibrated_y, u},
    {"Overconfident", calibrated_y, overconfident}
  ],
  bins: 10,
  title: "Calibration"
)

Dendrogram

model = Scholar.Cluster.Hierarchical.fit(x, linkage: :ward)

EvalViz.dendrogram(model, color_threshold: 8.0, title: "Ward linkage")

Try moving color_threshold and watch the clusters split and merge. That is the height you would cut the tree at.

threshold = Kino.Input.range("Cut height", min: 0.5, max: 25, default: 8, step: 0.5)

EvalViz.dendrogram(model, color_threshold: Kino.Input.read(threshold))

Silhouette

kmeans = Scholar.Cluster.KMeans.fit(x, num_clusters: 3, key: Nx.Random.key(1))

EvalViz.silhouette(x, kmeans.labels, num_clusters: 3, title: "k = 3")

Ask for more clusters than the data has and the diagram says so: the mean drops and bars start going negative, meaning those points sit closer to a neighbour.

too_many = Scholar.Cluster.KMeans.fit(x, num_clusters: 6, key: Nx.Random.key(1))

EvalViz.silhouette(x, too_many.labels, num_clusters: 6, title: "k = 6")

Elbow

The silhouette judges one k at a time. This judges them together: hand it the models and it plots how much each extra cluster bought.

EvalViz.elbow(
  Enum.map(2..8, &Scholar.Cluster.KMeans.fit(x, num_clusters: &1, key: Nx.Random.key(1))),
  title: "Choosing k"
)

The rule marks the k furthest from the line joining the first and last points, measured after rescaling both axes to 0..1: k spans single digits while inertia spans thousands, so raw distance would only ever measure the inertia.

A straight run of scores has no corner at all, and then nothing is marked rather than a k being invented.

PCA scree

Six features built from only three real directions, so the elbow lands at three.

{base, key} = Nx.Random.normal(key, shape: {200, 3})

mixing =
  Nx.tensor([
    [1.0, 0.2, 0.0, 0.5, 0.1, 0.3],
    [0.0, 1.0, 0.3, 0.1, 0.7, 0.2],
    [0.2, 0.0, 1.0, 0.3, 0.2, 0.9]
  ])

{small_noise, key} = Nx.Random.normal(key, 0.0, 0.05, shape: {200, 6})
wide = Nx.add(Nx.dot(base, mixing), small_noise)

pca = Scholar.Decomposition.PCA.fit(wide, num_components: 6)
EvalViz.scree(pca, title: "Explained variance")

Projection

Three groups living in eight dimensions, which is more than anyone can look at. Reducing them to two is how they become visible at all.

{ga, key} = Nx.Random.normal(key, 0.0, 1.0, shape: {70, 8})
{gb, key} = Nx.Random.normal(key, 2.6, 1.0, shape: {70, 8})
{gc, key} = Nx.Random.normal(key, 5.2, 1.0, shape: {70, 8})

groups = Nx.concatenate([ga, gb, gc])

truth =
  Nx.concatenate([
    Nx.broadcast(0, {70}),
    Nx.broadcast(1, {70}),
    Nx.broadcast(2, {70})
  ])

group_pca = Scholar.Decomposition.PCA.fit(groups, num_components: 3)
projected = Scholar.Decomposition.PCA.transform(group_pca, groups)

EvalViz.projection(projected, truth,
  label_names: ["left", "middle", "right"],
  title: "Eight dimensions, seen in two"
)

Both axes share one range on purpose. They measure the same kind of thing, so letting them scale apart would stretch the cloud and invent structure.

Pass a list of labellings to draw the same embedding twice, which is how a clustering gets checked against the truth.

kmeans = Scholar.Cluster.KMeans.fit(groups, num_clusters: 3)

EvalViz.projection(projected, [
  {"True class", truth},
  {"KMeans", kmeans.labels}
])

Cluster ids are arbitrary, so the colours need not agree between the panels. What you are comparing is which points ended up together.

The same function takes any reduction. t-SNE returns the embedding as a plain tensor, MDS returns a struct carrying it, and both go straight in.

mds = Scholar.Manifold.MDS.fit(groups, key: key, num_components: 2)

EvalViz.projection(mds, truth,
  label_names: ["left", "middle", "right"],
  title: "MDS"
)

Biplot and loadings

The scree plot says how much each component carries. These two say what the components are made of.

A biplot puts the features back on the picture: each arrow is one feature, pointing the way it pushes the points.

EvalViz.biplot(group_pca, groups,
  labels: truth,
  feature_names: Enum.map(0..7, &"f#{&1}"),
  title: "Biplot"
)

Arrows and points have unrelated units, so the arrows are stretched to sit against the cloud. Their directions, and their lengths next to each other, are what carry meaning.

The same numbers as a heatmap, when there are too many features for arrows:

EvalViz.loadings(group_pca,
  feature_names: Enum.map(0..7, &"f#{&1}"),
  title: "What each component is made of"
)

The colour scale is centred on zero, since a loading's sign says which way the feature pushes.

Regression

{feature, key} = Nx.Random.uniform(key, 0.0, 10.0, shape: {150, 1})
{err, _key} = Nx.Random.normal(key, 0.0, 1.2, shape: {150})

target = Nx.add(Nx.add(Nx.multiply(Nx.squeeze(feature), 2.5), 4.0), err)

fit = Scholar.Linear.LinearRegression.fit(feature, target)
predicted = Scholar.Linear.LinearRegression.predict(fit, feature)

EvalViz.predicted_vs_actual(target, predicted, title: "Predicted vs actual")

EvalViz.residuals(target, predicted, title: "Residuals")

Fit a straight line to curved data and the residuals arc, which is the plot earning its keep: the scatter above looks fine, this one does not.

curved = Nx.add(Nx.pow(Nx.squeeze(feature), 2), err)
bad_fit = Scholar.Linear.LinearRegression.fit(feature, curved)
bad_pred = Scholar.Linear.LinearRegression.predict(bad_fit, feature)

EvalViz.residuals(curved, bad_pred, title: "Linear fit on quadratic data")

Both regression plots also take a list of models, which puts them on one set of axes and one reference line, so the comparison is not two pictures with different scales.

shrunk = Nx.add(Nx.multiply(predicted, 0.85), 2.0)

EvalViz.predicted_vs_actual(
  [{"Linear", target, predicted}, {"Shrunk", target, shrunk}],
  title: "Two models, one diagonal"
)

EvalViz.residuals(
  [{"Linear", target, predicted}, {"Shrunk", target, shrunk}],
  title: "Two models, one zero line"
)

The scatter above shows where a pattern is left. These two show the shape of what is left: a histogram, and the residuals against the quantiles a normal sample would land on.

EvalViz.residual_distribution(target, predicted, title: "Residual distribution")

EvalViz.qq_plot(Nx.subtract(predicted, target), title: "Normal Q-Q")

The noise here was drawn from a normal, so the points sit on the line. Run the same plot on the bad fit and they do not, because what is left over is the curve the model could not follow rather than noise.

EvalViz.qq_plot(Nx.subtract(bad_pred, curved), title: "Q-Q of the bad fit")

Coefficients and correlation

What the model learned, and what the features were doing to begin with.

wide_fit = Scholar.Linear.LinearRegression.fit(wide, Nx.dot(wide, Nx.tensor([3.0, -2.0, 0.5, 0.1, 0.0, 1.4])))

EvalViz.coefficients(wide_fit,
  feature_names: Enum.map(0..5, &"f#{&1}"),
  title: "Coefficients"
)

Ordered by magnitude, since what matters is how far a feature moves the prediction, not which way. They are only comparable across features when the features were scaled to begin with: on raw columns a large coefficient may say nothing more than that the column is measured in small units.

EvalViz.correlation(wide,
  feature_names: Enum.map(0..5, &"f#{&1}"),
  title: "Correlation"
)

The colour scale is pinned to -1..1 rather than fitted, so a matrix of weak correlations does not colour like a matrix of strong ones. These six columns were built from three directions, and the heatmap says so.

The whole picture in one call

Everything above, one plot at a time. Most of the time you want the screen:

EvalViz.report(labels, weak,
  class_names: ["negative", "positive"],
  title: "The weak model, held out"
)

The confusion matrix and the histogram read the same threshold, so this is not four unrelated views: the matrix counts exactly what the rule on the histogram splits. Move the threshold and both move together.

EvalViz.report(labels, weak, threshold: 0.7, title: "Cutting at 0.7 instead")

Regression gets its own set, from the same function:

EvalViz.report(target, predicted, kind: :regression, title: "Linear fit")

Composing

report/3 is a convenience over grid/2, which lays out whatever you hand it.

EvalViz.grid(
  [
    EvalViz.roc_curve(labels, scores, title: "ROC"),
    EvalViz.calibration_curve(labels, scores, bins: 8, title: "Calibration"),
    EvalViz.silhouette(groups, kmeans.labels, num_clusters: 3, title: "Clusters"),
    EvalViz.scree(pca, title: "Explained variance")
  ],
  title: "Whatever goes together"
)

Every function returns a VegaLite struct, so you can keep going with the VegaLite API.

EvalViz.roc_curve(labels, scores, title: "Held-out set")
|> VegaLite.config(axis: [grid: false], view: [stroke: nil])

VegaLite.config/2 sets a key Vega-Lite only allows on the outermost specification, so it goes on the grid rather than on the plots inside it.

EvalViz.grid([EvalViz.roc_curve(labels, scores), EvalViz.det_curve(labels, scores)])
|> VegaLite.config(axis: [grid: false])