Color Clustering and Dominant Colors

Mix.install([
  {:nx, "~> 0.7"},
  {:nx_image, "~> 0.1"},
  {:scholar, "~> 0.3"},
  {:exla, "~> 0.7"},
  {:kino, "~> 0.13"},
  {:image, "~> 0.51"}
],
 config: [
    nx: [
      default_backend: EXLA.Backend,
      default_defn_options: [compiler: EXLA]
    ]
  ]
)

previous_inspect = Inspect.Opts.default_inspect_fun()

Inspect.Opts.default_inspect_fun(fn term, opts ->
  previous_inspect.(term, %Inspect.Opts{opts | charlists: :as_lists})
end)

Choose an image

image_input = Kino.Input.image("An image to be uploaded")

Image.from_kino/1 knows how to take a kino input and open it. We’ll resize the image since the k-means algorithm is slow if there isn’t a GPU available for Nx (which underpins Scholar.Cluster.KMeans.fit/2). The number of colors in the image isn’t significantly reduced so the results will be good enough. Image.shape/1 returns the width, height and number of bands (channels) in the image.

{:ok, image} = 
  image_input
  |> Kino.Input.read()
  |> Image.from_kino!()
  |> Image.resize(0.5)

Image.shape(image)

Dominant Colors

Dominant colors are those colors that appear with the highest frequency in an image. The distribution of color in an image can be seen with an image histogram. We can use Image.Histogram.as_image/1 to see that distribution in an image.

image
|> Image.Histogram.as_image!()
|> Image.to_kino()

The histogram image shows the distribution of the three color primaries that are the consituents of an sRGB image and a white line that shows the luminance distribution.

Now lets get the 32 most dominant colors in the image. Dominant colors is simply those colors that appear most often in the image.

dominant_colors = Image.dominant_color!(image, top_n: 32)

OK, great, but thats not a very friendly way to visualize the colors. Lets do something about that.

dominant_colors
|> Enum.map(fn color -> 
  Image.new!(50, 50, color: color) 
  |> Image.to_kino()
end)
|> Kino.Layout.grid(columns: 10)

Color Sorting

Well now we can see the dominant colors, in descending frequency order. Can we see the colors in sorted order? Yes, thats possible. But what does sorting colors even mean? In Image.Color.sort/2 we use the approach documented here. Colors are converted to a weighted [hue, luminance, value] color space which is one way to perceptually sort colors.

sorted_dominant_colors = Image.Color.sort(dominant_colors)

sorted_dominant_colors  
|> Enum.map(fn color -> 
  Image.new!(50, 50, color: color) 
  |> Image.to_kino()
end)
|> Kino.Layout.grid(columns: 10)

Color Difference

Depending on your image, its quite likely the most dominant colors are probably quite similar. In many cases they may well look the same. Can we describe how close the colors are to each other?

Yes, we can! The approach to color difference is called ΔE*, commonly referred to as Delta E. In Image the function is Image.delta_e/2.

Lets take the colors in sorted order and see the distance (color difference) between adjacent colors.

defmodule ColorDiff do
  def difference([c1, c2 | rest]), do: [elem(Image.delta_e(c1, c2), 1) | difference([c2 | rest])]
  def difference([_c1]), do: []
end

ColorDiff.difference(sorted_dominant_colors)

Depending on the image, some of the differences are likely to be quite small. In fact it is quite possible the differences are below the just noticeable difference (JND) threshold. Any number below 2.3 means the two colors are probably indistinguishable to the human eye.

Color Clustering (K-Means)

While dominant colors are interesting, they don’t fully represent the range of colors in an image. In order to do that we need some means of describing a color palette. To do that we use the K Means clustering algorithm which is implemented in the scholar library. When scholar is configured, the Image.k_means/2 function is made available.

Since scholar depends on nx we know that performance is proportional to image size - and more particularly on whether a GPU is available and supported.

For this section its recommended the image size be under 1_000_000 pixels or even smaller in order to provide reasonable performance without a GPU.

# Reduce the size of the image to a maximum number of 100_000 pixels
max_pixels = 100_000
{width, height, _bands} = Image.shape(image)
pixels = width * height
small_image = if pixels <= max_pixels, do: image, else: Image.resize!(image, max_pixels / pixels)

Now let call the K-means function to return the clusters into which all the colors of the image are assigned. We can set the number of clusters with the :num_clusters option which defaults to 16.

k_means = Image.k_means!(small_image, num_clusters: 32)

As with our dominant colors example, lets take a look at the centroids as color swatches.

k_means
|> Image.Color.sort()
|> Enum.map(fn color -> 
  Image.new!(50, 50, color: color) 
  |> Image.to_kino()
end)
|> Kino.Layout.grid(columns: 10)

We can see that this set of colors covers the full range of colors in the image, not just the dominant colors. Let’s see how different these colors are from each other. We should expect a much wider color difference between adjacent colors.

k_means
|> Image.Color.sort()
|> ColorDiff.difference()

Color Reduction

You might be wondering at this point, what would the image look like if it only used the colors returned from Image.k_means/2. Well we can do that too by using the Image.reduce_colors/2 function. This function calls Image.k_means/2 under the hood, then replaces each pixel in the image with the cluster color that is closest to it.

In this example we’ll see what the base image looks like using 2, 4, 8, 16, 32 and 64 colors.

Enum.map([2, 4, 8, 16, 32, 64], fn colors ->
  small_image
  |> Image.reduce_colors!(colors: colors)
  |> Image.to_kino()
end)
|> Kino.Layout.grid(columns: 3)

Depending on the image, you may well find that the perceived difference of the last 2 or 3 images is quite small. That is why one of the techniques used by different image formats to reduce file sizes is to reduce the number of bits used to represent colors. They may not use K-means but they do color clustering to derive a reduced color palette.