Image division

Mix.install([
  {:nx, "~> 0.9"},
  {:evision, "~> 0.2"},
  {:req, "~> 0.5"},
  {:kino, "~> 0.14"}
])

Modules

Install dependent modules in the setup cell. It’s good to see the dependencies made explicit. Jupyter does not make dependencies explicit.

This notebook installs the following modules.

• Nx: Perform matrix operations in Elixir
• evision: An OpenCV wrapper, Process images in Elixir
• Req: Download the image in this notebook
• Kino: Powerful UI/UX in notebooks

What to do in this notebook

Divide an image vertically or horizontally and then merge them together.

Get the image

Download the image of me, Ryo Wakabayashi.

img_path = "rwakabay.jpg"

Req.get!(
  "https://www.elixirconf.eu/assets/images/ryo-wakabayashi.jpg",
  output: File.stream!(img_path)
)

img_mat = Evision.imread(img_path)

When you load an image with evision, the image will appear in the notebook. The automatic display of results is much more convenient than with Jupyter

Then prepare for image processing.

# Get the filename, extension, and shape
file_ext = Path.extname(img_path)
file_basename = Path.basename(img_path, file_ext)
img_shape = img_mat.shape

{file_basename, file_ext, img_shape}

# Prepare a function to save a list of divided images
save_img_tensor_list = fn img_tensor_list, kind ->
  img_tensor_list
  |> Enum.with_index()
  |> Enum.map(fn {img_tensor, index} ->
    dst_filename = "#{file_basename}_#{kind}_#{index}#{file_ext}"

    img_mat = Evision.Mat.from_nx_2d(img_tensor)

    Evision.imwrite(dst_filename, img_mat)

    dst_filename
  end)
end

Divide horizontally

First, let’s divide the image horizontally

# chunk_size is number of pixels in width and height of the divided image
chunk_size = 60

img_mat
# Convert evision image data to Nx tensor
|> Evision.Mat.to_nx(Nx.BinaryBackend)
# Nx.to_batched divides the tensor into pieces of the chunk_size
|> Nx.to_batched(chunk_size)
# Evision OpenCV keeps colors in BGR order, so invert to RGB
|> Enum.map(&Nx.reverse(&1, axes: [2]))
# Display divided images
|> Enum.map(&Kino.Image.new(&1))
|> Kino.Layout.grid(columns: 1)
|> dbg()

I’m divided horizontally.

Kino.Layout.grid makes it easy display images side by side.

dbg shows the output of each pipele. I’* Original image

• Converted to a tensor
• Batched to a list of tensors
• Inverted
• Displayed

# Save divided images with the prepared funtion
img_mat
|> Evision.Mat.to_nx(Nx.BinaryBackend)
|> Nx.to_batched(chunk_size)
|> save_img_tensor_list.("h")

The file name of the saved image is now displayed.

Now let’s concatenate the divided images.

# Get a list of divided images
Stream.unfold(0, fn counter -> {counter, counter + 1} end)
|> Stream.map(&{&1, "#{file_basename}_h_#{&1}#{file_ext}"})
|> Stream.take_while(fn {_, f} -> File.exists?(f) end)
# Load each image
|> Enum.map(fn {index, filename} ->
  new_tensor =
    filename
    |> Evision.imread()
    |> Evision.Mat.to_nx(Nx.BinaryBackend)

  # Invert color for even numbers to show that they were divided
  case rem(index, 2) do
    0 ->
      Nx.reverse(new_tensor, axes: [2])

    _ ->
      new_tensor
  end
end)
# Concatenate the images
|> Nx.concatenate()
# Trim to image size
|> Nx.slice([0, 0, 0], Tuple.to_list(img_shape))
|> Kino.Image.new()

The images were concatenated and horizontal striped.

Divide vertically

Next, let’s divide the image vertically.

Lay the image to divide it vertically.

Bacause, Nx.to_batched can only split horizontally.

img_mat
|> Evision.Mat.to_nx(Nx.BinaryBackend)
# Swap the vertical and horizontal
|> Nx.transpose(axes: [1, 0, 2])
# Divide
|> Nx.to_batched(chunk_size)
# Swap the vertical and horizontal
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
# BGR to RGB
|> Enum.map(&Nx.reverse(&1, axes: [2]))
|> Enum.map(&Kino.Image.new(&1))
|> Kino.Layout.grid(columns: 10)
# If not transposed, the image can only be divided horizontally
|> dbg()

I’m divided vertically.

When Nx.transpose is turned off, I fall on its side and divided horizontally.

This is why Livebook and dbg work so well together for image processing.

# Save the divided images
img_mat
|> Evision.Mat.to_nx(Nx.BinaryBackend)
# Swap the vertical and horizontal
|> Nx.transpose(axes: [1, 0, 2])
# Divide
|> Nx.to_batched(chunk_size)
# Swap the vertical and horizontal
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
|> save_img_tensor_list.("v")

# Concatenate divided images
Stream.unfold(0, fn counter -> {counter, counter + 1} end)
|> Stream.map(&{&1, "#{file_basename}_v_#{&1}#{file_ext}"})
|> Stream.take_while(fn {_, f} -> File.exists?(f) end)
|> Enum.map(fn {index, filename} ->
  new_tensor =
    filename
    |> Evision.imread()
    |> Evision.Mat.to_nx(Nx.BinaryBackend)

  # Invert color for even numbers
  case rem(index, 2) do
    0 ->
      Nx.reverse(new_tensor, axes: [2])

    _ ->
      new_tensor
  end
end)
|> Nx.concatenate(axis: 1)
|> Nx.slice([0, 0, 0], Tuple.to_list(img_shape))
|> Kino.Image.new()

The images were concatenated and vertical striped.

Divide into tiles

Finally, divide the image into tiles.

To tile, divide both horizontally and vertically.

img_mat
|> Evision.Mat.to_nx(Nx.BinaryBackend)
# Divide horizontally
|> Nx.to_batched(chunk_size)
# Divide vertically
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
|> Enum.flat_map(&Nx.to_batched(&1, chunk_size))
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
# BGR to RGB
|> Enum.map(&Nx.reverse(&1, axes: [2]))
|> Enum.map(&Kino.Image.new(&1))
|> Kino.Layout.grid(columns: 10)
|> dbg()

I’m divided into tiles.

# Save divided images
img_mat
|> Evision.Mat.to_nx(Nx.BinaryBackend)
# Divide horizontally
|> Nx.to_batched(chunk_size)
# Divide vertically
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
|> Enum.flat_map(&Nx.to_batched(&1, chunk_size))
|> Enum.map(&Nx.transpose(&1, axes: [1, 0, 2]))
|> save_img_tensor_list.("t")

{width, _, _} = img_shape
h_size = div(width, chunk_size)

# Concatenate divided images
Stream.unfold(0, fn counter -> {counter, counter + 1} end)
|> Stream.map(&{&1, "#{file_basename}_t_#{&1}#{file_ext}"})
|> Stream.take_while(fn {_, f} -> File.exists?(f) end)
|> Enum.map(fn {t_index, filename} ->
  new_tensor =
    filename
    |> Evision.imread()
    |> Evision.Mat.to_nx(Nx.BinaryBackend)

  {new_tensor, t_index}
end)
|> Enum.chunk_every(h_size)
|> Enum.map(fn new_tensor_list ->
  new_tensor_list
  |> Enum.with_index()
  |> Enum.map(fn {{new_tensor, t_index}, v_index} ->
    # Invert color to check pattern
    cond do
      rem(v_index, 2) == rem(div(t_index, h_size), 2) ->
        Nx.reverse(new_tensor, axes: [2])

      true ->
        new_tensor
    end
  end)
  # Concatenate vertically
  |> Nx.concatenate(axis: 1)
end)
# Concatenate horizontally
|> Nx.concatenate()
# Trim to image size
|> Nx.slice([0, 0, 0], Tuple.to_list(img_shape))
|> Kino.Image.new()

I’m in a check pattern.