Generative Inpainting

File.cd!(__DIR__)
# for windows JP
System.shell("chcp 65001")
System.put_env("NNCOMPILED", "YES")

Mix.install([
  {:tfl_interp, path: ".."},
  {:cimg, "~> 0.1.19"},
  {:kino, "~> 0.7.0"}
])

0.Original work

“Generative Image Inpainting with Contextual Attention”

• https://arxiv.org/abs/1801.07892

“Free-Form Image Inpainting with Gated Convolution”

• https://arxiv.org/abs/1806.03589

GitHub: Generative Image Inpainting

• https://github.com/JiahuiYu/generative_inpainting

The tflite model deepfillv2.tflite is converted from their pretraind model.

Thanks a lot!!!

Implementation with TflInterp in Elixir

1.Defining the inference module: DeepFillV2

• Model

  deepfillv2.tflite: get from “https://github.com/shoz-f/tfl_interp/releases/download/0.0.1/deepfillv2.tflite” if not existed.

• Pre-processing

  Combine the original and mask images into a single image, then resize it to {@width, @height} and normalize it to a range of {0.0, 255.0} for further manipulation.

• Post-processing

  The inpainted image is outputted directly by this model.

defmodule DeepFillV2 do
  @width 680
  @height 512

  alias TflInterp, as: NNInterp

  use NNInterp,
    model: "./model/deepfillv2.tflite",
    url: "https://github.com/shoz-f/tfl_interp/releases/download/0.0.1/deepfillv2.tflite",
    inputs: [f32: {1, @height, 2 * @width, 3}],
    outputs: [u8: {1, @height, @width, 3}]

  def apply(img, mask) do
    # preprocess
    input0 =
      CImg.builder(img)
      |> CImg.append(mask, :x)
      |> CImg.resize({2 * @width, @height})
      |> CImg.to_binary(range: {0.0, 255.0})

    # prediction
    {w, h, _, _} = CImg.shape(img)

    session()
    |> NNInterp.set_input_tensor(0, input0)
    |> NNInterp.invoke()
    |> NNInterp.get_output_tensor(0)
    |> CImg.from_binary(@width, @height, 1, 3, [{:dtype, " CImg.resize({w, h})
  end
end

Launch DeepFillV2.

# TflInterp.stop(DeepFillV2)
DeepFillV2.start_link([])

Display the properties of the DeepFillV2 model.

TflInterp.info(DeepFillV2)

2.Let’s try it

Load a photo and apply DeepFillV2 to it.

origin = CImg.load("sample_raw.jpg")
mask = CImg.load("sample_mask.jpg")

result = DeepFillV2.apply(origin, mask)

Enum.map([origin, mask, result], &CImg.display_kino(&1, :jpeg))
|> Kino.Layout.grid(columns: 2)

□