Semantic Image Segmentation by DeepLab3

ImageSegment.livemd

Shozo Fukuda

@shoz-f

tinyML_livebook

More notebooks

Semantic Image Segmentation by DeepLab3

0.Original work

Liang-Chieh Chen and Yukun Zhu “Semantic Image Segmentation with DeepLab in TensorFlow”

JoonBeom Park “TFLite Segmentation Python”

https://github.com/joonb14/TFLiteSegmentation

Thanks a lot!!!

1.Helper module

Create the module to assist with tasks such as downloading a model.

defmodule Helper do
  @model_file "lite-model_deeplabv3_1_metadata_2.tflite"

  @wearhouse "https://github.com/joonb14/TFLiteSegmentation/raw/main/#{@model_file}"
  @local "/data/#{@model_file}"

  def model() do
    @local
  end

  def get() do
    Req.get!(@wearhouse).body
    |> then(fn x -> File.write(@local, x) end)
  end

  def rm() do
    File.rm(@local)
  end

  def exists?() do
    File.exists?(@local)
  end
end

Get the tflite model from @wearhouse and store it in @local.

Helper.get()

Implementation for Elixir/Nerves using TflInterp

2.Defining the inference module: DeepLab3

Pre-processing:
Resize the input image to the size of @deeplab3_shape and create a Float32 binary sequence normalized to the range {-1.0, 1.0}.
Post-processing:

defmodule DeepLab3 do
  # use TflInterp, model: "path of tflite file"
  use TflInterp

  @deeplab3_shape {257, 257}

  def apply(jpeg) do
    # preprocess
    bin =
      CImg.from_binary(jpeg)
      |> CImg.resize(@deeplab3_shape)
      |> CImg.to_binary(range: {-1.0, 1.0})

    # prediction
    outputs =
      __MODULE__
      |> TflInterp.set_input_tensor(0, bin)
      |> TflInterp.invoke()
      |> TflInterp.get_output_tensor(0)
      |> Nx.from_binary({:f, 32})
      |> Nx.reshape({257, 257, :auto})

    # postprocess
    _result =
      outputs
      |> Nx.argmax(axis: 2)
      |> Nx.as_type({:u, 8})
      |> Nx.to_binary()
      |> CImg.from_binary(257, 257, 1, 1, " CImg.color_mapping(:lines)
  end
end

Launch DeepLab3.

DeepLab3.start_link(model: Helper.model())

Displays the properties of the DeepLab3 model.

TflInterp.info(DeepLab3)

3.Let’s try it

Picam.next_frame()
|> tap(&amp;Kino.render(Kino.Image.new(&amp;1, :jpeg)))  # render the original image
|> DeepLab3.apply()
|> CImg.resize({320, 240})
|> CImg.to_binary(:jpeg)
|> Kino.Image.new(:jpeg)

4.TIL ;-)

Date: Feb. 8, 2022 / Nerves-livebook rpi3

Total processing time is about 7.8 seconds, excluding camera shooting. Of that time, the DeepLab3 inference - TflInterp.invoke(DeepLab3) - takes about 280 micro seconds, and the post-processing Nx.argmax takes about 5.8 seconds.

□

License

Back