Nx による画像処理

livebooks/11_Image.livemd

Ryo Wakabayashi

@RyoWakabayashi

elixir-learning

More notebooks

Nx による画像処理

Mix.install(
  [
    {:nx, "~> 0.6"},
    {:kino, "~> 0.12"},
    {:req, "~> 0.3"},
    {:pixels, "~> 0.3"}
  ],
  system_env: [
    {"CROSSCOMPILE", "false"}
  ]
)

Resolving Hex dependencies...
Resolution completed in 0.508s
New:
  castore 1.0.3
  complex 0.5.0
  elixir_make 0.6.3
  finch 0.16.0
  hpax 0.1.2
  jason 1.4.1
  kino 0.10.0
  mime 2.0.5
  mint 1.5.1
  nimble_options 1.0.2
  nimble_pool 1.0.0
  nx 0.6.0
  pixels 0.3.0
  req 0.3.11
  table 0.1.2
  telemetry 1.2.1
* Getting nx (Hex package)
* Getting kino (Hex package)
* Getting req (Hex package)
* Getting pixels (Hex package)
* Getting elixir_make (Hex package)
* Getting finch (Hex package)
* Getting jason (Hex package)
* Getting mime (Hex package)
* Getting castore (Hex package)
* Getting mint (Hex package)
* Getting nimble_options (Hex package)
* Getting nimble_pool (Hex package)
* Getting telemetry (Hex package)
* Getting hpax (Hex package)
* Getting table (Hex package)
* Getting complex (Hex package)
==> table
Compiling 5 files (.ex)
Generated table app
==> mime
Compiling 1 file (.ex)
Generated mime app
==> nimble_options
Compiling 3 files (.ex)
Generated nimble_options app
===> Analyzing applications...
===> Compiling telemetry
==> jason
Compiling 10 files (.ex)
Generated jason app
==> hpax
Compiling 4 files (.ex)
Generated hpax app
==> complex
Compiling 2 files (.ex)
Generated complex app
==> nx
Compiling 32 files (.ex)
Generated nx app
==> kino
Compiling 41 files (.ex)
Generated kino app
==> nimble_pool
Compiling 2 files (.ex)
Generated nimble_pool app
==> elixir_make
Compiling 1 file (.ex)
Generated elixir_make app
==> pixels
cc -c -g -O3  -I"/Users/rwakabay/.asdf/installs/erlang/26.0.2/erts-14.0.2/include" -fPIC -o c_src/pixels_jpeg.o c_src/pixels_jpeg.c
c_src/pixels_jpeg.c:37:11: warning: initializing 'char *' with an expression of type 'unsigned char *' converts between pointers to integer types where one is of the unique plain 'char' type and the other is not [-Wpointer-sign]
    char *image_data = ujGetImage(uj, NULL);
          ^            ~~~~~~~~~~~~~~~~~~~~
1 warning generated.
cc -c -g -O3  -I"/Users/rwakabay/.asdf/installs/erlang/26.0.2/erts-14.0.2/include" -fPIC -o c_src/pixels_nif.o c_src/pixels_nif.c
cc -c -g -O3  -I"/Users/rwakabay/.asdf/installs/erlang/26.0.2/erts-14.0.2/include" -fPIC -o c_src/pixels_png.o c_src/pixels_png.c
cc -c -g -O3  -I"/Users/rwakabay/.asdf/installs/erlang/26.0.2/erts-14.0.2/include" -fPIC -o c_src/ext/lodepng.o c_src/ext/lodepng.c
cc -c -g -O3  -I"/Users/rwakabay/.asdf/installs/erlang/26.0.2/erts-14.0.2/include" -fPIC -o c_src/ext/ujpeg.o c_src/ext/ujpeg.c
cc c_src/pixels_jpeg.o c_src/pixels_nif.o c_src/pixels_png.o c_src/ext/lodepng.o c_src/ext/ujpeg.o -dynamiclib -undefined dynamic_lookup -shared -o /Users/rwakabay/Library/Caches/mix/installs/elixir-1.15.2-erts-14.0.2/4b392808bf2548e6ef345b9dc8156cfd/_build/dev/lib/pixels/priv/pixels_nif.so
ld: warning: search path '/usr/local/lib' not found
Compiling 3 files (.ex)
warning: extra parentheses on a bitstring specifier "binary()" have been deprecated. Please remove the parentheses: "binary"
  lib/pixels/identify.ex:10: Pixels.Identify.identify/1

Generated pixels app
==> castore
Compiling 1 file (.ex)
Generated castore app
==> mint
Compiling 1 file (.erl)
Compiling 19 files (.ex)
Generated mint app
==> finch
Compiling 13 files (.ex)
warning: Logger.warn/1 is deprecated. Use Logger.warning/2 instead
  lib/finch/http2/pool.ex:362: Finch.HTTP2.Pool.connected/3

warning: Logger.warn/1 is deprecated. Use Logger.warning/2 instead
  lib/finch/http2/pool.ex:460: Finch.HTTP2.Pool.connected_read_only/3

Generated finch app
==> req
Compiling 5 files (.ex)
Generated req app

:ok

テンソル

単純な二次元リスト

inputs = [
  [1, 2],
  [3, 4]
]

[[1, 2], [3, 4]]

テンソルに変換する

Nx.tensor(inputs)

#Nx.Tensor<
  s64[2][2]
  [
    [1, 2],
    [3, 4]
  ]
>

テンソルに対して行列演算ができる

[
  [1, 2],
  [3, 4]
]
|> Nx.tensor()
|> Nx.divide(3)

#Nx.Tensor<
  f32[2][2]
  [
    [0.3333333432674408, 0.6666666865348816],
    [1.0, 1.3333333730697632]
  ]
>

テンソルをヒートマップとして表示する

[
  [9, 8, 7, 6, 5],
  [8, 7, 6, 5, 4],
  [7, 6, 5, 4, 3],
  [6, 5, 4, 3, 2],
  [5, 4, 3, 2, 1]
]
|> Nx.tensor()
|> Nx.to_heatmap()

#Nx.Heatmap<
  s64[5][5]
  
  　　　　　
  　　　　　
  　　　　　
  　　　　　
  　　　　　
>

画像処理

画像をダウンロードする

img_path = "rwakabay.png"

"https://www.elixirconf.eu/assets/images/ryo-wakabayashi.png"
|> Req.get!(output: img_path)

%Req.Response{
  status: 200,
  headers: [
    {"connection", "keep-alive"},
    {"content-length", "519082"},
    {"server", "GitHub.com"},
    {"content-type", "image/png"},
    {"last-modified", "Wed, 26 Jul 2023 15:16:09 GMT"},
    {"access-control-allow-origin", "*"},
    {"etag", "\"64c138b9-7ebaa\""},
    {"expires", "Mon, 21 Aug 2023 03:25:17 GMT"},
    {"cache-control", "max-age=600"},
    {"x-proxy-cache", "MISS"},
    {"x-github-request-id", "A1EE:6ED3:4EBFC5:53941A:64E2D6C5"},
    {"accept-ranges", "bytes"},
    {"date", "Mon, 21 Aug 2023 03:15:18 GMT"},
    {"via", "1.1 varnish"},
    {"age", "0"},
    {"x-served-by", "cache-itm18834-ITM"},
    {"x-cache", "MISS"},
    {"x-cache-hits", "0"},
    {"x-timer", "S1692587718.813184,VS0,VE192"},
    {"vary", "Accept-Encoding"},
    {"x-fastly-request-id", "64a006c1ba7e3d9759c7ec66253e761ff1b355a9"}
  ],
  body: "",
  private: %{}
}

画像を表示する

img_path
|> File.read!()
|> Kino.Image.new(:jpeg)

画像のピクセル情報

Pixels.read_file(img_path)

{:ok,
 %Pixels{
   width: 600,
   height: 600,
   data: <<202, 201, 196, 255, 205, 204, 199, 255, 203, 202, 197, 255, 202, 201, 196, 255, 204, 203,
     198, 255, 202, 201, 196, 255, 202, 201, 196, 255, 204, 203, 198, 255, 204, 203, 198, 255, 205,
     204, 199, 255, 205, 204, 199, 255, 205, ...>>
 }}

画像をテンソル化する

rgba =
  img_path
  |> Pixels.read_file()
  |> elem(1)
  |> Map.get(:data)
  |> Nx.from_binary(:u8)
  |> Nx.reshape({600, 600, 4})

#Nx.Tensor<
  u8[600][600][4]
  [
    [
      [202, 201, 196, 255],
      [205, 204, 199, 255],
      [203, 202, 197, 255],
      [202, 201, 196, 255],
      [204, 203, 198, 255],
      [202, 201, 196, 255],
      [202, 201, 196, 255],
      [204, 203, 198, 255],
      [204, 203, 198, 255],
      [205, 204, 199, 255],
      [205, 204, 199, 255],
      [205, 204, 199, 255],
      [204, 203, ...],
      ...
    ],
    ...
  ]
>

# 画像をネガポジ反転する
rgb =
  rgba
  |> Nx.slice_along_axis(0, 3, axis: -1)
  |> then(&amp;Nx.subtract(255, &amp;1))

a = Nx.slice_along_axis(rgba, 4, 1, axis: -1)

new_rgba = Nx.concatenate([rgb, a], axis: -1)

#Nx.Tensor<
  u8[600][600][4]
  [
    [
      [53, 54, 59, 255],
      [50, 51, 56, 255],
      [52, 53, 58, 255],
      [53, 54, 59, 255],
      [51, 52, 57, 255],
      [53, 54, 59, 255],
      [53, 54, 59, 255],
      [51, 52, 57, 255],
      [51, 52, 57, 255],
      [50, 51, 56, 255],
      [50, 51, 56, 255],
      [50, 51, 56, 255],
      [51, 52, ...],
      ...
    ],
    ...
  ]
>

Kino.Image.new(new_rgba)

Back