Hints

Documentation & Membrane Guide

Link to guide (and paragraph about available callbacks).

Link to Membrane documentation.

Page in the documentation about actions available in Elements and page about actions available in Pipelines.

Click, to unroll the hint to the specific exercise

Notice, that Membrane.FFmpeg.SWScale.Converter consumes and returns raw video frames.

Function get_child/1 might be helpful.

You have to implement only handle_buffer/4 callback in the ColorReverter module.

@impl true
def handle_buffer(:input, buffer, _ctx, state) do
  payload = 
    buffer.payload
    # |> some processing
  
  buffer = %{buffer | payload: payload}
  {[buffer: {:output, buffer}], state}
end

def preprocess(buffer_payload, input_stream_format) do
  buffer_payload
  |> Nx.from_binary(:u8, backend: EXLA.Backend)
  |> Nx.as_type(:f32)
  |> Nx.reshape({1, input_stream_format.height, input_stream_format.width, 3})
end

def preprocess(buffer, input_stream_format) do
  buffer.payload
  |> Nx.from_binary(:u8, backend: EXLA.Backend)
  |> Nx.as_type(:f32)
  |> Nx.reshape({1, input_stream_format.height, input_stream_format.width, 3})
  |> Nx.transpose(axes: [0, 3, 1, 2])
end

def postprocess(tensor, input_stream_format) do
  tensor
  |> Nx.backend_transfer(EXLA.Backend)
  |> Nx.reshape({3, input_stream_format.height, input_stream_format.width})
  |> Nx.transpose(axes: [1, 2, 0])
  |> clamp()
  |> Nx.round()
  |> Nx.as_type(:u8)
  |> Nx.to_binary()
end

defp clamp(tensor) do
  tensor
  |> Nx.max(0)
  |> Nx.min(255)
end

You can get use of :tick action and handle_tick/3 callback or check current time in handle_buffer/4.