Audio

Overview

The phone has three audio parts:

• the loudspeaker, a swappable module with its own amplifier,
• the earpiece, the small speaker you hold to your ear during a call,
• the microphone.

The earpiece and the microphone go through the WCD9335 audio codec. All three are driven through the phone's audio DSP. Sound is played on plughw:0,0 and recorded from hw:0,1; ALSA mixer controls decide whether playback goes to the loudspeaker or the earpiece.

Switch between them while nothing is playing: changing the route in the middle of a playback stops it with an input/output error. To move a sound that is playing, stop it, switch, and carry on from where it was; Switch while playing shows how.

The loudspeaker module is what differs between the Fairphone 3 and the 3+, and since it can be swapped, a phone sold as one may carry the other's:

Only the TAS2557 has a Speaker Switch control. This cell reads the sound card name to know whether to set it. ExAudio sets the mixer controls; set applies a list of them and shows the result of each one:

tas2557? = File.read!("/proc/asound/cards") =~ "Fairphone 3+"

speaker_switch = fn value -> if tas2557?, do: [{"Speaker Switch", value}], else: [] end

set = fn controls ->
  for {name, value} <- controls, do: {name, ExAudio.cset(name, value)}
end

IO.puts("Loudspeaker amplifier: #{if tas2557?, do: "TAS2557", else: "AW8898"}")

Loudspeaker

Route audio to the loudspeaker

The firmware enables this route at boot. Run it again to come back from the earpiece.

speaker =
  [
    {"Earpiece Switch", :off},
    {"SLIMBUS_0_RX Audio Mixer MultiMedia1", :off},
    {"QUIN_MI2S_RX Audio Mixer MultiMedia1", :on}
  ] ++ speaker_switch.(:on)

set.(speaker)

Play a tone

Generate one second of a 440 Hz tone, save it as a WAV file and play it with aplay.

rate = 48_000
volume = 0.1

tone = for n <- 0..(rate - 1), do: volume * :math.sin(2 * :math.pi() * 440 * n / rate)

NervesAI.Audio.WAV.write("/tmp/tone.wav", tone, rate: rate)
System.cmd("aplay", ["-D", "plughw:0,0", "/tmp/tone.wav"], stderr_to_stdout: true)

Change volume (0.0 to 1.0) and run it again.

A short melody

notes = [262, 294, 330, 349, 392, 440, 494, 523]

melody =
  for freq <- notes, n <- 0..(div(rate, 4) - 1) do
    volume * :math.sin(2 * :math.pi() * freq * n / rate)
  end

NervesAI.Audio.WAV.write("/tmp/melody.wav", melody, rate: rate)
System.cmd("aplay", ["-D", "plughw:0,0", "/tmp/melody.wav"], stderr_to_stdout: true)

Earpiece

Route audio to the earpiece

This turns the loudspeaker off and sends the same playback device to the earpiece instead.

earpiece =
  speaker_switch.(:off) ++
    [
      {"QUIN_MI2S_RX Audio Mixer MultiMedia1", :off},
      {"SLIMBUS_0_RX Audio Mixer MultiMedia1", :on},
      {"SLIM RX0 MUX", "AIF_MIX1_PB"},
      {"RX INT0_1 MIX1 INP0", "RX0"},
      {"RX INT0 DEM MUX", "CLSH_DSM_OUT"},
      {"RX0 Digital Volume", 84},
      {"EAR PA Volume", 4},
      {"Earpiece Switch", :on}
    ]

set.(earpiece)

Listen

The earpiece is quiet: hold the top of the phone to your ear, then run this to play the melody again.

System.cmd("aplay", ["-D", "plughw:0,0", "/tmp/melody.wav"], stderr_to_stdout: true)

Run the loudspeaker route above to switch back.

Switch while playing

A call app moves the sound between the earpiece and the loudspeaker while it plays. The route can only change while nothing plays, so this player feeds the sound to aplay a tenth of a second at a time. Switching stops aplay, waits until the sound card has closed the stream, applies the new route and starts a new aplay with the rest of the sound. There is a short gap at the switch.

defmodule SwitchablePlayer do
  @chunk_ms 100
  @prefill 3

  # Plays mono samples (floats between -1.0 and 1.0) at `rate` and
  # returns the player.
  def start(samples, rate) do
    pcm = for sample <- samples, into: <<>>, do: <<round(sample * 32_767)::little-signed-16>>
    spawn(fn -> play(pcm, rate) end)
  end

  # Moves the sound to another route: `route` is called while nothing plays.
  def switch(player, route), do: send(player, {:switch, route})

  defp play(pcm, rate) do
    port = open(rate)
    pcm = feed(port, pcm, rate, @prefill)
    loop(port, pcm, rate, System.monotonic_time(:millisecond), 0)
  end

  defp loop(port, <<>>, _rate, _t0, _n), do: send(port, {self(), :close})

  defp loop(port, pcm, rate, t0, n) do
    receive do
      {:switch, route} ->
        stop(port)
        route.()
        play(pcm, rate)
    after
      max(t0 + n * @chunk_ms - System.monotonic_time(:millisecond), 0) ->
        loop(port, feed(port, pcm, rate, 1), rate, t0, n + 1)
    end
  end

  defp feed(port, pcm, rate, chunks) do
    size = min(byte_size(pcm), div(rate * 2 * @chunk_ms, 1000) * chunks)
    <<chunk::binary-size(size), rest::binary>> = pcm
    Port.command(port, chunk)
    rest
  end

  defp open(rate) do
    aplay = System.find_executable("aplay")
    args = ~w(-q -D plughw:0,0 -t raw -f S16_LE -c 1 -r #{rate} -)
    Port.open({:spawn_executable, aplay}, [:binary, :exit_status, args: args])
  end

  # SIGKILL: on SIGTERM, an aplay waiting for more sound keeps waiting.
  defp stop(port) do
    {:os_pid, pid} = Port.info(port, :os_pid)
    System.cmd("kill", ["-KILL", Integer.to_string(pid)])

    receive do
      {^port, {:exit_status, _}} -> :ok
    after
      2_000 -> :ok
    end

    wait_closed(30)
  end

  # The card shuts the old route down after the stream closes, and the
  # device is busy until then.
  defp wait_closed(0), do: :timeout

  defp wait_closed(tries) do
    case File.read("/proc/asound/card0/pcm0p/sub0/status") do
      {:ok, "closed" <> _} -> :ok
      _ -> Process.sleep(100) && wait_closed(tries - 1)
    end
  end
end

Play the melody four times on the loudspeaker:

set.(speaker)
player = SwitchablePlayer.start(List.flatten(List.duplicate(melody, 4)), rate)

While it plays, move it to the earpiece:

SwitchablePlayer.switch(player, fn -> set.(earpiece) end)

And back to the loudspeaker:

SwitchablePlayer.switch(player, fn -> set.(speaker) end)

Microphone

Route the microphone

This connects the phone's main microphone to the recording device.

microphone = [
  {"MultiMedia2 Mixer SLIMBUS_0_TX", :on},
  {"SLIM TX7 MUX", "DEC7"},
  {"ADC MUX7", "DMIC"},
  {"DMIC MUX7", "DMIC1"},
  {"AIF1_CAP Mixer SLIM TX7", :on}
]

set.(microphone)

Which microphone?

The phone has two microphones, both digital microphones on the codec, and they sit at opposite ends:

It is the same on the Fairphone 3 and the 3+. The route above selects DMIC1. The codec's other inputs (DMIC0, DMIC3 to DMIC5 and the analog ones) have nothing connected and record silence or noise.

DMIC MUX7 picks the microphone. This shows which one is selected:

dmics = ~w(ZERO DMIC0 DMIC1 DMIC2 DMIC3 DMIC4 DMIC5)
{:ok, index} = ExAudio.cget("DMIC MUX7")
IO.puts("Recording from #{Enum.at(dmics, String.to_integer(index))}")

And this switches. Pick "DMIC2", run the recording below, then compare it with one made on "DMIC1": holding the top or the bottom of the phone close to your mouth makes the difference obvious.

use_microphone = fn mic -> set.([{"DMIC MUX7", mic}]) end

use_microphone.("DMIC2")

Record

Records 5 seconds. Talk, clap or whistle while it runs.

arecord = fn seconds, path ->
  System.cmd("arecord", ~w(-D hw:0,1 -f S16_LE -r 48000 -c 1 -d #{seconds} #{path}),
    stderr_to_stdout: true
  )
end

# The first capture after boot returns silence; a short one wakes the path up.
arecord.(1, "/tmp/warmup.wav")
IO.puts("Recording for 5 seconds...")
arecord.(5, "/tmp/me.wav")

How loud was it?

The recording is split into tenths of a second, and each bar shows the loudest sample in that slice.

{:ok, %{rate: rate, pcm_f32: pcm}} = NervesAI.Audio.WAV.read("/tmp/me.wav")

for(<<sample::float-32-little <- pcm>>, do: abs(sample))
|> Enum.chunk_every(div(rate, 10))
|> Enum.map(fn slice ->
  peak = Enum.max(slice)
  String.duplicate("█", round(peak * 50)) <> " #{Float.round(peak, 2)}"
end)
|> Enum.join("\n")
|> IO.puts()

A silent room stays near 0; speaking close to the phone reaches 0.3 or more.

Play it back

Plays the recording on the loudspeaker:

set.(speaker)
System.cmd("aplay", ["-D", "plughw:0,0", "/tmp/me.wav"], stderr_to_stdout: true)

Or in the earpiece:

set.(earpiece)
System.cmd("aplay", ["-D", "plughw:0,0", "/tmp/me.wav"], stderr_to_stdout: true)

Play your own file

aplay plays WAV files. Copy one to the phone (for example with sftp nerves.local) and play it:

System.cmd("aplay", ["-D", "plughw:0,0", "/data/my_sound.wav"], stderr_to_stdout: true)