Software MansionSM
Software Mansion•2y ago
TonyLikeSocks

WebRTC Endpoint + Mixing Multiple Tracks into a single mp4

I have a working app that allows a user to "talk" to an LLM. I'm using Membrane to help coordinate the audio. For QA purposes, we record the tracks (one for each endpoint). I'm trying to setup a bin that mixes the two tracks using the Membrane.LiveAudioMixer so I can have a single file. There's no errors thrown, but the resulting file is only 40 bytes, so I suspect I have something misconfigured.

Each time a pad is added, I try piping it into the LiveAudioMixer and then take that output, encode it and write it to the file.

def handle_setup(_context, state) do
    log_path = Application.fetch_env!(:smartvox, :log_path)
    File.mkdir(log_path)

    spec = [
      child(:mixer, %Membrane.LiveAudioMixer{
        stream_format: %Membrane.RawAudio{
          channels: 1,
          sample_rate: 16_000,
          sample_format: :s16le
        }
      })
      |> child(:encoder, %Membrane.Opus.Encoder{
          application: :voip,
          input_stream_format: %Membrane.RawAudio{
            channels: 1,
            sample_rate: 16_000,
            sample_format: :s16le
          }
      })
      |> child(:parser, Membrane.Opus.Parser)
      |> child({:muxer, state.room_id}, Membrane.MP4.Muxer.ISOM)
      |> child({:sink, state.room_id}, %Membrane.File.Sink{
        location: "#{log_path}/#{state.room_id}.mp4"
      })
    ]

    {[spec: spec], state}
  end

  def handle_pad_added(Pad.ref(:input, track_id) = pad, _ctx, state) do
    track = state.tracks[track_id]

    spec = [
      bin_input(pad)
      |> child({:track_receiver, track_id}, Smartvox.Endpoints.Conversation.TrackRecevier)
      |> child({:depayloader, track_id}, Track.get_depayloader(track))
      |> child({:decoder, track_id}, %Membrane.Opus.Decoder{
        sample_rate: 16_000,
      })
      |> via_in(:input)
      |> get_child(:mixer)
    ]

    {[spec: spec], state}
  end
Was this page helpful?