SMM
Software Mansion / Membranenoozo

Pipeline for muxing 2 msr files (audio and video) into a single flv file

I have the following pipeline which takes 2 msr files (recorded to disk using the RecordingEntrypoint from rtc_engine) and need to create a single video+audio file from it (trying flv at the moment but not tied to a specific type, just want something that popular tools can read and manipulate). My problem is that the end FLV file only plays audio. Here's the pipeline: 
spec = [
      # Part of pipeline which saves to FLV file
      # child(:muxer, Membrane.MP4.Muxer.ISOM)
      child(:muxer, Membrane.FLV.Muxer)
      |> child(:sink, %Membrane.File.Sink{location: output_file}),

      # Part of pipeline which reads from file and prepare video for muxing
      child(:source, %Membrane.File.Source{location: video_track_file})
      |> child(:deserializer_video, Membrane.Stream.Deserializer)
      |> child(:rtp_video, %Membrane.RTP.DepayloaderBin{
        depayloader: Membrane.RTP.H264.Depayloader,
        clock_rate: 90_000
      })
      |> child(:parser_video, %Membrane.H264.Parser{
        generate_best_effort_timestamps: %{framerate: {0, 1}},
        output_stream_structure: :avc1
      })
      |> via_in(Pad.ref(:video, 0))
      |> get_child(:muxer),
      # Part of pipeline which reads from file and prepare audio for muxing
      child(:source_audio, %Membrane.File.Source{location: audio_track_file})
      |> child(:deserializer_audio, Membrane.Stream.Deserializer)
      |> child(:rtp_audio, %Membrane.RTP.DepayloaderBin{
        depayloader: Membrane.RTP.Opus.Depayloader,
        clock_rate: 48_000
      })
      |> child(:opus_decoder, Membrane.Opus.Decoder)
      |> child(:aac_encoder, Membrane.AAC.FDK.Encoder)
      |> child(:aac_parser, %Membrane.AAC.Parser{
        out_encapsulation: :none,
        output_config: :audio_specific_config
      })
      |> via_in(Pad.ref(:audio, 0))
      |> get_child(:muxer)
    ]
spec = [
      # Part of pipeline which saves to FLV file
      # child(:muxer, Membrane.MP4.Muxer.ISOM)
      child(:muxer, Membrane.FLV.Muxer)
      |> child(:sink, %Membrane.File.Sink{location: output_file}),

      # Part of pipeline which reads from file and prepare video for muxing
      child(:source, %Membrane.File.Source{location: video_track_file})
      |> child(:deserializer_video, Membrane.Stream.Deserializer)
      |> child(:rtp_video, %Membrane.RTP.DepayloaderBin{
        depayloader: Membrane.RTP.H264.Depayloader,
        clock_rate: 90_000
      })
      |> child(:parser_video, %Membrane.H264.Parser{
        generate_best_effort_timestamps: %{framerate: {0, 1}},
        output_stream_structure: :avc1
      })
      |> via_in(Pad.ref(:video, 0))
      |> get_child(:muxer),
      # Part of pipeline which reads from file and prepare audio for muxing
      child(:source_audio, %Membrane.File.Source{location: audio_track_file})
      |> child(:deserializer_audio, Membrane.Stream.Deserializer)
      |> child(:rtp_audio, %Membrane.RTP.DepayloaderBin{
        depayloader: Membrane.RTP.Opus.Depayloader,
        clock_rate: 48_000
      })
      |> child(:opus_decoder, Membrane.Opus.Decoder)
      |> child(:aac_encoder, Membrane.AAC.FDK.Encoder)
      |> child(:aac_parser, %Membrane.AAC.Parser{
        out_encapsulation: :none,
        output_config: :audio_specific_config
      })
      |> via_in(Pad.ref(:audio, 0))
      |> get_child(:muxer)
    ]
What am i doing wrong? Help appreciated!
N
noozo•67d ago
Thanks to @Radosław the problem seems to be difference between the enconding on the webrtc part and the encoding i'm doing for muxing. Follow up question is: How can i modify the encoding on the webrtc part to be h264 instead of VP8 (which is how the recording plugin is storing those msr files)?
R
Radosław•67d ago
If you use membrane_videoroom here you can change it: https://github.com/membraneframework-labs/membrane_videoroom/blob/d3b0c49968d625afcee9777b043793dd7d59e418/lib/videoroom/room.ex#L239-L241
GitHub
membrane_videoroom/lib/videoroom/room.ex at d3b0c49968d625afcee9777...
Contribute to membraneframework-labs/membrane_videoroom development by creating an account on GitHub.
N
noozo•67d ago
ty @Radosław you've been a life saver
Want results from more Discord servers?
Add your server
More Posts
MP3 output is audible, but test not passHi everyone. I made small changes in `membrane_mp3_lame_plugin` to support other input config (the oGrab keyframe image data from h264 stream?We are looking at some h264 coming from an RTSP stream. Membrane is doing a fine job with the HLS deUnity client?Thanks a lot to the Membrane team. The number of examples and availability of code has been incredibWebRTC stream not workingI've hit an issue trying to get an MPEGTS stream displaying in the browser via WebRTC. All seems toError when compiling free4chatThis is probably pretty basic but I'm at square one. I get an error when I'm compiling free4chat, a Screen ShareIs there any reason why jellyfish_videoroom when is running localy share screen is not working ? I sTesting Membrane ElementI'm trying to setup a simple test of a membrane element, but I'm stumped a bit on how to assert thatClustering and scale out behaviourContext: I've a membrane application running in an elixir cluster of 1. It receives a RTSP stream (U