SMM
Software Mansion / Membranespscream

toilet capacity of outbound_rtx_controller

Hi, I'm getting the following error on SessionBin: 
[error] <0.1282.0>/:sip_rtp/{:outbound_rtx_controller, 1929338881} Toilet overflow.

Atomic demand reached the size of -201, which means that there are 201
...
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.

[error] <0.1282.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}
[error] <0.1282.0>/:sip_rtp/{:outbound_rtx_controller, 1929338881} Toilet overflow.

Atomic demand reached the size of -201, which means that there are 201
...
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.

[error] <0.1282.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}
sip_rtp is SessionBin with following inputs for video: 
        child(:sip_video, %UDP.Endpoint{
          local_port_no: src_video_port,
          local_address: opts.server_ip,
          destination_port_no: video_port,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
        child(:sip_video_rtcp, %UDP.Endpoint{
          local_port_no: src_video_port + 1,
          local_address: opts.server_ip,
          destination_port_no: video_port + 1,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
        child(:sip_video, %UDP.Endpoint{
          local_port_no: src_video_port,
          local_address: opts.server_ip,
          destination_port_no: video_port,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
        child(:sip_video_rtcp, %UDP.Endpoint{
          local_port_no: src_video_port + 1,
          local_address: opts.server_ip,
          destination_port_no: video_port + 1,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
and has following output for video: 
  @impl true
  def handle_child_notification({:video_vr_subscriber, ssrc}, _src, _ctx, state) do
    spec = {[
      get_child(:video_vr_subscriber)
      |> via_out(Pad.ref(:output, ssrc))
      |> child(:h264_encoder, %Membrane.H264.FFmpeg.Encoder{...})
      |> child(:h264_parser, %Membrane.H264.Parser{...})
      |> via_in(Pad.ref(:input, state.video_ssrc), options: [
        payloader: %RTP.H264.Payloader{...}
      ])
      |> get_child(:sip_rtp)
      |> via_out(Pad.ref(:rtp_output, state.video_ssrc), options: [encoding: :H264, payload_type: 109])
      |> via_in(Pad.ref(:input))
      |> get_child(:sip_video),
    ], stream_sync: :sinks}

    {[spec: spec], state}
  end
  @impl true
  def handle_child_notification({:video_vr_subscriber, ssrc}, _src, _ctx, state) do
    spec = {[
      get_child(:video_vr_subscriber)
      |> via_out(Pad.ref(:output, ssrc))
      |> child(:h264_encoder, %Membrane.H264.FFmpeg.Encoder{...})
      |> child(:h264_parser, %Membrane.H264.Parser{...})
      |> via_in(Pad.ref(:input, state.video_ssrc), options: [
        payloader: %RTP.H264.Payloader{...}
      ])
      |> get_child(:sip_rtp)
      |> via_out(Pad.ref(:rtp_output, state.video_ssrc), options: [encoding: :H264, payload_type: 109])
      |> via_in(Pad.ref(:input))
      |> get_child(:sip_video),
    ], stream_sync: :sinks}

    {[spec: spec], state}
  end
How can I debug cause of this?
S
spscream•68d ago
should I to link somethink to SessionBin to handle rtx, if yes, what exactly?
F
Feliks•68d ago
In your case, error is raised because :sip_rtp/{:outbound_rtx_controller, 1929338881} sends buffers to :sip_video at a rate that :sip_video can't keep up (what is a little weird for me, because :sip_video is UDP.Endpoint). For some reason, logs that you have posted are cut in the middle. Try putting this: |> via_in(Pad.ref(:input), toilet_capacity: 1000) before get_child(:sip_video) in the second spec, that you sent. You can change 1000 to any bigger number, that you want. Let me know, if this change fixed your problem.
S
spscream•68d ago
Ok, you give me some directions for debug, thank you btw I'm getting the same error with 1000 now: 
21:01:43.622 pid=<0.1975.0> [error] <0.1286.0>/:sip_rtp/{:outbound_rtx_controller, 4064417025} Toilet overflow.

Atomic demand reached the size of -1001, which means that there are 1001
buffers sent without demanding it, which is above toilet capacity (1000)
when storing data from output working in push mode. It means that some element in the pipeline
processes the stream too slow or doesn't process it at all.
To have control over amount of buffers being produced, consider using output in :auto or :manual
flow control mode. (see `Membrane.Pad.flow_control`).
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.


21:01:43.626 pid=<0.1286.0> [error] <0.1286.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}
21:01:43.622 pid=<0.1975.0> [error] <0.1286.0>/:sip_rtp/{:outbound_rtx_controller, 4064417025} Toilet overflow.

Atomic demand reached the size of -1001, which means that there are 1001
buffers sent without demanding it, which is above toilet capacity (1000)
when storing data from output working in push mode. It means that some element in the pipeline
processes the stream too slow or doesn't process it at all.
To have control over amount of buffers being produced, consider using output in :auto or :manual
flow control mode. (see `Membrane.Pad.flow_control`).
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.


21:01:43.626 pid=<0.1286.0> [error] <0.1286.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}
I set toilet_capacity to 10000 and stream is flowing now. How can I debug the current buffer state of :sip_video? add debug to handle_buffer?
S
spscream•68d ago
https://gist.github.com/spscream/a978129cf3d99bc2d1d5cb8b1eefe944 this is code for my UDP.Endpoint
Gist
endpoint.ex
GitHub Gist: instantly share code, notes, and snippets.
Q
Qizot•68d ago
Could it be that you get a burst of packets at the beginning and then it behaves fine?
S
spscream•68d ago
it might be so. Should I handle toilet overflow some way on my side, e.g. dropping input buffers? I don't want to have very large value for toilet_capacity
F
Feliks•67d ago
In this case, toilet_capacity specifies, how many buffers from outbound_rtx_controller can wait in the mailbox of UDP.Endpoint (to be strict, the exact boundary will be little bit bigger that value passed to this option, but the general sense stays the same). If @Qizot ot is right, UDP.Endpoint receives big batch of packets, which size exceeds significantly toilet_capacity, but it is possible, that UDP.Endpoint will be able to consume this batch over time. If your program works fine after increasing toilet_capacity and you don't see any other bugs, I wouldn't worry about setting this option to a bigger number. The real problem would occur, if the size of UDP.Endpoint mailbox were increasing over time. This scenario should cause 2 effects: 1) after increasing toilet_capacity, you will get Toilet overflowed error, only after a longer time 2) amount of memory used by BEAM will be increasing over a time
Want results from more Discord servers?
Add your server
More Posts
On JF Tracks and Reconnecting (in React)So I noticed a few things about the react-sdk and JF tracks in general. Note I have react code that h264 encoder problemshi guys, I'm using h264 encoder plugin for video encoding and sending it via rtp to client. SometimPipeline for muxing 2 msr files (audio and video) into a single flv fileI have the following pipeline which takes 2 msr files (recorded to disk using the RecordingEntrypoinMP3 output is audible, but test not passHi everyone. I made small changes in `membrane_mp3_lame_plugin` to support other input config (the oGrab keyframe image data from h264 stream?We are looking at some h264 coming from an RTSP stream. Membrane is doing a fine job with the HLS deUnity client?Thanks a lot to the Membrane team. The number of examples and availability of code has been incredibWebRTC stream not workingI've hit an issue trying to get an MPEGTS stream displaying in the browser via WebRTC. All seems toError when compiling free4chatThis is probably pretty basic but I'm at square one. I get an error when I'm compiling free4chat, a Screen ShareIs there any reason why jellyfish_videoroom when is running localy share screen is not working ? I sTesting Membrane ElementI'm trying to setup a simple test of a membrane element, but I'm stumped a bit on how to assert thatClustering and scale out behaviourContext: I've a membrane application running in an elixir cluster of 1. It receives a RTSP stream (URTSP authentication problem?Hi, I am playing with an RTSP camera (**Tapo C210**) that is on the local network. The path to the sExtending the jellyfish video room demo with a queueHey I am curious what a scalable way would be to add a queue in front of jellyfish rooms. I have a sDebugging bundlex/unifex errorsHello, I've been tinkering with membrane cross-compiled to Nerves (rpi4). I've had features (e.g. mRTP to HLS Disconnect and Reconnect Audio StreamHello everyone, I currently have a microphone input which is sending UDP to a server and into a RTPCustom RTC EndpointSorry, more random questions! What's the best place to start when trying to get a custom endpoint tHTTPAdaptiveStream issue with hls.jsI was wondering if anyone else has experieneced this issues with HLS? I have a working pipeline genSpinning up a new GenServer for each roomI have been learning from the videoroom demo and I have a few questions. ```elixir # meeting.ex @