toilet capacity of outbound_rtx_controller

Hi, I'm getting the following error on SessionBin:

[error] <0.1282.0>/:sip_rtp/{:outbound_rtx_controller, 1929338881} Toilet overflow.

Atomic demand reached the size of -201, which means that there are 201
...
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.

[error] <0.1282.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}

[error] <0.1282.0>/:sip_rtp/{:outbound_rtx_controller, 1929338881} Toilet overflow.

Atomic demand reached the size of -201, which means that there are 201
...
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.

[error] <0.1282.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}

sip_rtp is SessionBin with following inputs for video:

        child(:sip_video, %UDP.Endpoint{
          local_port_no: src_video_port,
          local_address: opts.server_ip,
          destination_port_no: video_port,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
        child(:sip_video_rtcp, %UDP.Endpoint{
          local_port_no: src_video_port + 1,
          local_address: opts.server_ip,
          destination_port_no: video_port + 1,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),

        child(:sip_video, %UDP.Endpoint{
          local_port_no: src_video_port,
          local_address: opts.server_ip,
          destination_port_no: video_port,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),
        child(:sip_video_rtcp, %UDP.Endpoint{
          local_port_no: src_video_port + 1,
          local_address: opts.server_ip,
          destination_port_no: video_port + 1,
          destination_address: video_dst
        })
        |> via_in(:rtp_input)
        |> get_child(:sip_rtp),

and has following output for video:

  @impl true
  def handle_child_notification({:video_vr_subscriber, ssrc}, _src, _ctx, state) do
    spec = {[
      get_child(:video_vr_subscriber)
      |> via_out(Pad.ref(:output, ssrc))
      |> child(:h264_encoder, %Membrane.H264.FFmpeg.Encoder{...})
      |> child(:h264_parser, %Membrane.H264.Parser{...})
      |> via_in(Pad.ref(:input, state.video_ssrc), options: [
        payloader: %RTP.H264.Payloader{...}
      ])
      |> get_child(:sip_rtp)
      |> via_out(Pad.ref(:rtp_output, state.video_ssrc), options: [encoding: :H264, payload_type: 109])
      |> via_in(Pad.ref(:input))
      |> get_child(:sip_video),
    ], stream_sync: :sinks}

    {[spec: spec], state}
  end

  @impl true
  def handle_child_notification({:video_vr_subscriber, ssrc}, _src, _ctx, state) do
    spec = {[
      get_child(:video_vr_subscriber)
      |> via_out(Pad.ref(:output, ssrc))
      |> child(:h264_encoder, %Membrane.H264.FFmpeg.Encoder{...})
      |> child(:h264_parser, %Membrane.H264.Parser{...})
      |> via_in(Pad.ref(:input, state.video_ssrc), options: [
        payloader: %RTP.H264.Payloader{...}
      ])
      |> get_child(:sip_rtp)
      |> via_out(Pad.ref(:rtp_output, state.video_ssrc), options: [encoding: :H264, payload_type: 109])
      |> via_in(Pad.ref(:input))
      |> get_child(:sip_video),
    ], stream_sync: :sinks}

    {[spec: spec], state}
  end

How can I debug cause of this?

7 Replies

spscream•5mo ago

should I to link somethink to SessionBin to handle rtx, if yes, what exactly?

Feliks•5mo ago

In your case, error is raised because :sip_rtp/{:outbound_rtx_controller, 1929338881} sends buffers to :sip_video at a rate that :sip_video can't keep up (what is a little weird for me, because :sip_video is UDP.Endpoint). For some reason, logs that you have posted are cut in the middle. Try putting this: |> via_in(Pad.ref(:input), toilet_capacity: 1000) before get_child(:sip_video) in the second spec, that you sent. You can change 1000 to any bigger number, that you want. Let me know, if this change fixed your problem.

spscream•5mo ago

Ok, you give me some directions for debug, thank you btw I'm getting the same error with 1000 now:

21:01:43.622 pid=<0.1975.0> [error] <0.1286.0>/:sip_rtp/{:outbound_rtx_controller, 4064417025} Toilet overflow.

Atomic demand reached the size of -1001, which means that there are 1001
buffers sent without demanding it, which is above toilet capacity (1000)
when storing data from output working in push mode. It means that some element in the pipeline
processes the stream too slow or doesn't process it at all.
To have control over amount of buffers being produced, consider using output in :auto or :manual
flow control mode. (see `Membrane.Pad.flow_control`).
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.


21:01:43.626 pid=<0.1286.0> [error] <0.1286.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}

21:01:43.622 pid=<0.1975.0> [error] <0.1286.0>/:sip_rtp/{:outbound_rtx_controller, 4064417025} Toilet overflow.

Atomic demand reached the size of -1001, which means that there are 1001
buffers sent without demanding it, which is above toilet capacity (1000)
when storing data from output working in push mode. It means that some element in the pipeline
processes the stream too slow or doesn't process it at all.
To have control over amount of buffers being produced, consider using output in :auto or :manual
flow control mode. (see `Membrane.Pad.flow_control`).
You can also try changing the `toilet_capacity` in `Membrane.ChildrenSpec.via_in/3`.


21:01:43.626 pid=<0.1286.0> [error] <0.1286.0>/ Terminating with reason: {:membrane_child_crash, :sip_video, :killed}

I set toilet_capacity to 10000 and stream is flowing now. How can I debug the current buffer state of :sip_video? add debug to handle_buffer?

spscream•5mo ago

https://gist.github.com/spscream/a978129cf3d99bc2d1d5cb8b1eefe944 this is code for my UDP.Endpoint

Gist

endpoint.ex

GitHub Gist: instantly share code, notes, and snippets.

Qizot•5mo ago

Could it be that you get a burst of packets at the beginning and then it behaves fine?

spscream•5mo ago

it might be so. Should I handle toilet overflow some way on my side, e.g. dropping input buffers? I don't want to have very large value for toilet_capacity

Feliks•5mo ago

In this case, toilet_capacity specifies, how many buffers from outbound_rtx_controller can wait in the mailbox of UDP.Endpoint (to be strict, the exact boundary will be little bit bigger that value passed to this option, but the general sense stays the same). If @Qizot ot is right, UDP.Endpoint receives big batch of packets, which size exceeds significantly toilet_capacity, but it is possible, that UDP.Endpoint will be able to consume this batch over time. If your program works fine after increasing toilet_capacity and you don't see any other bugs, I wouldn't worry about setting this option to a bigger number. The real problem would occur, if the size of UDP.Endpoint mailbox were increasing over time. This scenario should cause 2 effects: 1) after increasing toilet_capacity, you will get Toilet overflowed error, only after a longer time 2) amount of memory used by BEAM will be increasing over a time

Gaming

Programming

toilet capacity of outbound_rtx_controller