Fly.io + UDP

I've got an membrane_webrtc server setup where someone can "call" an LLM and talk with them (audio only, no video). It largely works, though my users are reporting random disconnects. The console errors match the attached image. I'm a little thrown since the

url

url

in that message specifies UDP as the transport. I deployed to fly.io, and explicitly did not open up the UDP ports in my fly.toml, so I'm wondering why the app is failing with a UDP timeout. Am I incorrect in assuming that I can force all traffic over TCP by just not opening it up? Shoudl I also figure out UDP?

On UDP, I read through this: https://github.com/fishjam-dev/fishjam-docs/blob/main/docs/deploying/fly_io.md

But it's Fishjam specific, and it doesn't line up neatly with my app which is based on the old membrane video room repo (https://github.com/membraneframework-labs/membrane_videoroom). Where does fly-global-services get specified in that case? I'm not explicitly setting a TURN_LISTEN_IP. I traced through things I think it could be here in

turn_ip

turn_ip

turn_ip

turn_ip (and then the

turn_mock_ip

turn_mock_ip

turn_mock_ip

turn_mock_ip is my external IPv4 address)

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

    turn_mock_ip = Application.fetch_env!(:smartvox, :integrated_turn_ip)
    turn_ip = 'fly-global-services'
    #turn_ip = {0, 0, 0, 0} #prior value

    integrated_turn_options = [
      ip: turn_ip,
      mock_ip: turn_mock_ip,
      ports_range: Application.fetch_env!(:smartvox, :integrated_turn_port_range),
      cert_file: turn_cert_file
    ]

Which is in my

room.ex

room.ex

room.ex

room.ex -- those options are used here:

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

TURNManager.ensure_tcp_turn_launched(integrated_turn_options, port: tcp_turn_port)

But that throws an error when I override turn_ip that way.

So, (1) does that console error suggest I should be looking into why UDP is failing and (2) any tips on trying to configure UDP on Fly.io specifically?

GitHub

fishjam-docs/docs/deploying/fly_io.md at main · fishjam-dev/fishjam...

Fishjam documentation. Contribute to fishjam-dev/fishjam-docs development by creating an account on GitHub.

Michał Śledź•9/4/24, 5:33 PM

Hi @TonyLikeSocks

Michał Śledź•9/4/24, 5:33 PM

We don't support Fly.io

Michał Śledź•9/4/24, 5:33 PM

Their network configuration was problematic for WebRTC based apps at least the last time we checked

Michał Śledź•9/4/24, 5:34 PM

and we also no longer maintain RTC Engine

TonyLikeSocksOP•9/4/24, 6:07 PM

yeah, I heard. Migrating right now isn’t in the cards to ex webrtc though. I’ve got too much built out.

Does the error message suggest I need UDP?

Michał Śledź•9/4/24, 6:13 PM

as long as you have TCP enabled, I belive that chrome should fallback to it when it cannot connect via UDP

TonyLikeSocksOP•9/4/24, 6:24 PM

Yeah, that was our assumption when we launched. That error message is really throwing me

Michał Śledź•9/4/24, 6:25 PM

you can go under chrome://webrtc-internals and see wheter tcp turn is on and what's its status

TonyLikeSocksOP•9/4/24, 6:42 PM

Ok - so the advice is to migrate off fly. It should work, but the evidence suggests it is not

TonyLikeSocksOP•9/4/24, 6:43 PM

At least not reliably

Michał Śledź•9/4/24, 6:47 PM

It's hard to say something more without looking at the code are going deeper into the problem

We always recomand bare providers like Hetzner for example, which we use for all our apps :/

TonyLikeSocksOP•9/4/24, 6:52 PM

Thanks @mickel8 Super helpful convo. I’ll work on moving to a bare provider.

Michał Śledź•9/4/24, 7:05 PM

Hope it will work!

TonyLikeSocksOP•9/4/24, 11:43 PM

Thanks for the tip on chrome://webrtc-internals. Testing now (with no changes, still on fly) -- Chrome thinks UDP is connected and sending / receiving bytes. But I also see the same timeout errors, and I never exposed UDP in my fly.toml. Very odd. For future searches, Fly.io is weird.

shuntrho•9/6/24, 6:32 AM

Hi @TonyLikeSocks , I agree with mickel8 that it's probably best to migrate away from fly.io, as we've been having issues with deployment there.
Anyway, while you're there you can try to fix the error with turn ip. You cannot passfly-global-servicesfly-global-services address to turn options. The ipip in the integrated_turn_optionsintegrated_turn_options has to be a tuple. That means that you have resolve the address into an ipv4 before passing it to turn options. For this, you can use functions from Fishjam's ConfigParser:
https://github.com/fishjam-dev/fishjam/blob/main/lib/fishjam/config_reader.ex#L39
and https://github.com/fishjam-dev/fishjam/blob/main/lib/fishjam/config_reader.ex#L368 .

Make sure to read https://fly.io/docs/networking/udp-and-tcp/ guide. If this approach fails, you may also try somehow forcing chrome to use TCP turn

Fly

Running apps on UDP and TCP

Documentation and guides from the team at Fly.io.

GitHub

fishjam/lib/fishjam/config_reader.ex at main · fishjam-dev/fishjam

General purpose media server. Supports WebRTC, HLS, RTSP, SIP - fishjam-dev/fishjam

TonyLikeSocksOP•9/6/24, 6:49 PM

Thanks @shuntrho. That’s helpful to know on needing to pass in the tuple not a domain name.

I am migrating away from fly.

Fly.io + UDP

Similar Threads

Similar Threads

Similar Threads