Issue with Websocket latency over serverless http proxy since runpod outage
We have a runpod serverless endpoint which we have been using to stream frames over direct one-to-one websocket. We have a lightweight version of this endpoint we've been using that streams simple diagnostic images, and a production version that streams AI generated frames. Frames are configured to stream at 18fps in both cases to create an animation.
We now see that both versions of this endpoint fail to stream frames at a reasonable rate, hovering around 1 fps. The lightweight diagnostic frames take virtually no time to generate, and we have confirmed with logging that the AI generated frames in the production version are not generating any slower, and should still be able to meet the 18 fps demand. But we see that the time to send frames over websocket is on the order of 1s per frame, and is very unstable. See below a snippet from our logs showing fast image generation times, but slow times for sending images over websocket
Compare this to the attached screen shot showing a previously working version in which we can see from the logs that we are receiving many more than 1 frame within a one second window.
We only started seeing this issue after runpod came back up from the outage earlier today. We have been testing with this setup in a variety of configurations over the last two weeks and have only started seeing this issue as of today after the runpod outage occurred.
We would very much appreciate some attention on this issue @Dj. It is very impactful at the moment for our org. Could you let us know if there are other tests we could do on our end that would provide helpful data to assess root cause and identify a solution? Thanks very much for your help.
Tagging @huemin for visibility.
We now see that both versions of this endpoint fail to stream frames at a reasonable rate, hovering around 1 fps. The lightweight diagnostic frames take virtually no time to generate, and we have confirmed with logging that the AI generated frames in the production version are not generating any slower, and should still be able to meet the 18 fps demand. But we see that the time to send frames over websocket is on the order of 1s per frame, and is very unstable. See below a snippet from our logs showing fast image generation times, but slow times for sending images over websocket
Compare this to the attached screen shot showing a previously working version in which we can see from the logs that we are receiving many more than 1 frame within a one second window.
We only started seeing this issue after runpod came back up from the outage earlier today. We have been testing with this setup in a variety of configurations over the last two weeks and have only started seeing this issue as of today after the runpod outage occurred.
We would very much appreciate some attention on this issue @Dj. It is very impactful at the moment for our org. Could you let us know if there are other tests we could do on our end that would provide helpful data to assess root cause and identify a solution? Thanks very much for your help.
Tagging @huemin for visibility.
