How to WebSocket to Serverless Pods
@Merrell / @flash-singh Wondering if I can get code-pointers on how to use the API to expose ports on serverless programatically, so that I can do stuff like web sockets?
87 Replies
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
they used to be able to have in the UI the ability to open up ports just like GPU Pod, but they said they removed it to not cause customer confusion / etc. But supposedly, since it's just a GPU Pod under the hood of serverless, anything should be possible on serverless that is on GPU Pod.
Well either way, even if its just a dedicated port on serverless for web sockets, that still nice, to be able to stream data / files
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
I think there is no ports exposed at all right now, just a simple handler function; except for certain customers i guess of theirs are using it, but not widely known knowledge
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
you would need to use our graphql api to add ports to a serverless template, from there the workers will have ports, by default gui will not let you add ports
tcp ports are randomly assigned but we do expose them in env variables, so technically you can expose that port back to your infra using progress updates within a job
heres the flow that someone else used
- run a job
- worker will launch and env is populated, in progress of the job share back port / ip details
- catch port / ip details in your app using /status or /stream, do whatever you want with the port / ip
- when your done using the port / ip, send message through that port / ip service to mark the job done on worker side
- once job is done, the worker will turn off automatically if there are no more jobs
Thank you! ❤️ Appreciate it! ❤️
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
on any pod, you can get env variables and ports are exposed as part of env, ill have to check where we document it
is it possible to stream with this? websockets are definitely more efficient
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
considering my method of streaming yes
we just spam the request to get all chunks
websockets should work the same since your using ports, also you can use server-sent events, SSE
https://medium.com/deliveryherotechhub/what-is-server-sent-events-sse-and-how-to-implement-it-904938bffd73
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
heres an example, if you expose 22 port in your template, env will show this
RUNPOD_TCP_PORT_22=22097
that is saying port 22 is mapped to 22097 externally to usersUnknown User•2y ago
Message Not Public
Sign In & Join Server To View
yes, serverless runs on secure cloud and all secure cloud pods have public ip, just keep in mind ip and port can be different for every job you run in serverless
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
did it work?
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
yeah sure
Unknown User•2y ago
Message Not Public
Sign In & Join Server To View
hi @nerdylive did the public port work for you in serverless machine? I'm trying to do the same thing!
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
oh really? why is that? I thought serverless pods and pods are the same secure cloud infra under the hood
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
umm i actually need to expose public ports from serverless machine specifically.. i really wonder why it cannot be achieved..
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
will ngrok work in this case? local tunneling?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
ok great! i'll try ngrok or cloudflare tunneling!
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
cuz i need to run a ui on serverless machine, (comfyui)
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
pod starts up too slow
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
serverless is way faster cold start time than pods
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
i think they cache the container image for serverless
i don't understand why they don't support port expose in serverless, it will be so much more flexible
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
umm but low startup time is a technical need (a very decent need)...they should aim to optimize for that technically
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
yeah..i just hope runpod can be more flexible and powerful on their serverless offering
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
the serverless is really amazing in cold starts. beats every other cloud service (EC2 is too slow..) its the major reason i choose runpod
oh really, i guess i just want the ports hahah
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
apart from local tunneling like ngrok, what other way to do this?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
oh i see, is it different from local tunneling?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
ok no worries. cloudflare tunnel is local tunneling i think
we do support but haven't exposed it as a feature, its something we can do if needed
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
be nice and ping me about it and your reasoning for needing it
it might be possible using our graphql api, you need to set ports on the template which we hide in GUI for serverless
comfyui is likely a cool feature to expose to your users, spin up a job, that jobs lasts until your user quits comfy ui, and you can expose a port / ip per job
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
The trick is to have your serverless worker establish a web socket connection to server. Once that happens you have 2 way communication over the web socket. The serverless worker needs to act as a web socket client, it cannot be a web socket server.
yes i finally made it work with cloudflare tunneling, to spin up a serverless job and start tunneling until the job is canceled.
similar to this code:
https://colab.research.google.com/github/ltdrdata/ComfyUI-Manager/blob/main/notebooks/comfyui_colab_with_manager.ipynb#scrollTo=jjjjjjjjjjjjjj
people in comfyui community is gonna love this sooo much.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
What are you building? Hope you don't mind me asking.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
starting comfy in seconds is so coool 💖 thanks for the great infra at runpod @flash-singh i love the runpod serverless so much, the cold start time is amazing
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
comfyui is sort of stateful because frontend is not fully decoupled from backend code, comfyui relies on the server's file system to serve some static js files (it is really a flaw of design 😕 making serverless comfyui so hard to achieve)
a remote comfyui would be cool.. seperate the front end from the back end.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
Hi im building a comfyui serverless machines platform to let you config your own "comfyui machines" (custom nodes versions, comfyui versions, etc.) and you can share your customized "comfyui machine" with others as serverless app, so people can reuse your comfyui environment
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
yes this is exactly what I'm building. serverless comfyui! let me know if you are interested to test out or colaborate! @Encyrption
Yes, for sure! this could work with the marketplace I am building.
the problem is to keep the model files, input files etc...
oh wow what kind of marketplace are you building? would love to hear about it!
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
@flash-singh thanks for the advice. I tried to use graphql for it, but when mutate saveTemplate, if
isServerless: true is in the payload then it will ignore the ports: '8080/http' that i set, and still set ports to null
so i can only bypass isServerless: true i wonder is this flag required for serverless template? can i just not set it?Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
we will need to update this to allow users this behavious even for serverless, so far one of the reasons we haven't made serverless too flexible is to provide a better experience, slowly we will introduce more and more features
Omg this sounds great! I’m super happy to hear you’re investing more in serverless.❤️❤️
I think this is the future of ai ml service! The magic fast boot time 🚀
Does anyone know if this is possible now?
You can use the RunPod proxy URL if you add /ws onto the end of it and it will work with websockets with HTTP port. There is a bit of a delay before it kicks in though.
Does that also work for serverless, thought it only worked for Pods?
Yes, it works for serverless
It is a bit tricky to pull off the timing though. Your handler has to get your POD_ID from the ENV variable (RUNPOD_POD_ID) and send it out via an update like this:
You can get this info by polling the STATUS endpoint. With the POD_ID you can create the proxy URL with /ws appended.
This works with the RUN endpoint. I am not sure if it is possible when using the RUNSYNC endpoint.
WS_URL = f'wss://{POD_ID}-{WS_PORT}.proxy.runpod.net/ws'
Ahh that makes sense, thank you so much
There are a lot of video on doing this that I've used on Youtube.
GitHub
GitHub - asdkazmi/runpod-serverless-socketio
Contribute to asdkazmi/runpod-serverless-socketio development by creating an account on GitHub.
one more example: https://github.com/runpod-workers/worker-websocket
GitHub
GitHub - runpod-workers/worker-websocket
Contribute to runpod-workers/worker-websocket development by creating an account on GitHub.
is everyone in this thread building a "persistent comfyui but you pay only when workflow runs?"
how do you do this? would like to know.
i want to run comfyui on the cloud because my local machine can't handle it beyond some casual light use
Related to this -- I just came across this blog post: https://blog.runpod.io/introduction-to-websocket-streaming-with-runpod-serverless/ which is excitingly-titled, but the content has nothing to do with websockets 🙂
RunPod Blog
Introduction to Websocket Streaming with RunPod Serverless
In this followup to our 'Hello World' tutorial, we'll create a serverless endpoint that processes base64-encoded files and streams back the results. This will demonstrate how you can work with file input/output over our serverless environment by encoding the file as data within a JSON payload.
Prerequisites
* Complete our
Unknown User•7mo ago
Message Not Public
Sign In & Join Server To View