How to WebSocket to Serverless Pods
@Merrell / @flash-singh Wondering if I can get code-pointers on how to use the API to expose ports on serverless programatically, so that I can do stuff like web sockets?
81 Replies
Wow if this is possible then it's great
But I think they'll allow some ports for you to use rather than allowing you to control the open ports on serverless ..
they used to be able to have in the UI the ability to open up ports just like GPU Pod, but they said they removed it to not cause customer confusion / etc. But supposedly, since it's just a GPU Pod under the hood of serverless, anything should be possible on serverless that is on GPU Pod.
Well either way, even if its just a dedicated port on serverless for web sockets, that still nice, to be able to stream data / files
Ooh that's weird customer confusion? So, Now they simplify it by automating the assignment of ports ye
Yeah I'm interested in this too π
I think there is no ports exposed at all right now, just a simple handler function; except for certain customers i guess of theirs are using it, but not widely known knowledge
Oh yea for gpu pods* I mean tcp ports are randomly assigned right
you would need to use our graphql api to add ports to a serverless template, from there the workers will have ports, by default gui will not let you add ports
tcp ports are randomly assigned but we do expose them in env variables, so technically you can expose that port back to your infra using progress updates within a job
heres the flow that someone else used
- run a job
- worker will launch and env is populated, in progress of the job share back port / ip details
- catch port / ip details in your app using /status or /stream, do whatever you want with the port / ip
- when your done using the port / ip, send message through that port / ip service to mark the job done on worker side
- once job is done, the worker will turn off automatically if there are no more jobs
Thank you! β€οΈ Appreciate it! β€οΈ
Oh is it already possible by now and documented in the reference?
Which env variable BTW?
on any pod, you can get env variables and ports are exposed as part of env, ill have to check where we document it
is it possible to stream with this? websockets are definitely more efficient
Yeah oh really? We sockets more efficient?
considering my method of streaming yes
we just spam the request to get all chunks
websockets should work the same since your using ports, also you can use server-sent events, SSE
https://medium.com/deliveryherotechhub/what-is-server-sent-events-sse-and-how-to-implement-it-904938bffd73
Hey @flash-singh have you got the env variable yet?
Please send the env name for exposed ports if there is
heres an example, if you expose 22 port in your template, env will show this
RUNPOD_TCP_PORT_22=22097
that is saying port 22 is mapped to 22097 externally to usersOhh it works on pods and serverless? ok thanks
Oh and will i be able to get the RUNPOD_PUBLIC_IP everytime?
for serverless
yes, serverless runs on secure cloud and all secure cloud pods have public ip, just keep in mind ip and port can be different for every job you run in serverless
i did this on dockerfile:
EXPOSE 22/tcp
I did the graphql port assignment too on tcpdid it work?
Nah as I said it returned only none object on python
Why is that?
Do you wanna check out my template to see if the ports is there?
yeah sure
@flash-singh 3pgp4kglgq heres the template id
sorry for the wait lol
can you make sure if its already right
and any updates on the public ip env or the port env?
or any alternatives for websockets in serverless? ( it needs ports right? which defaults to http or https )
i should've used the stream instead for now yeah
hi @nerdylive did the public port work for you in serverless machine? I'm trying to do the same thing!
no
its not supported yet
right @flash-singh ?
Another alternative is to start ws conection to your own server
oh really? why is that? I thought serverless pods and pods are the same secure cloud infra under the hood
from serverless -> your server
yeah like different platform
umm i actually need to expose public ports from serverless machine specifically.. i really wonder why it cannot be achieved..
well their limitations
will ngrok work in this case? local tunneling?
Hmm yes
ngrok, cloudflared works
but you have to pass it programatically, since it'll be ineffecient to run it manually from a terminal right
ok great! i'll try ngrok or cloudflare tunneling!
btw what are you trying to do with that?
cuz i need to run a ui on serverless machine, (comfyui)
oh
pod starts up too slow
damn hahahah
you're gonna make a proxy for comfyui ?
serverless is way faster cold start time than pods
true
i think they cache the container image for serverless
i don't understand why they don't support port expose in serverless, it will be so much more flexible
hmm this feels abit like abusing the power of serverless lol
yeah they say they usually allow them but because of its complexity for users they decided to remove it... from what i remember
umm but low startup time is a technical need (a very decent need)...they should aim to optimize for that technically
True
Can be done in many ways too ( from their side ofc )
yeah..i just hope runpod can be more flexible and powerful on their serverless offering
Flexible like what
I thought it's already powerful enough π
the serverless is really amazing in cold starts. beats every other cloud service (EC2 is too slow..) its the major reason i choose runpod
oh really, i guess i just want the ports hahah
Oh rlly
apart from local tunneling like ngrok, what other way to do this?
Ec2 serverless? Hmm
Idk, never thought about this..
Reverse tunneling is the way to bypass it
oh i see, is it different from local tunneling?
I'm not sure what's local tunneling
ok no worries. cloudflare tunnel is local tunneling i think
we do support but haven't exposed it as a feature, its something we can do if needed
hmm how can i get it then?
be nice and ping me about it and your reasoning for needing it
it might be possible using our graphql api, you need to set ports on the template which we hide in GUI for serverless
comfyui is likely a cool feature to expose to your users, spin up a job, that jobs lasts until your user quits comfy ui, and you can expose a port / ip per job
@briefPeach @flash-singh
Ohk then, I don't think I need it now since I have used stream and it works kinda well for now
The trick is to have your serverless worker establish a web socket connection to server. Once that happens you have 2 way communication over the web socket. The serverless worker needs to act as a web socket client, it cannot be a web socket server.
yes i finally made it work with cloudflare tunneling, to spin up a serverless job and start tunneling until the job is canceled.
similar to this code:
https://colab.research.google.com/github/ltdrdata/ComfyUI-Manager/blob/main/notebooks/comfyui_colab_with_manager.ipynb#scrollTo=jjjjjjjjjjjjjj
people in comfyui community is gonna love this sooo much.
Not like per request?
I thought comfyui is like stateless
Is it not?
What are you building? Hope you don't mind me asking.
Yeah that's what I said to b4
starting comfy in seconds is so coool π thanks for the great infra at runpod @flash-singh i love the runpod serverless so much, the cold start time is amazing
I was talking about comfyui
comfyui is sort of stateful because frontend is not fully decoupled from backend code, comfyui relies on the server's file system to serve some static js files (it is really a flaw of design π making serverless comfyui so hard to achieve)
a remote comfyui would be cool.. seperate the front end from the back end.
Oh yeah sessions
Hi im building a comfyui serverless machines platform to let you config your own "comfyui machines" (custom nodes versions, comfyui versions, etc.) and you can share your customized "comfyui machine" with others as serverless app, so people can reuse your comfyui environment
But it can be stateless right
By keeping the session ids
yes this is exactly what I'm building. serverless comfyui! let me know if you are interested to test out or colaborate! @Encyrption
Yes, for sure! this could work with the marketplace I am building.
the problem is to keep the model files, input files etc...
oh wow what kind of marketplace are you building? would love to hear about it!
Woo what marketplace?
Yeah could use some front-end storage and backend storage for that
You can add custom logic for that if you wanna store them in serverless btw
@flash-singh thanks for the advice. I tried to use graphql for it, but when mutate saveTemplate, if
isServerless: true is in the payload then it will ignore the ports: '8080/http' that i set, and still set ports to null
so i can only bypass isServerless: true i wonder is this flag required for serverless template? can i just not set it?Last time I tried on graphql it didn't work well, I guess you need Runpod's assistance to activate that on an endpoint
we will need to update this to allow users this behavious even for serverless, so far one of the reasons we haven't made serverless too flexible is to provide a better experience, slowly we will introduce more and more features
Omg this sounds great! Iβm super happy to hear youβre investing more in serverless.β€οΈβ€οΈ
I think this is the future of ai ml service! The magic fast boot time π
Does anyone know if this is possible now?
You can use the RunPod proxy URL if you add /ws onto the end of it and it will work with websockets with HTTP port. There is a bit of a delay before it kicks in though.
Does that also work for serverless, thought it only worked for Pods?
Yes, it works for serverless
It is a bit tricky to pull off the timing though. Your handler has to get your POD_ID from the ENV variable (RUNPOD_POD_ID) and send it out via an update like this:
You can get this info by polling the STATUS endpoint. With the POD_ID you can create the proxy URL with /ws appended.
This works with the RUN endpoint. I am not sure if it is possible when using the RUNSYNC endpoint.
WS_URL = f'wss://{POD_ID}-{WS_PORT}.proxy.runpod.net/ws'
Ahh that makes sense, thank you so much
There are a lot of video on doing this that I've used on Youtube.