R
RunPod3mo ago
justin

How to WebSocket to Serverless Pods

@Merrell / @flash-singh Wondering if I can get code-pointers on how to use the API to expose ports on serverless programatically, so that I can do stuff like web sockets?
75 Replies
nerdylive
nerdylive3mo ago
Wow if this is possible then it's great But I think they'll allow some ports for you to use rather than allowing you to control the open ports on serverless ..
justin
justin3mo ago
they used to be able to have in the UI the ability to open up ports just like GPU Pod, but they said they removed it to not cause customer confusion / etc. But supposedly, since it's just a GPU Pod under the hood of serverless, anything should be possible on serverless that is on GPU Pod. Well either way, even if its just a dedicated port on serverless for web sockets, that still nice, to be able to stream data / files
nerdylive
nerdylive3mo ago
Ooh that's weird customer confusion? So, Now they simplify it by automating the assignment of ports ye Yeah I'm interested in this too 👌
justin
justin3mo ago
I think there is no ports exposed at all right now, just a simple handler function; except for certain customers i guess of theirs are using it, but not widely known knowledge
nerdylive
nerdylive3mo ago
Oh yea for gpu pods* I mean tcp ports are randomly assigned right
flash-singh
flash-singh3mo ago
you would need to use our graphql api to add ports to a serverless template, from there the workers will have ports, by default gui will not let you add ports tcp ports are randomly assigned but we do expose them in env variables, so technically you can expose that port back to your infra using progress updates within a job heres the flow that someone else used - run a job - worker will launch and env is populated, in progress of the job share back port / ip details - catch port / ip details in your app using /status or /stream, do whatever you want with the port / ip - when your done using the port / ip, send message through that port / ip service to mark the job done on worker side - once job is done, the worker will turn off automatically if there are no more jobs
justin
justin3mo ago
Thank you! ❤️ Appreciate it! ❤️
nerdylive
nerdylive3mo ago
Oh is it already possible by now and documented in the reference? Which env variable BTW?
flash-singh
flash-singh3mo ago
on any pod, you can get env variables and ports are exposed as part of env, ill have to check where we document it
Builderman
Builderman3mo ago
is it possible to stream with this? websockets are definitely more efficient
nerdylive
nerdylive3mo ago
Yeah oh really? We sockets more efficient?
Builderman
Builderman3mo ago
considering my method of streaming yes we just spam the request to get all chunks
flash-singh
flash-singh3mo ago
websockets should work the same since your using ports, also you can use server-sent events, SSE https://medium.com/deliveryherotechhub/what-is-server-sent-events-sse-and-how-to-implement-it-904938bffd73
nerdylive
nerdylive3mo ago
Hey @flash-singh have you got the env variable yet? Please send the env name for exposed ports if there is
flash-singh
flash-singh3mo ago
heres an example, if you expose 22 port in your template, env will show this RUNPOD_TCP_PORT_22=22097 that is saying port 22 is mapped to 22097 externally to users
nerdylive
nerdylive3mo ago
Ohh it works on pods and serverless? ok thanks Oh and will i be able to get the RUNPOD_PUBLIC_IP everytime? for serverless
flash-singh
flash-singh3mo ago
yes, serverless runs on secure cloud and all secure cloud pods have public ip, just keep in mind ip and port can be different for every job you run in serverless
nerdylive
nerdylive3mo ago
port_22 = os.environ.get('RUNPOD_TCP_PORT_22')
public_ip = os.environ.get('RUNPOD_PUBLIC_IP')

#ITs all none when i return them
return {
'message': "public ip: {}, port_ref: {}".format(public_ip, port_8188) # None, None
}
port_22 = os.environ.get('RUNPOD_TCP_PORT_22')
public_ip = os.environ.get('RUNPOD_PUBLIC_IP')

#ITs all none when i return them
return {
'message': "public ip: {}, port_ref: {}".format(public_ip, port_8188) # None, None
}
i did this on dockerfile: EXPOSE 22/tcp I did the graphql port assignment too on tcp
flash-singh
flash-singh3mo ago
did it work?
nerdylive
nerdylive3mo ago
Nah as I said it returned only none object on python Why is that? Do you wanna check out my template to see if the ports is there?
flash-singh
flash-singh3mo ago
yeah sure
nerdylive
nerdylive3mo ago
@flash-singh 3pgp4kglgq heres the template id sorry for the wait lol can you make sure if its already right and any updates on the public ip env or the port env? or any alternatives for websockets in serverless? ( it needs ports right? which defaults to http or https ) i should've used the stream instead for now yeah
briefPeach
briefPeach2w ago
hi @nerdylive did the public port work for you in serverless machine? I'm trying to do the same thing!
nerdylive
nerdylive2w ago
no its not supported yet right @flash-singh ? Another alternative is to start ws conection to your own server
briefPeach
briefPeach2w ago
oh really? why is that? I thought serverless pods and pods are the same secure cloud infra under the hood
nerdylive
nerdylive2w ago
from serverless -> your server yeah like different platform
briefPeach
briefPeach2w ago
umm i actually need to expose public ports from serverless machine specifically.. i really wonder why it cannot be achieved..
nerdylive
nerdylive2w ago
well their limitations
briefPeach
briefPeach2w ago
will ngrok work in this case? local tunneling?
nerdylive
nerdylive2w ago
Hmm yes ngrok, cloudflared works but you have to pass it programatically, since it'll be ineffecient to run it manually from a terminal right
briefPeach
briefPeach2w ago
ok great! i'll try ngrok or cloudflare tunneling!
nerdylive
nerdylive2w ago
btw what are you trying to do with that?
briefPeach
briefPeach2w ago
cuz i need to run a ui on serverless machine, (comfyui)
nerdylive
nerdylive2w ago
oh
briefPeach
briefPeach2w ago
pod starts up too slow
nerdylive
nerdylive2w ago
damn hahahah you're gonna make a proxy for comfyui ?
briefPeach
briefPeach2w ago
serverless is way faster cold start time than pods
nerdylive
nerdylive2w ago
true
briefPeach
briefPeach2w ago
i think they cache the container image for serverless i don't understand why they don't support port expose in serverless, it will be so much more flexible
nerdylive
nerdylive2w ago
hmm this feels abit like abusing the power of serverless lol yeah they say they usually allow them but because of its complexity for users they decided to remove it... from what i remember
briefPeach
briefPeach2w ago
umm but low startup time is a technical need (a very decent need)...they should aim to optimize for that technically
nerdylive
nerdylive2w ago
True Can be done in many ways too ( from their side ofc )
briefPeach
briefPeach2w ago
yeah..i just hope runpod can be more flexible and powerful on their serverless offering
nerdylive
nerdylive2w ago
Flexible like what I thought it's already powerful enough 😅
briefPeach
briefPeach2w ago
the serverless is really amazing in cold starts. beats every other cloud service (EC2 is too slow..) its the major reason i choose runpod oh really, i guess i just want the ports hahah
nerdylive
nerdylive2w ago
Oh rlly
briefPeach
briefPeach2w ago
apart from local tunneling like ngrok, what other way to do this?
nerdylive
nerdylive2w ago
Ec2 serverless? Hmm Idk, never thought about this.. Reverse tunneling is the way to bypass it
briefPeach
briefPeach2w ago
oh i see, is it different from local tunneling?
nerdylive
nerdylive2w ago
I'm not sure what's local tunneling
briefPeach
briefPeach2w ago
ok no worries. cloudflare tunnel is local tunneling i think
flash-singh
flash-singh2w ago
we do support but haven't exposed it as a feature, its something we can do if needed
nerdylive
nerdylive2w ago
hmm how can i get it then?
flash-singh
flash-singh2w ago
be nice and ping me about it and your reasoning for needing it it might be possible using our graphql api, you need to set ports on the template which we hide in GUI for serverless comfyui is likely a cool feature to expose to your users, spin up a job, that jobs lasts until your user quits comfy ui, and you can expose a port / ip per job
nerdylive
nerdylive2w ago
@briefPeach @flash-singh Ohk then, I don't think I need it now since I have used stream and it works kinda well for now
Encyrption
Encyrption2w ago
The trick is to have your serverless worker establish a web socket connection to server. Once that happens you have 2 way communication over the web socket. The serverless worker needs to act as a web socket client, it cannot be a web socket server.
briefPeach
briefPeach2w ago
yes i finally made it work with cloudflare tunneling, to spin up a serverless job and start tunneling until the job is canceled. similar to this code: https://colab.research.google.com/github/ltdrdata/ComfyUI-Manager/blob/main/notebooks/comfyui_colab_with_manager.ipynb#scrollTo=jjjjjjjjjjjjjj people in comfyui community is gonna love this sooo much.
nerdylive
nerdylive2w ago
Not like per request? I thought comfyui is like stateless Is it not?
Encyrption
Encyrption2w ago
What are you building? Hope you don't mind me asking.
nerdylive
nerdylive2w ago
Yeah that's what I said to b4
briefPeach
briefPeach2w ago
starting comfy in seconds is so coool 💖 thanks for the great infra at runpod @flash-singh i love the runpod serverless so much, the cold start time is amazing
nerdylive
nerdylive2w ago
I was talking about comfyui
briefPeach
briefPeach2w ago
comfyui is sort of stateful because frontend is not fully decoupled from backend code, comfyui relies on the server's file system to serve some static js files (it is really a flaw of design 😕 making serverless comfyui so hard to achieve)
Encyrption
Encyrption2w ago
a remote comfyui would be cool.. seperate the front end from the back end.
nerdylive
nerdylive2w ago
Oh yeah sessions
briefPeach
briefPeach2w ago
Hi im building a comfyui serverless machines platform to let you config your own "comfyui machines" (custom nodes versions, comfyui versions, etc.) and you can share your customized "comfyui machine" with others as serverless app, so people can reuse your comfyui environment
nerdylive
nerdylive2w ago
But it can be stateless right By keeping the session ids
briefPeach
briefPeach2w ago
yes this is exactly what I'm building. serverless comfyui! let me know if you are interested to test out or colaborate! @Encyrption
Encyrption
Encyrption2w ago
Yes, for sure! this could work with the marketplace I am building.
briefPeach
briefPeach2w ago
the problem is to keep the model files, input files etc... oh wow what kind of marketplace are you building? would love to hear about it!
nerdylive
nerdylive2w ago
Woo what marketplace? Yeah could use some front-end storage and backend storage for that You can add custom logic for that if you wanna store them in serverless btw
briefPeach
briefPeach2w ago
@flash-singh thanks for the advice. I tried to use graphql for it, but when mutate saveTemplate, if isServerless: true is in the payload then it will ignore the ports: '8080/http' that i set, and still set ports to null so i can only bypass isServerless: true i wonder is this flag required for serverless template? can i just not set it?
nerdylive
nerdylive2w ago
Last time I tried on graphql it didn't work well, I guess you need Runpod's assistance to activate that on an endpoint
flash-singh
flash-singh2w ago
we will need to update this to allow users this behavious even for serverless, so far one of the reasons we haven't made serverless too flexible is to provide a better experience, slowly we will introduce more and more features
briefPeach
briefPeach2w ago
Omg this sounds great! I’m super happy to hear you’re investing more in serverless.❤️❤️ I think this is the future of ai ml service! The magic fast boot time 🚀
Want results from more Discord servers?
Add your server
More Posts