Exposing http ports on serverless
There's no way to expose http ports on serverless is there? When I'm creating a new template and flip the template type from Pod to Serverless that option goes away.
66 Replies
I see stuff in the codebase about
RUNPOD_REALTIME_PORT but I'm not sure what the use case is for that and I haven't found documentation for it. Also I'd like to expose two ports to my serverless instancesWhy would you want to expose HTTP ports on serverless? Its not designed for that. Use pods instead.
I'm creating a version of this https://www.instagram.com/p/C8CzGuOubKp/ that anybody can use without setup. Sending frames over a websocket connection at 24fps.
I have it working with pods, but I'd rather not manage standing up and tearing down the instances for each user
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
That's what I think! But I can't find anyone else interested in real time ai or ai vtubing
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
it all runs on one 3090
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
Yup. sdxl turbo https://github.com/GenDJ/GenDJ/blob/main/diffusion_processor.py
GitHub
GenDJ/diffusion_processor.py at main · GenDJ/GenDJ
Contribute to GenDJ/GenDJ development by creating an account on GitHub.
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
is there any off the shelf or open source solution for managing a bunch of pods? Ideally I'd like to have some sitting idle for users so they can instantly start using it without waiting a few minutes for the server to stand up and for the start script to run setting up the project etc
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
how? That would be amazing. I'm also serving a web server hosting the frontend from the pod but I can probably figure out how to decouple that from the websocket and processing
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
ah so it's in that land
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
the pod version of this is already a custom template. Works amazingly well
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
well it's handy for the pod version
I think the non-pod version will have to be a pretty different architecture with a whole webapp
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
yeah that's why I was hoping to have some solution out there for managing the fleet of pods. Obviously for cost reasons I wanna have a few active waiting for people as possible, so it'd have to be dynamic and pretty complex logic of when to stand new ones up/tear old ones down
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
side note currently I'm stuffing the actual models inside of the container which causes my docker images to be like 20 gigs, takes forever to upload and is a terrible developer experience. How are people doing it?
why graphql api instead of python sdk?
I was gonna use this https://github.com/runpod/runpod-python
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
can I auto mount a network drive on all new pods I stand up?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
ahh dealbreaker
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
well I've noticed some regions aren't good for maintaining the websocket connection. It keeps dropping. and also regions run out of gpus quite frequently in my experience. I can't couple myself to one region, especially not when the current images I have are working fine just huge and annoying
tradeoff not worth it
wish there were more gpus in usa regions
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
most lol
RO is working fine
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
never had the gpus I want available in US
ever lol
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
DCs?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
do you work at runpod? you're helpful btw tks for answering my random questions
I've felt like I'm wandering through a dark forest lol
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
so I guess my next step is to build the whole dang pod management thing
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
but geez the serverless stuff all works great I just would love to use that with the ports exposed
spin up an job when a user wants to do the live thing, spin it down when theyre done
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
not sure what you mean
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
I think you need the port exposed to make a websocket connection no?
hold on spinning up a pod without the ws port open to confirm
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
oh snap
well nah it needs to go both ways
well wait no that might work
im gonna quick spin something up locally and ngrok into it and see if that works
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
ur a genius
the only annoying thing is now I need to maintain double the websocket connections
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
browser -><- my server -><- runpod
instead of just sending the frames directly between the runpod instance and the browser
in fact the server -><- runpod connection maybe I want to find a better way to do it than websockets. Some messaging queue or something
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
well that's how it works currently
wanna try it out lol
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View
dming the link 1 sec
With GenDJ would it be possible to have it process video from a file rather than live? Take in mp4 return mp4?
it would totally be possible, just haven't built it yet
currently it just turns the frames of your webcam into jpgs and sends them to the server, you'd do the same thing with the video as it plays
I'll take a look at the source, might be able to adapt.
PRs welcome lol
Also, for <- my server -><- runpod -> you are not limited to websocket. If it were me I would have servless connect to my_server via a openvpn. Really easy to setup and automate.
interesting. I tried cloudflare tunnel but couldnt figure it out. hadn't thought of vpn
would that scale? Like many people could be using it at the same time?
and how would I send the frames?
Assuming you don't care about hyper focusing on encryption there is very little overhead. You could send frames via SCP or you could mount a disk from the server... If you want to keep it off disk then your could receive it via a webhook.. really anyway you can on network.
With that said I am in the process of building out a front end that acts as web socker server and client... sometimes it is the best way to go.
tbh might want to flip back to the original i2i-realtime and use zmq https://github.com/kylemcdonald/i2i-realtime/blob/main/worker_app.py
Yes, combine that with openvpn connection and your in business.
RunPod has teased about adding backend networks that serverless can connect to and along with your external servers. Until that happens OpenVPN is is pretty close to that.
quick question: do we pay for pod startup time?
Unknown User•17mo ago
Message Not Public
Sign In & Join Server To View