Runpod

R

Runpod

We're a community of enthusiasts, engineers, and enterprises, all sharing insights on AI, Machine Learning and GPUs!

Join

⚡|serverless

⛅|pods

🔧|api-opensource

📡|instant-clusters

🗂|hub

can't pull image from dockerhub

2024-12-11T12:28:04Z 257642480b4e Extracting [==================================================>] 33.06GB/33.06GB 2024-12-11T12:28:04Z failed to pull image: failed to register layer: archive/tar: invalid tar header @Zeke...

serverless socket.io support

Hello, I want to use socket.io-based serverless endpoint using RunPod serverless. I'm curious if this is possible. When I create a serverless API and connect to that API address via socket.io, which internal instance exactly does it connect to? Because I want to connect to an instance where the queue_length is less than 30 in the serverless instance. However, looking at serverless, this seems impossible. Is this possible?...

Running llama 3.3 70b using vLLM and 160gb network volume

Hi, I want to check if 160 gb is enough for llama 70b and whether I can use use a smaller network volume

I don't know my serverless balance goes down

Hi, I recently made some changes to my platform. It analyzes videos using 3 different computer vision models. I have a serverless endpoint for each. I think that somewhere I am making some requests that I should not be making, or that some endpoints are active when they should not. For example, this happened from 14:20CET to 15:15CET today. I had $85.443 and when I came back I had $83.934. I was doing other tests on my app during this time, but I wasn't calling any endpoint here....

Structure of "job" JSON

I understand that at the very least there is job["id"] and job["input"] and we utilize it. It will help me a great deal if I could send additional information like job["source"] or other metadata to the handler function. It seems like no matter how I structure the JSON, only id and input end up in the job JSON to the handler. Is this indeed the case? ...

Automatic1111 UI with serverless stable diffusion

Hi guys, sorry if this is a noobish question. I want to create a web app that has the automatic1111 sd ui, and then call the serverless API, so I don't have to run a pod continuously. Has anyone done this before? I would really appreciate if someone could point me in the right direction of how to do this....

Serverless github endpoint stuck at uploading phase

First of all i'd like to thank the Runpod team for their amazing work ! 🎉 Although it worked on the initial deployement, I seem to have an issue with subsequent ones. The worker build and deploy correctly but the UI is stuck at the "uploading" phase and the active build doesn't update....
No description

Best Practice for SAAS

I'm new to this. If I wanted to create a SAAS application to offer custom chat for customers using their data. What would be the best practice structure on the Runpod end? Would I have a single enpoint that is shared? Would you setup dedicated pods? ...

Serverless Workers redis client?

Anyone seeing an error like this today? "error": "redis err: client is nil". I'm not using redis in my serverless env, although I am using ComfyUI but I don't think they use it either. Requests are getting hung for a few minutes, eating billing, then failing anyways....

Serverless request returns None from python client but web status says completed successfully

Hello, I have been baffled by this issue for weeks and im pulling my hair out. I have a serverless endpoint that always comes back as None from the python runpod client with no error messages in the logs or from my inference script. Yet the runpod.io metrics for my requests always show has completed. ...
No description

Template id missing in serverless dashboard

Just noticed the template id is completely missing from the endpoints. I have some templates with same docker image tag but different template id so that when I'm testing new images on the test endpoint it doesn't mess up the production endpoint. Can the template id pls be added back like next to docker image?
No description

Disk size when building a github repository as an image on Serverless

I have a question about the disk size when building a github repository as an image on Serverless. Does the option to set the disk size in the serverless settings affect the computer that builds it? For example, if the image being built is about 17GB in size and the computer needs 65GB of storage to build it, should I set the disk size to >17GB or >65GB?

How to get progress updates from Runpod?

Hi all - my goal is to get progress updates from a job request - presently I'm polling a job status request every two seconds, and I would like to get feedback of the % completed. Looking through the documentation, I'm updated the handler function in rp_handler.py by adding the following code: ```...

How can I use Multiprocessing in Serverless ?

Hi I am trying to do something somewhat simple ```def run(self): print("TRAINER: Starting training") train = Train() trainer = self.ctx.Process(target=train.train, args=(self.config.config_path,))...

Can't make serverless endpoints from GHCR container with new Runpod website update

I notice a new UI update released tonight(?) When I go to create a serverless endpoint, I no longer have a choice to im no longer to use images impages on my private GHCR. Is this intentional?...
No description

Can anyone help me deploy a qwen/qwq-32B-Preview model from huggingface with vllm serverless

I'm having issues with configurations. I used 1 gpu of 80gb with container image : runpod/worker-v1-vllm:stable-cuda12.1.0. and had set the dtype as bfloat16, but the model is giving rubbish outputs....

New vllm Serverless interface issue

Hi guys, I logged in early to run my vllm-worker, which have been worker perfectly before, but I noticed that the interface for serverless have changed. I noticed there's no openai compatible url anymore. My codes were also experiencing internal server errors. Would appreciate it if you could share fixes to this issue. I'm not sure if this page is updated according to the new interface: https://docs.runpod.io/serverless/workers/vllm/openai-compatibility
No description

With new pre-built serverless images how do we learn the API schema?

I see we can now select from some pre-built images for serverless. How can we learn the API schema for the input for these pre-built images? Thanks! 🙂

drained of my funds somehow. HELP??

hey guys i dont know who would be able to help me out here but i had set up a serverless endpoint with a custom template. all it does is generate a custom image when the user clicks to generate one. it runs me less than $0.20 a day, usually less. But on one particular day, I was charged me entire account funds ($24) and i truly dont know why that's happened. how could the worker be running all day? how didn't it time out? and also, im pretty sure it wasn't on my end because i have an idle timeout set to 5 minutes maximum so i truly don't know what's going on. can someone help me? attached is the screenshot of average usage + the time i was charged everything: it's funny because the day before, i reloaded funds (Nov 22, $25), and then the next day i was essentially drained of all my funds (Nov 23, a little more than $24)....
No description