Runpod•13mo ago

Throttled download speed from container registry while still being billed

Hello, Im trying to run a custom image from ghcr on a pod, which ive done many times. Its taking a long time, likely going to be 45 mins total just to download the image. I'm also noticing that im being billed while I wait for this very slow download speed.

Is this intentional? If so it seems like a conflict of interests between runpod and the users.

yhlong00000•12/20/24, 9:32 PM

It might be worth trying a different container registry or switching to a different data center to see if you get better speeds.

Ben Lau•12/21/24, 9:56 AM

What is the region chosen? I encountered a similar problem using EU-RO-1

yhlong00000•12/21/24, 2:10 PM

The network is complex, multiple factors from source to destination, the load on data center links, the location of your image registry and its download speed capabilities, and the routing path between your image registry and our data centers. Since our data centers are located in different countries, the routing path’s congestion and other variables also play a significant role.

enfys•12/21/24, 4:49 PM

I'm getting the same, this "still fetching" while being billed - CA-MTL-1 datacenter.

Ben Lau•12/21/24, 5:48 PM

@yhlong00000 In fact, the speed to download model from huggingface and civitai is acceptable... I wonder, can we pay extra storage fee to hold docker image in the network volume.

Eenfys I'm getting the same, this "still fetching" while being billed - CA-MTL-1 datace...

Jason•12/22/24, 2:32 PM

How long it stuck like that , try to create a new pod

Jason•12/22/24, 2:33 PM

if its all like that, try reporting this in a ticket

BBen Lau @yhlong00000 In fact, the speed to download model from huggingface and civitai i...

Jason•12/22/24, 2:34 PM

thats kind of like docker registry, currently not possible, but i think they are planning to make something like this

Yyhlong00000 The network is complex, multiple factors from source to destination, the load on...

nevermind•12/22/24, 3:28 PM

Mb Runpod needs to host their own registry? If you ask me, we would pay an extra per hour just to launch pods faster (or mb buy some premium for an extra XX$). Other providers give such opportunity, and it seems Runpod experiences lack of this feature

Jason•12/22/24, 3:33 PM

Yeah I guess they do, yes runpod hasnt supported something like this I guess but if you wanna try to do your own I guess you can launch a cpu pods in the same region then run your own registry

Jason•12/22/24, 3:33 PM

I'm not sure it going to work well with that method tho

JJason Yeah I guess they do, yes runpod hasnt supported something like this I guess but...

Ben Lau•12/22/24, 7:58 PM

Interesting idea. The user may first start a CPU pod to provide the image and shut it down when the GPU pod is started. (I think Runpod will not be happy with this situation.)

However, can it set a fixed domain name for the CPU pod? If not, then the user needs to check the domain name of the CPU pod and replace the Docker image URL for the GPU pod.

Jason•12/22/24, 11:44 PM

Yes you can set a fixed domain name by using like cloudflare tunnels

Throttled download speed from container registry while still being billed

Similar Threads

Similar Threads

Similar Threads