Runpod•17mo ago

Does the pod hardware differ a lot in US?

Hi,
We deployed several times in US region (secure cloud) with runpod cli, but the inference performance/speed differs a lot, even model loading time differs a lot, what's the reason? and how do I know what data center I'm using. it only shows 'US'.
thanks

lil_seanOP•8/27/24, 3:58 AM

i use lscpu, looks like their cpu are same model, and right now the only difference i can see is Nvidia driver version
GPU is 4090

lil_seanOP•8/27/24, 4:04 AM

the sysbench shows some memory difference

lil_seanOP•8/27/24, 4:06 AM

the top commands shows some cpu difference(the inference was stopped)

lil_seanOP•8/27/24, 4:21 AM

the load average and user time shows big differences, when they have same processes environments (infernece program stopped)

lil_seanOP•8/27/24, 5:44 AM

maybe the difference is because the host VM cpu usage,
i thinks the memory benchmark results proved the performance difference of my programs on two pods

lil_seanOP•8/27/24, 5:51 AM

is it possible that the busy cpu on host instance (maybe from other containers in the same instance), causes high memory contention and thus slow down the memory speed

lil_seanOP•8/27/24, 7:14 AM

Hi runpod, we really need your help, it's severely affecting our inference performance

lil_seanOP•8/27/24, 4:21 PM

i'm not using network volume, i just create pod from runpod cli and pass in arg "US"

Jason•8/27/24, 4:21 PM

and you have another problem where there is a different performance on pods with same specification?

Llil_sean Hi, We deployed several times in US region (secure cloud) with runpod cli, but t...

Jason•8/27/24, 4:21 PM

Oh my bad, i think thats the way it is, the normal UI, not sure how to check it but can't you specify the region when creating using runpod cli?

lil_seanOP•8/27/24, 4:22 PM

but now the point is we see inference performance difference in different pods in US,

lil_seanOP•8/27/24, 4:22 PM

yes, have another problem, but still don't know root cause, i see some memory benchamrk difference on slow and fast pod in US

Jason•8/27/24, 4:23 PM

have you created a ticket for that?

Llil_sean yes, have another problem, but still don't know root cause, i see some memory be...

Jason•8/27/24, 4:23 PM

And maybe you can specify which pod id is slow?

lil_seanOP•8/27/24, 4:23 PM

where to create ticket?

PoddyAPP•8/27/24, 4:23 PM

@lil_xiang

Escalated To Zendesk

The thread has been escalated to Zendesk!

Jason•8/27/24, 4:23 PM

there

lil_seanOP•8/27/24, 4:24 PM

cool，thank！ i'll create one

Llil_sean cool，thank！ i'll create one

Jason•8/28/24, 5:28 AM

i think it has been created check that button

Does the pod hardware differ a lot in US?

Similar Threads

Similar Threads

Similar Threads