Speedtest for slow pod
As discussed in the last slow download speed thread I've run the speed test script from @justin to maybe help you guys have a look what's going on. I am most often having issues with uploading, as the speed always falls below 1 mbit. I've noticed that restarting the upload (I'm normally using runpodctl btw.) does help, but most often for maybe a minute. Right now I'm trying SFTP, it does help a bit and is around 3-4 MiB/s for uploading. Runpodctl upload was again below 1 MiB/s.
Screenshot attached from Pod overview, showing Community Cloud & CZ region.
Summary of speed test script attached as txt.
Solution:Jump to solution
It's not impossible, but that would be extremely unlikely to be honest; I haven't seen that happening in the past.
- On Secure Cloud: the speed is symmetrical per server. So other servers usage do not affect the bandwidth of others.
- On Community Cloud: often, there is a backbone for all servers at a specific data center. That means higher utilization of the network has an impact on all servers all at once....
39 Replies
Honestly I wonder if the problem is runpodctl ποΈ this sounds like. Are u uploading from one pod to another? Or? Uploading from local to pod?
Using runpodctl^?
Local to pod, usually using runpodctl. Right now I'm using sftp
Lol i was like thinking is the problem going to be upload speed π bc that the only thing i dont test, but thanks for sharing all the other info
Ok i think
runpodctl sucks LOL
Personally i think runpod could look into that
sftp is goodβ¦
π yea I missed the upload speed test, but anyways, it is also an issue with the download
well ur download speeds look fine
Oh wait
lol
ur download sucks
I expected more or do you say it is due to the download source
ah ok
π
the thing is lately (since a couple of days) this is an issue with all the pods I use. Usually it was only casually with some of them
U only ran the download tests? not the speed cli?
and I'm already filtering for high or extreme
@Finley Hopefully can take a look tomorrow
ποΈ already helpful hahaha. but yea
ur discrepancy looks big
across all sources
to what is stated
I just asked chatgpt to convert all ur bytes to Mbps per second and seems off
I only ran
bash speedtesth.sh - is there more? didn't take that much time to look into it πAh haha nah that should be it surprised it didnt have more ontop
i thought should have had a soeed test on top
and not just download
could be my phone too
and i cant see the text properly
Do you mind if u run it one more time lol
i mean ik is already good / if u didnt release it yet
but yea, u are dropping down to 50 mbps if chatgpt conversion is right at some point
that def is weird ποΈ
sure let me run it again. doesnt hurt
yea im hoping there just more of the text file xD i just tested this so im like im sure there should be more
was adding a max time out
so it doesnt run forever
in case a scenario like urs happen lol
ah so I need to redownload
ah u can just recopy and paste it
or redown the sh
yea
ποΈ
it does clean up afterwards, right?
Yup
it downloads to a null
so no files stays saved on ur system
ah: 0 upgraded, 0 newly installed, 0 to remove and 8 not upgraded.
speedtest.sh: line 165: speedtest-cli: command not found
is in the output
due to:
Unpacking speedtest-cli (2.1.3-2) ...
dpkg: error processing archive /var/cache/apt/archives/speedtest-cli_2.1.3-2_all.deb (--unpack):
trying to overwrite '/usr/bin/speedtest', which is also in package speedtest 1.2.0.84-1.ea6b6773cf
Errors were encountered while processing:
/var/cache/apt/archives/speedtest-cli_2.1.3-2_all.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)
huh
can u do apt-get install speedtest-cli
damn it
lol
i thought i had that installed auto
same error
Huh surprised nvm the
nvm then
everything else works
Chatgpt says ur download speeds are:
Here are the download speeds converted from bytes per second to Mbps, kept in the same order as they were listed in the file:
1. 321.51 Mbps
2. 585.76 Mbps
3. 49.17 Mbps
4. 24.69 Mbps
5. 13.42 Mbps
6. 24.69 Mbps
7. 16.96 Mbps
So def a large drop off
S3 bucket and the broadband is matching about - so that is two sources so might be an issue. considering we see a progressive drop off of speed as more files are tested that is also weird
Anyways yea maybe @JM / @Finley can also look tmmrw. But that is weird. U start high but ur speed decreases o.o so drastically
huh chatgpt says u have speedtest already
wonder why keeps hitting this tho cause my script chrcks it oh well
guess everything else runs
π’ so close to a good script
so close so far
u also encounter this on
secure tho?
ive just tested on 5 secure pods
when writing the script
didn't try
i see
maybe will serve u better π€·
i dont try community too much, ik secure bit more expensive
but idk bit more consistent
but yea hopefully a staff can help check ur network out or just know if this a bad pod
might have not been caught bc maybe π€ something weird happening with continous loads
glad i added a stress test portion with the s3 cause that was my theory why some these are so inconsistent repeats vs the speedtest cli
anyways sorry for the spam off to sleep in east coast
edit: fixed my installation step ;D, had some messed up stuff about it but should hopefully work in the future. tested on community / secure fresh installs. was also getting install errors so wanted to fix it
no worries, thanks for support π
Hey @bennyx0x0x
I tested your pod; I am clocking at 798mbps download speed
That's solid for community cloud. Did you try uploading your local data to other storage services to compare? Also, accuracy of speed tests inside the pods are very highly dependent on selecting a speedtest server that makes sense. Often, the default one is very far away and not the fastest one.
Just wondering π€ so if someone like Benny got different download speeds and there is significant drop off but other pods in the same region do consistent speeds is that indicating a problem with a pod or a source?
Like if i launch two community cloud pods in the same region - but one like Bennyβs slow down on two of the download tests but the other stays the same. Or is that still too hard to tell
Or is it that if a pod has good speed relatively most the time and drops off in another testing
it prob the pod is bad with servers in that area?
since pod ingress/egress wouldnt change
Solution
It's not impossible, but that would be extremely unlikely to be honest; I haven't seen that happening in the past.
- On Secure Cloud: the speed is symmetrical per server. So other servers usage do not affect the bandwidth of others.
- On Community Cloud: often, there is a backbone for all servers at a specific data center. That means higher utilization of the network has an impact on all servers all at once.
The second bullet point make it so you might be seeing more variations on community though!
Oh interesting. So the latter means heavy downloads could affect other community clouds? So a secure cloud server would be the most stable?
I just find it interesting benny testing from the document went 500mb, to 300 mb, down to 30 mb range (s3 and broadband)
Interesting π€
Heavy downloads from servers at the same location*
Interesting - okay so a community cloud could be sharing network bandwidth while secure cloud doesnt
Community Cloud is a way for RunPod to give back to the community that brought us to life, offering cheaper alternatives. Obviously though, community hosts might not have as much funds to buy switches and routers that cost $2M+ each haha
yep, the upload elsewhere is fine and stable. I also noticed that restarting the upload (same with downloads), it first fully utilizing my bandwith, but then it starts to drop. I always had this mixed results in community, but lately its pretty often. That's why I always thought it's due to the shared usage and other users who massively use the bandwith
That's possible, but shouldn't happens 100% of the time. We don't see 24/7 bandwidth usage, especially on community π
I guess, just wondering, if you ran a test like my bash script on secure cloud and community cloud:
If community cloud was stable at first and began to drop off in the latter test, would that indicate some sort of rate limit on the community pod past a certain level? Or is that just like bad luck and potentially bandwidth issues due to sharing the server infras.
Or if secure cloud, let's say for 4/6 of the tests was good, 2/6 tests are bad (but not just at the end of the testing), would that more likely indicate that the 2/6 tests the servers are just bad - but the pod itself never slows down incoming data?
Interesting though, I should add a way to check the region / distance of the downloads π€
Networking issues are definitely interesting, well learnt something new today about the pods in a data center vs community cloud for networking π€, never knew about the potential for shared bandwidth thing but that makes sense.
Well thank you JM, certaintly interesting.
(I guess the main thing is that unless there consistent result across the board that something is just doing so badly) then it could be a source of other issues lol
yep thanks for your time @JM
Of course π
For your first question, most likely bandwitdh saturation, correct. For #2, it m-i-g-h-t be bandwidth saturation, but 100 times less likely, as bandwidth is capped per server, and backbones are often very very fast (like 25gbps or 100gbps)
Good to know haha
Thank you~ learnt a lot