Urgent! So slow Download Speeds (Both Secure / Community Clouds)
Hi, I am using sd web ui, and when I generate an image from UI or API, I can see that thestable diffusion inference is so fast (1.5 sec) but after it finished it literally just loads the output image for 5-10 seconds at 300kb download speed. (Output image sizes are very low like 512x768)
32 Replies
Sorry for tagging, but we are having huge issues on our production and we need to resolve this issue hopefully @Madiator2011 @Justin
We are running 2x4090 (community), and 1xA100 80GB (secure cloud), and both have the same network speed issue
Is there any chance that the runpod proxy urls are throttled ?
Please post the pod IDs
2x4090:
clxu7lem3ph9xu
1xA100: 4hp88h35ympavi
May someone really look into this? We have moved our all infrastructure to Runpod for our production after seeing runpod is great for us and right now if you guys can help us figure this out, would very appreciate it
Please let me know if there is anything we should do from our sideLooking into this now
Thank you!!
Is the service running on pod
4hp88h35ympavi? What are you doing that you are observing slow download?And if you generate an image, the generated image will show up after 5 seconds later
Normally we were directly using the sd web ui api, but we realized that the request times are so high and I went checking directly on web ui, and once the inference is complete, I can see the download progress of the output image
Are you downloading into the pod or uploading from the pod?
sd-web-ui is downloading from the pod. I can see that everything is so fast inside the pod, e.g when image generation completes, it directly saves the image in pod, and after that web ui tries to display that saved image from pod
https://clxu7lem3ph9xu-4003.proxy.runpod.net/file=/tmp/gradio/tmplrsew8f0.png
https://4hp88h35ympavi-3001.proxy.runpod.net/file=/workspace/sd/stable-diffusion-webui/outputs/txt2img-images/2023-12-31/00005-3006222233.png
Example image urls that it is trying to download from pod
And this is for example taking 5-10 seconds which is so weird
@Justin You can also reproduce easily by just paste above image url in the browser and set disable cache true so you can experience the slow download speedso this the the speed of downloading to your local pc?
We can say that as I am running the sd web ui on my browser.
But local pc does not matter here I think. We can see that this download speed from pod which is very slow wherever run it.
I still mostly think Runpod proxy layer is throttling the speed and we cannot reach the download speed that the pod actually has and shown in the dashboard.
I just need to confirm when you say "download speed" is that mean the speed to get something out of the pod or something into the pod
I mean "get something out of the pod".
May you quickly perform a speed test on your own PC? And have you ran a speed test in the pod yet?
Yes here are the results
This is Pod
-- And this is my local
Thank you for this information, looking into this further
A question on from my side: I saw in the support channels that more users experienced the similar issues related to this network/download speed issues, and your team mostly suggested to use the cli to download/upload things. I am not sure why this issue was ignored previously even though it was reported multiple times
There are many factors that go into network speeds which make it difficult to troubleshoot. The point to point path can have outages along the route that cause delays, often times the source of the slow connection is outside of our control. For your case I am looking to see if there is anything that can be done on our side
I see, but again it is still weird that it is happening on both secure and community cloud
So that is my point why I am trying to always remind it
but all good, lets see if you can figure out something
Hope we can find the issue at least.
Hmm @Justin I just tested something, and I can tell that the issue is 100% on runpod proxy urls.
Just to test and see what will happen if I don't use Runpods proxy url and if I tunnel it myself to public url, and I just tunneled the port from pod by using cloudflared https://developers.cloudflare.com/cloudflare-one/connections/connect-networks/do-more-with-tunnels/trycloudflare/
And when I use this new manually tunneled url independent from runpods proxy urls, and now this url works perfectly, and there is no slowness accessing the images or resources using this url
-- I hope this can help you find the issue on runpods proxy urls
@xPaghkman We went ahead and re-booted our proxy server if you would like to give it a try now
having the same issue. Networking in general is really slow compared to what it used to be 3 days ago
uploading images to a1111 takes forever, to the point that im not sure if it failed and the gradio ui is just not respoding (more than 15 minutes)
browsing images stored in the pod, feels like browsing the internet with a phone line in the 80s. you can count the rows of pizels being drawn
im working on the RO data center, secure cloud
Pretty sure you mean the 90s, I don't think the internet was publicly accessible in the 80's and people still relied on things like BBS, Fidonet etc 😁
im from Argentina. i didnt had internet until the 2000, and at that time, it was dialup. but yeah, whatever decade it was
@Brolios Have you tried the way I am doing above ? manual tunnelling to the local server port ?
I think we gonna have to do like that until issue gets resolved by runpod
yes, and that is faster
but the slowdown makes the applications un usable
i generate images with A1111 web gui, and then i want to look at the generated image. It takes longer for the image to load, than to generate a batch of 10
@Justin @Madiator2011 Do we have any update on this issue? Looks like repooting proxy server did not work, and we are using our own tunneled urls for now.
oh, now that i read it, no im not using my own tunnel.
will need to, otherwise im left hanging wothout a server
You can just do it easily, just few commands, follow cloudflares doc, and it just tunnels your local port to new url.
Otherwise pods are not usable with this connection speeds on runpods proxy urls
@Justin @Madiator2011 -- Is there any update on this? Feels like this critical issue is getting forgotten again. Because of the slowness of everything that is using runpod proxy servers, I had to also tunnel notebook ports, and even I tunneled a dedicated terminal access to Pod, because proxy connections by runpods are not usable with that speed.
I hope you guys can fix the issue and we would not end up writing a library for the community that manually exposes any port from the pod.
@xPaghkman So I deployed A100 pod on secure cloud of auto1111 and with proxy link first connection to web ui took 4 seconds to load UI and images download are fast without issue.
Though If you could share a little bit more info I will see if I can replicate your issue.
What is the region of the pods: (Secure) (Community)
What is exactly full template name you are using:
I expierence the same issue. It was fast until the pod was running for about 6 hours. Using --share on relauncher.py and using the gradio link works, but is not as fast as the runpod proxies used to be in the past.
worked for a few days and now its happening again
started 1 or 2 hours ago
trying the cloudfare one