31 Replies
Are the bucket and the Fly app in different regions? (Possibly in extremes like Singapore vs Eastern US?)
they're same singapore region
I send request to fly.io, fly.io download a file from R2 (500MB), fly.io throw timeout error (60 secs) before being able to process the file and return processing progress.
upload from an EC2 instance is fine ~80MB/s
Interesting, this is only on Fly? I'm assuming EC2 was also in Singapore, and you are too, is the download speed better when you download the file from R2 on your machine?
ec2 in singapore, too.
download speed to my laptop is slow because of slow wifi.
I can test download from R2 to EC2
Download from R2 to EC2 instance.
Let me try and repro this
Thank you so much, that will help us cutdown paid on fly.io because it's per second billing 👍
Hm so I set up a simple Docker app that just launches
rclone and downloads a file from an APAC R2 bucket on Fly, and I don't see speeds that low 🤔
My Dockerfile is just
How are you making sure that your bucket is in the right region? Did you issue a GetBucketLocation?
Interestingly though, Fly's Grafana says otherwise
I don't know if the Network I/O is only taking "data out" into account though
I presume it is, you might want to look into this. Maybe the speed you're seeing isn't accurate?
Hm or maybe my graph is flawed because the download is happening during the app's "build" phase
Either way, I'd double-check your bucket's location first
My example file was ~550MB, with average 10MB/s, I assume it made up over 60 seconds so that fly.io throw time out error.
I see that Fly's Grafana is not accurate 🤔
I can't find the info on dash, I remember I created it with Singapore region 🙂
Ah yeah regions don't show up on the dash yet, but if you have an S3 SDK set up, you should be able to issue a
GetBucketLocation command to check where the bucket actually is
Currently, I made my fly app to periodically
res.write downloading progress to avoid fly.io timeout error.I've only used Fly occasionally, but is the 60s timeout on the TCP response?
sounds like TCP.
I see
Let me try and deploy my app in a different region to see if the low speeds are a Fly thing or an APAC thing
is 10MB/s downloading considered good speed?
No that's really low
On my local machine I can get up to 75-80 from a bucket in Frankfurt (which is near where I am)
when I download from R2 to EC2, it's 10MB/s too
I can do a second test
2nd test is same
Okay this is more like it (EWR = US East)
So this looks like an APAC problem, probably nothing to do with Fly
it's 8PM at my location and it's rush hour now. I can redo the test later at the morning and at lunch.
Yeah that would be interesting to see as well, but I'll check this out internally as well
redo test at 8:30AM
Redo test at lunch time 0:00PM
Hm, still pretty abysmal