Hey we tested R2 uploads with same
Hey, we tested R2 uploads with same instance compared to S3 and DO Spaces, R2 seems significantly slower (about 500ms to S3 and DO's 100ms) is there any way to get around this?
20 Replies
Buckets are currently created wherever you create them from. It sounds like your bucket might have ended up farther away from where you're running your tests from, than S3/DO. Are you testing locally? Can you can give me your account ID and bucket name?
I should mention that the big idea around automatic placement has to do with intelligent relocation based on traffic, but till that happens, this is admittedly a little weird.
wherever I create them from?
as in based on my IP?
9363b1441db5d5b6f035de27c431630e
account id
bucket name is stablecog-dev
however, the difference is rather hard to explain by distance
given how fast the light travels 🙂
R2 regularly gets 1 second uploads in my tests
S3, DO Spaces, CoreWeave Object Storage, nothing else I tried comes even close to that long
If it was doing 200-300ms I would assume it's because of the region difference
But 500ms-1000ms, is a bit hard to explain by region
I'm not testing locally btw, I tested with 2 different machines with 2Gbit uploads
and 10Gbit downloads
DO Spaces and S3 regularly hits 35-50ms for the same upload, that could be explained by region (as opposed to say 150ms). But 500ms makes me think Cloudflare is throttling it
and 1000ms, idk what's up with that, these are 100kb files
Actually CoreWeave's storage solution also hits 50ms, though I didn't see 35ms with that one
Also, isn't there a way to force us-east on the bucket? Am I supposed to use a VPN or something when creating the bucket?
Oh no, whichever Cloudflare datacenter your CreateBucket request hits is what I meant.
It looks like your bucket is in Warsaw.
Region hints are currently in the works, that should work if you want to create a bucket in a specific region
so currently there is no way whatsoever for me to force a bucket in USA?
Warsaw still doesn't explain the 500-1000ms times though
Yeah agree
Sorry I'm just looking at some things to maybe figure out what's going on
Any updates?
Is there anyway to see where our bucket resides at btw, from our end. We'll try to create a bucket in the us-east region but not sure how we can confirm it's in that region after creation
GetBucketLocation is implemented in the S3 API, that should help.
Unfortunately there are some implementation details that take the bucket name as a seed when choosing its location, which essentially means that once a bucket "chooses" a location, deleting it and creating a new bucket with the same name will always end up in that location. (I don't really know how to make this sound less weird without explaining internals, sorry!)
This is all far from ideal, and is on our radar to fix as soon as it becomes possible, but I don't really have a good story for region-specifics right now.
@frederik_b looked into the 500-1000ms latency you're seeing and there is work going on that might make it slightly more bearable in the short term, but that number is still unreasonably/abnormally high
@sdnts itd be great to hear when those improvements are implemented. i’m basically waiting until i see some of these performance issues get resolved before i give R2 a real try
this is what get bucket location gives me
this all sounds weird indeed. We want to use R2, but we simply can't because of these problems. Auto sounds good in theory but in practice we basically need to create and delete buckets until we get one in the region we want
instead of just picking it 😄
I understand, yeah. Lots of improvements to be made for better regional latency. When region hints go out and start working reliably and the way people expect, we'll likely make noise here, would be great to have you reconsider then! I'm hoping to get that working as soon as possible!
cc @phillip
so is GetBucketLocation supposed to work?
for me to see where my bucket is?
or is there any way whatsoever for me to see it?
because I really would like to know where the bucket it 🙂
Ohh right it's not accurate for you because we haven't rolled out region hints to everyone yet. Sorry this was a bad suggestion from me then.
or is there any way whatsoever for me to see it?The answer right now is No then
can you roll it for us 😅?
as in is that possible
we really need this
if there is a beta program of some sort I mean
To be transparent, it doesn't work reliably yet (like I said, it is a WIP), but Phillip (our PM) can add you to the list of people we'll reach out once it starts working!
that would be nice, should I send him a DM?
You can! I've tagged him and DMed him internally as well
thank you, I dmed him
Just to clarify on this, the latency seemed abnormally high even for a bucket that was located far away, correct?
so it's unlikely that region hints alone would resolve the latency problem?
Correct
Region hints by themselves wouldn’t solve it, I’m just looking for as much info as I can get