we went ahead and reverted to using the public bucket URL but this should obviously not be happening

we went ahead and reverted to using the public bucket URL but this should obviously not be happening to people who use R2 as their domain. It literally tells you that you can take advantage of CF's caching features by linking a domain

JoshverdOP•2/14/23, 3:04 AM

guess I'll send a ticket Sadge

JJoshverd we went ahead and reverted to using the public bucket URL but this should obviou...

Cyb3r-Jak3•2/14/23, 3:07 AM

Let me know the ticket ID and I can escalate it

Kkagitac That might be how `r2.dev` zone is handling the request. @sdnts any idea there?

Sid•2/14/23, 3:19 AM

I’ll check this out! I can’t tell what the issue is off the top of my head, but this does look like a zone config issue.

Sid•2/14/23, 3:24 AM

Yeah internally they're ~the same thing

Karew•2/14/23, 3:26 AM

This is tangental, but: What exactly is r2.dev supposed to be for? It's not very useful currently because of the rate limits.

SSid Yeah internally they're ~the same thing 😄

kian•2/14/23, 3:27 AM

Is it a Worker the same as the S3 API? Some of the symptoms make me think of encodeBodyencodeBody

Karew•2/14/23, 3:31 AM

Right, so wouldn't you just want to create something like subdomain-dev.example.comsubdomain-dev.example.com for your R2 bucket testing instead of dealing with r2.dev?

Karew•2/14/23, 3:32 AM

So you can test all that stuff

Kkian Is it a Worker the same as the S3 API? Some of the symptoms make me think of `en...

Sid•2/14/23, 3:32 AM

It all comes down to R2's worker! The fact that this works fine with the S3 API is what makes me think this might be zone-config related.

KKarew Right, so wouldn't you just want to create something like `subdomain-dev.example...

Sid•2/14/23, 3:33 AM

TBH that's what I would do as well. The reason we rate-limit r2.dev is because we cannot cache things on there (strong consistency requirements + we don't have that kind of control)

Karew•2/14/23, 3:34 AM

It's me, I have production code that works around the problem

SSid It all comes down to R2's worker! The fact that this works fine with the S3 API ...

kian•2/14/23, 3:36 AM

Fair enough - I guess there's some secret toggles that the rest of us don't have on our zones haha.

It just reminded me of encodeBodyencodeBody since if you omit that when sending a response that has content-encoding: gzipcontent-encoding: gzip then you run into the same double compression issue

Kkian Fair enough - I guess there's some secret toggles that the rest of us don't have...

Sid•2/14/23, 3:44 AM

It could still be that. Honestly I’m also just guessing, it’s not immediately obvious to me what the issue might be without digging deeper, but I’ve asked our oncall to take a look.

KKarew It's me, I have production code that works around the problem

Sid•2/14/23, 3:44 AM

Well then. I’ll make sure we take this kind of stuff into account then!

Karew•2/14/23, 3:46 AM

I'm happy to help test any changes, and my full issue is documented here https://gist.github.com/csuhta/0001d1bb74200412bc1d7f9e11ec4ea5

Karew•2/14/23, 3:46 AM

The workaround I mentioned was to mostly just stop pre-compressing stuff

kian•2/14/23, 3:46 AM

gzip my beloved

Unsmart•2/14/23, 4:44 AM

afaik there is no rate limits

kian•2/14/23, 4:46 AM

There’s limits but none that you’d hit with a 30s delay between Ops

Unsmart•2/14/23, 5:40 AM

Interesting didn't think concurrent request rate was a 429 but I guess that makes sense... But you would only have 1 concurrent request if you only did it once every 30s so I'm guessing you accidentally have some recursion going on

JJoshverd we went ahead and reverted to using the public bucket URL but this should obviou...

Erisa•2/14/23, 9:02 AM

You can send me the ticket ID and I'll look at it

digitalpoint•2/14/23, 5:17 PM

There's sometimes a reporting delay on usage. Did you just delete everything in the bucket? It's usually fairly realtime, but I feel like there are times where it's an hour or two delayed.

kian•2/14/23, 5:18 PM

the only culprit I’m aware of is multipart uploads which zeblote mentioned

Bbee i did not check if multi-part deleted stuff, i assumed it hadn't because the gb ...

kian•2/14/23, 5:19 PM

there’s an rclone command to list them for a bucket - if that returns nothing then it won’t be those

kian•2/14/23, 5:19 PM

digitalpoint•2/14/23, 5:21 PM

Anecdotally, it feels like when a bucket has no usage (not doing any reads or writes of objects) is when it seems to be most prevalent to usage stat lag... maybe the action of reading/writing kicks off a backend process to update usage or something. At one point I started to dig into it with GraphQL requests to see when usage records were being compiled for the bucket (can see the time of the usage log), but by the time I built it, it had already worked itself out, so...