TBH that's what I would do as well. The reason we rate-limit r2.dev is because _we_ cannot cache thi

TBH that's what I would do as well. The reason we rate-limit r2.dev is because we cannot cache things on there (strong consistency requirements + we don't have that kind of control)

Karew•2/14/23, 3:34 AM

It's me, I have production code that works around the problem

SSid It all comes down to R2's worker! The fact that this works fine with the S3 API ...

kian•2/14/23, 3:36 AM

Fair enough - I guess there's some secret toggles that the rest of us don't have on our zones haha.

It just reminded me of encodeBodyencodeBody since if you omit that when sending a response that has content-encoding: gzipcontent-encoding: gzip then you run into the same double compression issue

Kkian Fair enough - I guess there's some secret toggles that the rest of us don't have...

SidOP•2/14/23, 3:44 AM

It could still be that. Honestly I’m also just guessing, it’s not immediately obvious to me what the issue might be without digging deeper, but I’ve asked our oncall to take a look.

KKarew It's me, I have production code that works around the problem

SidOP•2/14/23, 3:44 AM

Well then. I’ll make sure we take this kind of stuff into account then!

Karew•2/14/23, 3:46 AM

I'm happy to help test any changes, and my full issue is documented here https://gist.github.com/csuhta/0001d1bb74200412bc1d7f9e11ec4ea5

Karew•2/14/23, 3:46 AM

The workaround I mentioned was to mostly just stop pre-compressing stuff

kian•2/14/23, 3:46 AM

gzip my beloved

Unsmart•2/14/23, 4:44 AM

afaik there is no rate limits

kian•2/14/23, 4:46 AM

There’s limits but none that you’d hit with a 30s delay between Ops

Unsmart•2/14/23, 5:40 AM

Interesting didn't think concurrent request rate was a 429 but I guess that makes sense... But you would only have 1 concurrent request if you only did it once every 30s so I'm guessing you accidentally have some recursion going on

JJoshverd we went ahead and reverted to using the public bucket URL but this should obviou...

Erisa•2/14/23, 9:02 AM

You can send me the ticket ID and I'll look at it

digitalpoint•2/14/23, 5:17 PM

There's sometimes a reporting delay on usage. Did you just delete everything in the bucket? It's usually fairly realtime, but I feel like there are times where it's an hour or two delayed.

kian•2/14/23, 5:18 PM

the only culprit I’m aware of is multipart uploads which zeblote mentioned

Bbee i did not check if multi-part deleted stuff, i assumed it hadn't because the gb ...

kian•2/14/23, 5:19 PM

there’s an rclone command to list them for a bucket - if that returns nothing then it won’t be those

kian•2/14/23, 5:19 PM

digitalpoint•2/14/23, 5:21 PM

Anecdotally, it feels like when a bucket has no usage (not doing any reads or writes of objects) is when it seems to be most prevalent to usage stat lag... maybe the action of reading/writing kicks off a backend process to update usage or something. At one point I started to dig into it with GraphQL requests to see when usage records were being compiled for the bucket (can see the time of the usage log), but by the time I built it, it had already worked itself out, so...

kian•2/14/23, 11:30 PM

kian•2/14/23, 11:33 PM

sooooooon

kian•2/14/23, 11:34 PM

eventually you can specify a region when making the bucket - i.e in the S3 API - but that isn't here yet

kian•2/14/23, 11:34 PM

i see docs PRs for it though so i expect it comes sooner rather than later

Erisa•2/15/23, 12:28 AM

Silly question on my part really but, does the bucket still exist?

Erisa•2/15/23, 1:16 AM

If you go to the bucket can you remove it that way?

Erisa•2/15/23, 1:20 AM

I mean remove the domain

Erisa•2/15/23, 1:37 AM

oh dear

Erisa•2/15/23, 1:39 AM

if any member of the R2 team happens to pass through they can probably look at the bucket, otherwise if you are able to create a support ticket then i can escalate it in the morning

Unsmart•2/15/23, 1:44 AM

Yeah thats a known issue if you dont delete the domain before deleting the bucket you get stuck with the dns

Unsmart•2/15/23, 1:44 AM

oh wait this seems a bit more weird lol

TBH that's what I would do as well. The reason we rate-limit r2.dev is because _we_ cannot cache thi

Similar Threads

Similar Threads

Similar Threads