Concurrency limits
Ya... the setup is 2 buckets. One for the public stuff (and it is indeed accessed directly on a public domain and optimized appropriately with Cache Rules and Tiered Cache). That's for things intended to be public (things like user avatars). The private bucket that is accessed via the API is the one he's having a problem with. The objects there are accessed via the API because the ability to view/download those are user permission based. The application checks for the appropriate permissions for the user that's logged in and then passed it through as necessary based on the permissions. He isn't doing a crazy amount of traffic or anything... he's at ~19M class B operations this month across all his buckets.
The issue here is with the private bucket... since it's user permission based, the object is passed through via API. It's no where remotely close to 1,000 concurrent read operations even across all objects/users, I'd guess maybe it peaks around 10ish.
6 Replies
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View
Was most definitely 429. The error (and JSON) coming back from Cloudflare was:
429 Too Many Requests: {"Code":"ServiceUnavailable","Message":"Reduce your rate of simultaneous reads on the same object."}
Now that I know Durable Objects are used on the backend for metadata requests and that Durable Objects are single-threaded, it made more sense that I avoid metadata requests than try to figure out why a few concurrent metadata requests were failing.
Given that, I ended up rolling out a change to our users that avoid the metadata request completely (it wasn't something mission critical, rather a sanity check for the abstracted filesystem):
https://xenforo.com/community/resources/app-for-cloudflare%C2%AE.8750/update/44961/
Made change to XenForo's attachment data entity to be more efficient (normally XenForo checks if an attachment exists before making an additional call to actually get it). This will reduce an API call for every attachment view because we don't need to check if the attachment exists (we know it does already because we have a record of it in attachment data).
XenForo community
App for Cloudflare® - Updates
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View
One of the reports I received from a user had this date/time from their error log:
ErrorException: Cloudflare: Client error:The date/time is going to be in local time of the user viewing it (which I don't know off the top of my head). I could ask them if it would help.HEAD https://id-xxx.r2.cloudflarestorage.com/attachments/attachments/1159/1159937-3c44ba1e1c54d0114d1f646babd887c8.dataresulted in a429 Too Many Requestsresponse / null src/XF/Error.php:77 Generated by: Marc Aug 23, 2023 at 2:51 AM {"Code":"ServiceUnavailable","Message":"Reduce your rate of simultaneous reads on the same object."}
Unknown User•10mo ago
Message Not Public
Sign In & Join Server To View
@Frederik, I've had this happen during a database diff backup on 2023-10-20 06:30:11.164 CEST.
The backup lasted around 10 seconds and there is no other traffic to the bucket. According to GraphQL stats, before 429 was returned, there had been 14 list object, 7 put object, 12 get object and 11 head object requests.