Bucket sharding has been the go-to for a long time in regards to the use-case of lots of concurrency
Bucket sharding has been the go-to for a long time in regards to the use-case of lots of concurrency for a single bucket.
DWhen using aws4fetch, is there any downside to signing the
cf-worker header?
CIs there any way to overcome this, apart from bucket sharding?
ENo, as noted on the ticket, the product needs to be improved
C#3075872
Cyes, simultaneous uploads
Cticket id #3075872, same ticket which you were looking into it.
CI wrote the above, but all I can understand from below reply, we already retrying but its just making it worse for reads too.
EIt's due to limitations and resiliency of the product, but it isn't necessarily "300 concurrent = break", it's more nuanced than that. If you aren't able to work around the issue with retries or bucket sharding then I'm afraid that may mean that the product isn't currently well suited to your use-case
Cwe process almost 1-2 million of files a day, what type sharding can work for this?
EYou would have to split them into different buckets
EAs each bucket is "separate" as far as this issue is concerned
Cyeah, I'm not saying hard limit, but its just my observation, where our application almost unusable.
EUnderstood
ESorry about that, we checked all we could but it's just how it is at the moment
CIs this something that they are currently working on? Is there any public Roadmap people could see to track such things?
Ceven if we tried to shard to lowest possible way for our architecture (per client), it will be more 1200 buckets, but errors coming for small traffic of our actual traffic, we didn't even had a chance to put all traffic.
Calso per account 1000 bucket limit is there
EIt's being worked on but there's no public roadmap or ETA
EIt's not a quick fix, its a longer term project
EYes though you can request an increase if need be
UThey charge per operation instead which will cover the cost of any bandwidth they need to in addition to other things.
UI mean AWS is way more mature so can scale better. If you look up a bit a single bucket on R2 can do about 300ish concurrent. But also AWS just has really high profit margins.
UYou could shard across many buckets instead.
