It's something that was already on my backlog but there's a bunch of technical and non-technical hea

It's something that was already on my backlog but there's a bunch of technical and non-technical headwinds. I don't think that's something that we can support any time soon. The main obstacle is that all entrypoints into R2 require that you specify the size of the object you're giving us (which AFAIK isn't different from any other object storage system). Practically this means you'll need to buffer the compressed file in memory (so that the length becomes known) which then starts having a limit to the size of file you can compress. You can kind of do this via a Worker binding by using

CompressionStream

CompressionStream

CompressionStream

CompressionStream but you'll have to handle what happens if you have too many concurrent requests at once or a poorly compressible file and suddenly you're potentially hitting the Workers memory limit.

I want to evolve R2 here to be much more flexible but that feature is definitely a couple of years out if not longer.

Bbeelzabub It's something that was already on my backlog but there's a bunch of technical a...

John Spurlock•3/18/22, 5:21 PM

Oh I meant on the way out of R2, not on the way in. Assuming there will be some sort of public read equiv for R2's GET Object call, let's say from a browser, it would be ideal if R2 compressed the response sent to the client based on what the client supports (in accept-encodingaccept-encoding)

Epailes•3/18/22, 5:36 PM

How does the 500MB file limit if bound to workers function if the workers themselves have max 128MB limit? (Shared between all instances of that worker in a particular datacentre as well, so if there's 10 running they should all be 12.8MB or less really).

EEpailes How does the 500MB file limit if bound to workers function if the workers themse...

zegevlier•3/18/22, 7:43 PM

That's the first time I've heard that, where did you get that from?

Edit: Just to clarify, the multiple workers instances sharing memory bit.

kian•3/18/22, 8:18 PM

EEpailes How does the 500MB file limit if bound to workers function if the workers themse...

Isaac McFadyen•3/18/22, 9:11 PM

That's for RAM, files can be streamed as long as you don't do certain methods on them.

Isaac McFadyen•3/18/22, 9:12 PM

workers-and-pages-discussions

IIsaac McFadyen That's for RAM, files can be streamed as long as you don't do certain methods on...

Epailes•3/18/22, 10:13 PM

Ah that's good to hear, thanks for linking that

Zzegevlier That's the first time I've heard that, where did you get that from? Edit: Just ...

Epailes•3/18/22, 10:24 PM

As Isaac mentioned I was thinking about memory so it's not relevant to R2, but if it's workers in general see: https://developers.cloudflare.com/workers/platform/limits/#memory
If you want to chat about it more it's probably worth moving to the #workers-discussions channel

Zzegevlier That's the first time I've heard that, where did you get that from? Edit: Just ...

Isaac McFadyen•3/18/22, 10:26 PM

Yeah that is accurate about the memory. It's not specific to R2 but Workers can share memory if they're called from the same DC and at the same time.

beelzabubOP•3/20/22, 4:05 PM

The 500 MB limit for request body is documented here: https://developers.cloudflare.com/workers/platform/limits/#request-limits
The 128 MB per Worker instance is documented https://developers.cloudflare.com/workers/platform/limits/#memory but it's listed in many places throughout.

The architecture about isolates and how each one processes requests is described at https://developers.cloudflare.com/workers/learning/how-workers-works/

Workers can definitely process requests that are > 128 MB of body. The thing to watch out for is that your instantaneous memory usage allocated per request is kept bounded so that your isolate doesn't exceed the 128 MB limit if you handle a lot of requests simultaneously. The memory we'll see a worker is using is going to be approximately # of outstanding requests for that worker * max memory allocated per request# of outstanding requests for that worker * max memory allocated per request.

andrew•3/24/22, 2:30 PM

very curious to find out what kind of bandwidth can be achieved pulling data from multiple R2 objects simultaneously into a non-worker destination

beelzabubOP•3/25/22, 5:00 PM

What's the speed you get from speed.cloudflare.com?

MyZeD•3/25/22, 7:15 PM

749Mbps / 31.1Mbps / 20.4ms / 3.71ms

MyZeD•3/25/22, 7:17 PM

Why i dont get the next destination?

MMyZeD Why i dont get the next destination?

Isaac McFadyen•3/25/22, 7:18 PM

Depends on your ISP and how they peer with Cloudflare. Some ISPs have strange peering... I get routed to a colo 800km further than my closest one

IIsaac McFadyen Depends on your ISP and how they peer with Cloudflare. Some ISPs have strange pe...

MyZeD•3/25/22, 7:18 PM

wtf

beelzabubOP•3/25/22, 9:52 PM

Theoretically pulling 1 file should be able to saturate your available link to Cloudflare but we haven't profiled it yet so

. Pulling multiple files concurrently should be able to saturate that. This assumes large files (for some definition of large) where the latency of the request is insignificant. For small files (for some definition of small) you'll need to have a LOT of concurrent requests going. It's hard to predict arbitrary performance questions. My best advice is to wait till you have access/Open Beta & try it out? Happy to discuss specific scenarios.

ben•3/25/22, 10:36 PM

Will R2 support moving/renaming files and folders? Right now in S3, you have to copy and delete each file.

Bben Will R2 support moving/renaming files and folders? Right now in S3, you have to ...

Isaac McFadyen•3/25/22, 11:38 PM

You're correct about S3 but they abstract it away in the UI in case you weren't aware, there's a Rename Object button that does that behind the scenes.

IIsaac McFadyen You're correct about S3 but they abstract it away in the UI in case you weren't ...

ben•3/26/22, 12:00 AM

Sorry, I should have been more clear. I was thinking more about cost, so on s3, renaming a big folder means sending a copy request for each file in it (iirc) which could get expensive. Will R2 be like this as well?

andrew•3/26/22, 4:05 AM

i guess how expensive that will be will depend upon what the per request pricing ends up being

andrew•3/26/22, 4:05 AM

R2 will zero-rate infrequent storage operations under a threshold — currently planned to be in the single digit requests per second range. Above this range, R2 will charge significantly less per-operation than the major providers.

andrew•3/26/22, 4:05 AM

https://blog.cloudflare.com/introducing-r2-object-storage/

andrew•3/26/22, 4:06 AM

if the per-operation price is sufficiently low, then "renaming" a folder might be negligible. i don't think any per-operation prices have come out yet, correct?

ben•3/26/22, 5:13 AM

I guess we'll have to wait and see. It would be nice if we could move/rename an entire folder with one request though.

andrew•3/26/22, 3:05 PM

yeah it's just not something that goes well with the way object storage is typically architected. objects (and even portions of objects) along with their names are usually very heavily distributed via sharding

andrew•3/26/22, 3:09 PM

which is great for scalability but things like "fast atomic renames of an entire set of objects" isn't really practical without introducing complexity that likely would compromise said scalability
some kind of sharding-aware rename operation is probably feasible but i'm not sure it'd be worth it

beelzabubOP•3/28/22, 2:56 PM

Copy + delete should work and (unless I'm mistaken) since delete is 0-rated it's just the cost of 1 write. However, that's not actually a strongly consistent operation. An atomic move of an object is easy for us to implement, we just haven't yet. It's already on the backlog.

Moves of folders is more challenging. Nothing we can do that's as trivial as a folder "move". We could offer a list-like API to do the rename of objects within a folder. That would be billed at write rates but operate with list-like performance (i.e. 1k objects per request and you get back a continuation token).

beelzabubOP•3/28/22, 3:20 PM

@benhong would that list-like behavior work for you?