Will R2 support moving/renaming files and folders? Right now in S3, you have to copy and delete each

Will R2 support moving/renaming files and folders? Right now in S3, you have to copy and delete each file.

Bben Will R2 support moving/renaming files and folders? Right now in S3, you have to ...

You're correct about S3 but they abstract it away in the UI in case you weren't aware, there's a Rename Object button that does that behind the scenes.

IIsaac McFadyen You're correct about S3 but they abstract it away in the UI in case you weren't ...

benOP•3/26/22, 12:00 AM

Sorry, I should have been more clear. I was thinking more about cost, so on s3, renaming a big folder means sending a copy request for each file in it (iirc) which could get expensive. Will R2 be like this as well?

andrew•3/26/22, 4:05 AM

i guess how expensive that will be will depend upon what the per request pricing ends up being

andrew•3/26/22, 4:05 AM

R2 will zero-rate infrequent storage operations under a threshold — currently planned to be in the single digit requests per second range. Above this range, R2 will charge significantly less per-operation than the major providers.

andrew•3/26/22, 4:05 AM

https://blog.cloudflare.com/introducing-r2-object-storage/

andrew•3/26/22, 4:06 AM

if the per-operation price is sufficiently low, then "renaming" a folder might be negligible. i don't think any per-operation prices have come out yet, correct?

benOP•3/26/22, 5:13 AM

I guess we'll have to wait and see. It would be nice if we could move/rename an entire folder with one request though.

andrew•3/26/22, 3:05 PM

yeah it's just not something that goes well with the way object storage is typically architected. objects (and even portions of objects) along with their names are usually very heavily distributed via sharding

andrew•3/26/22, 3:09 PM

which is great for scalability but things like "fast atomic renames of an entire set of objects" isn't really practical without introducing complexity that likely would compromise said scalability
some kind of sharding-aware rename operation is probably feasible but i'm not sure it'd be worth it

beelzabub•3/28/22, 2:56 PM

Copy + delete should work and (unless I'm mistaken) since delete is 0-rated it's just the cost of 1 write. However, that's not actually a strongly consistent operation. An atomic move of an object is easy for us to implement, we just haven't yet. It's already on the backlog.

Moves of folders is more challenging. Nothing we can do that's as trivial as a folder "move". We could offer a list-like API to do the rename of objects within a folder. That would be billed at write rates but operate with list-like performance (i.e. 1k objects per request and you get back a continuation token).

beelzabub•3/28/22, 3:20 PM

@benhong would that list-like behavior work for you?

benOP•3/28/22, 7:17 PM

That sounds great! So no avoiding paying the write rate for each object?

Bben That sounds great! So no avoiding paying the write rate for each object?

itsmatteomanf•3/29/22, 8:31 AM

From what I'm reading it looks to me it's still one write per object...

benOP•3/29/22, 8:32 AM

Alright, makes sense. Pretty darn cheap anyway haha

beelzabub•3/29/22, 5:27 PM

I don't know what the billing on a hypothetical API would be.

Laurent Pellegrino•3/29/22, 10:06 PM

Is R2 close to public beta?

lmtr0•3/29/22, 10:08 PM

thats one of the questions on the qna

lmtr0•3/29/22, 10:09 PM

no spoilers /s

James•3/29/22, 10:09 PM

I believe the most public ETA is beta in Q2 and GA in H2. Y'all are going to love it though

lmtr0•3/29/22, 10:10 PM

OK, I understood "I believe the most public" from the phrase.....
whats ETA, Q2, GA and H2?

lmtr0•3/29/22, 10:10 PM

Y'all are going to love it though

I think so

lmtr0•3/29/22, 10:10 PM

I mean

lmtr0•3/29/22, 10:11 PM

just the pricing is lovable

lmtr0•3/29/22, 10:11 PM

lmao

James•3/29/22, 10:11 PM

Sorry, beta in quarter 2 (April - Jun), and general availability in the second half of the year, July - Dec

James•3/29/22, 10:11 PM

That was the last publicly disclosed timeline, as far as I know

lmtr0•3/29/22, 10:11 PM

thankx

lmtr0•3/29/22, 10:12 PM

English is not my 1st lang so it's difficult to understand this stuff sorry

James•3/29/22, 10:12 PM

Absolutely no problem! Your english is great

I forget sometimes that most people don't know what GA, Q2, H2, etc. means

JJames Sorry, beta in quarter 2 (April - Jun), and general availability in the second h...

Isaac McFadyen•3/29/22, 10:13 PM

Yeah, that was the last I heard as well. From the CTO John Graham-Cumming on Twitter.

JJames Absolutely no problem! Your english is great 🙂 I forget sometimes that most peo...

Isaac McFadyen•3/29/22, 10:13 PM

TBH I only knew Q2 as well

Isaac McFadyen•3/29/22, 10:13 PM

But I assumed H2 must mean this year at some point

IIsaac McFadyen But I assumed H2 must mean this year at some point 🤷‍♂️

James•3/29/22, 10:14 PM

2nd half, yeah. And GA = general availability

James•3/29/22, 10:14 PM

If you're an ent customer though, definitely reach out to your CSM about testing!

JJames Absolutely no problem! Your english is great 🙂 I forget sometimes that most peo...

lmtr0•3/29/22, 10:15 PM

thx

JJames 2nd half, yeah. And GA = general availability

lmtr0•3/29/22, 10:15 PM

thx

andrew•3/29/22, 10:52 PM

regarding the feature that automatically pulls and expires objects from an existing s3 bucket, will that feature have object size limits?

beelzabub•3/30/22, 4:01 PM

I don't think it would.

beelzabub•3/30/22, 6:20 PM

Eventually yes. Automated object lifecycle management won't be something we have at open beta.

beelzabub•3/30/22, 6:22 PM

By eventually I mean the plan is sometime this year, not "indeterminate time in the future"

Aandrew regarding the feature that automatically pulls and expires objects from an exist...

beelzabub•3/30/22, 6:27 PM

I think the limit will be a bit under 5TB (more precisely ~5.3GB short of 5TB). It's on our roadmap to fix this limitation (which will also allow us to support files > 5TB).

beelzabub•3/30/22, 11:21 PM

It's on our roadmap but I don't know what shape it'll look like or what the timeline for that looks like.

What's the use-case your thinking of?

beelzabub•3/31/22, 12:34 AM

Ok. That's a bit different than what I was referring to which is cross account access for Cloudflare accounts. In this case it sounds like you have your own customers and they have accounts with you. What you would do is write a Worker to authorize the request (using whatever authorization mechanism you are using) to figure out which bucket the request has access to and then use the S3 API to upload to the right bucket.

Or you could make your life even easier and use a single bucket that's bound into your worker and your authorization is at the prefix level. Unlike S3 and other providers, buckets are organizational boundaries only. Each key will have roughly independent performance of any other key (within the bucket or in a different bucket). Same kind of thing will eventually apply to things like object lifecycles.

Of course you don't strictly need Workers. You could access R2 using the S3 endpoint from any server technology you want, just Workers will have the best performance and cost I think.

Is that helpful?

beelzabub•3/31/22, 12:47 AM

Gotcha. It sounds like you're looking for to build a SaaS product that provides a solution for each platform (ie. existing Azure Blobs, S3, and GCS customers use your product but manage buckets on the platform of their choice),. We don't yet have a good story for that use-case. I can't recall if that's on the roadmap for GA later this year. I think you'll find out more once we announce Open Beta.

beelzabub•3/31/22, 12:59 AM

Are you also needing requester pays billing on requests?

beelzabub•3/31/22, 1:00 AM

And what kind of access are you needing to the other account?

beelzabub•3/31/22, 1:01 AM

ie. read + write restricted to one bucket, read to one bucket, or is read+write to all buckets/read to all buckets sufficient?

Will R2 support moving/renaming files and folders? Right now in S3, you have to copy and delete each

Similar Threads

Similar Threads

Similar Threads