Cloudflare Developers•4y ago

Sorry, beta in quarter 2 (April - Jun), and general availability in the second half of the year, Jul

Sorry, beta in quarter 2 (April - Jun), and general availability in the second half of the year, July - Dec

JamesOP•3/29/22, 10:11 PM

That was the last publicly disclosed timeline, as far as I know

lmtr0•3/29/22, 10:11 PM

thankx

lmtr0•3/29/22, 10:12 PM

English is not my 1st lang so it's difficult to understand this stuff sorry

JamesOP•3/29/22, 10:12 PM

Absolutely no problem! Your english is great

I forget sometimes that most people don't know what GA, Q2, H2, etc. means

JJames Sorry, beta in quarter 2 (April - Jun), and general availability in the second h...

Isaac McFadyen•3/29/22, 10:13 PM

Yeah, that was the last I heard as well. From the CTO John Graham-Cumming on Twitter.

JJames Absolutely no problem! Your english is great 🙂 I forget sometimes that most peo...

Isaac McFadyen•3/29/22, 10:13 PM

TBH I only knew Q2 as well

Isaac McFadyen•3/29/22, 10:13 PM

But I assumed H2 must mean this year at some point

IIsaac McFadyen But I assumed H2 must mean this year at some point 🤷‍♂️

JamesOP•3/29/22, 10:14 PM

2nd half, yeah. And GA = general availability

JamesOP•3/29/22, 10:14 PM

If you're an ent customer though, definitely reach out to your CSM about testing!

JJames Absolutely no problem! Your english is great 🙂 I forget sometimes that most peo...

lmtr0•3/29/22, 10:15 PM

thx

JJames 2nd half, yeah. And GA = general availability

lmtr0•3/29/22, 10:15 PM

thx

andrew•3/29/22, 10:52 PM

regarding the feature that automatically pulls and expires objects from an existing s3 bucket, will that feature have object size limits?

beelzabub•3/30/22, 4:01 PM

I don't think it would.

beelzabub•3/30/22, 6:20 PM

Eventually yes. Automated object lifecycle management won't be something we have at open beta.

beelzabub•3/30/22, 6:22 PM

By eventually I mean the plan is sometime this year, not "indeterminate time in the future"

Aandrew regarding the feature that automatically pulls and expires objects from an exist...

beelzabub•3/30/22, 6:27 PM

I think the limit will be a bit under 5TB (more precisely ~5.3GB short of 5TB). It's on our roadmap to fix this limitation (which will also allow us to support files > 5TB).

beelzabub•3/30/22, 11:21 PM

It's on our roadmap but I don't know what shape it'll look like or what the timeline for that looks like.

What's the use-case your thinking of?

beelzabub•3/31/22, 12:34 AM

Ok. That's a bit different than what I was referring to which is cross account access for Cloudflare accounts. In this case it sounds like you have your own customers and they have accounts with you. What you would do is write a Worker to authorize the request (using whatever authorization mechanism you are using) to figure out which bucket the request has access to and then use the S3 API to upload to the right bucket.

Or you could make your life even easier and use a single bucket that's bound into your worker and your authorization is at the prefix level. Unlike S3 and other providers, buckets are organizational boundaries only. Each key will have roughly independent performance of any other key (within the bucket or in a different bucket). Same kind of thing will eventually apply to things like object lifecycles.

Of course you don't strictly need Workers. You could access R2 using the S3 endpoint from any server technology you want, just Workers will have the best performance and cost I think.

Is that helpful?

beelzabub•3/31/22, 12:47 AM

Gotcha. It sounds like you're looking for to build a SaaS product that provides a solution for each platform (ie. existing Azure Blobs, S3, and GCS customers use your product but manage buckets on the platform of their choice),. We don't yet have a good story for that use-case. I can't recall if that's on the roadmap for GA later this year. I think you'll find out more once we announce Open Beta.

beelzabub•3/31/22, 12:59 AM

Are you also needing requester pays billing on requests?

beelzabub•3/31/22, 1:00 AM

And what kind of access are you needing to the other account?

beelzabub•3/31/22, 1:01 AM

ie. read + write restricted to one bucket, read to one bucket, or is read+write to all buckets/read to all buckets sufficient?

beelzabub•3/31/22, 1:02 AM

For some reason I can't DM you.

Erwin•3/31/22, 1:11 AM

If you want those customers to give you access to one of their S3 buckets (or Azure/GCP equivalent), you can do that with regular API requests in a Worker..

beelzabub•4/1/22, 9:40 PM

Happy Friday everyone. As we're starting to head towards Open Beta, I just wanted to give you guys a sneak peek at the bindings we'll be publishing next week: https://gist.github.com/vlovich/9620194cd3e42a41e79d4cacbf1856ab

For users already onboarded in the private beta, you'll need to publish

r2_public_beta_bindings

r2_public_beta_bindings

sometime late next week (I'm not 100% sure when we'll allow such scripts - I'll keep you updated).

Please provide feedback. There's a few spots I've spotted:

R2Error

R2Error

stack

stack

any

any

instead of string (& in fact

name

name

message

message

stack

stack

can probably be elided because of inheritance? not sure).

R2Error

R2Error

code

code

needs a documentation note. This code is the same code that the Cloudflare V4 API that will be documented.

onlyIf

onlyIf

needs a documentation note. The

Headers

Headers

it accepts require HTTP header names (

IfModifiedSince

IfModifiedSince

IfNotModifiedSince

IfNotModifiedSince

etc).

R2Object

R2Object

writeHttpMetadata

writeHttpMetadata

is missing a note. It writes out the headers from

httpMetadata

httpMetadata

as you would expect (content-type, cache-control, etc)
Same goes for

httpMetadata

httpMetadata

on put (i.e.

cache-control

cache-control

cacheControl

cacheControl

expires

expires

cacheExpiry

cacheExpiry

). Note that S3 uses the HTTP "Expires" name but I think that confuses people into thinking that has something to do with lifecycle. It does not. It's simply the cache expiry metadata you associate with an object.

R2PutOptions

R2PutOptions

has

md5

md5

and

sha1

sha1

as optionals to specify the digest of the body, but those are mutually exclusive and the type needs to be reified to reflect that (the runtime enforces this, it's just not reflected in the type system).

Enjoy everyone!

Gist

final r2 bindings.ts

GitHub Gist: instantly share code, notes, and snippets.

beelzabub•4/1/22, 9:44 PM

We've focused on a tight core set of functionality. The S3 API has some extra features currently (e.g. copy object, multipart uploads) but we wanted to keep the canvas a bit cleaner and make sure we kept good ergonomics for the API.

JamesOP•4/1/22, 9:53 PM

This is awesome! Content length is no longer a separate required argument to

put

put

it seems then - woot!

Walshy•4/1/22, 10:11 PM

No length anymore whooo

Isaac McFadyen•4/1/22, 10:25 PM

?r2

Flare•4/1/22, 10:25 PM

R2 is still in very early access, please do not ask to be added. There is no ETA for a public beta but watch out on the blog (https://blog.cloudflare.com/) or follow the #cloudflare-blog channel!

Isaac McFadyen•4/1/22, 10:25 PM

@raeesbhatti ^

JJames This is awesome! Content length is no longer a separate required argument to `pu...

beelzabub•4/1/22, 11:57 PM

You need to be careful though. The length of the

ReadableStream

ReadableStream

must be known. For example, incoming request with a

content-length

content-length

and you're forwarding along the body. Or the response body of a fetch that has a

content-length

content-length

. Or the readable half of a

FixedLengthStream

FixedLengthStream

transform stream.

beelzabub•4/1/22, 11:58 PM

That is not a helpful bot. We're going to start opening things up for our loyal Discord users in advance of Open Beta. We're just trying to smooth out some rough edges right now that aren't fun.

Bbeelzabub You need to be careful though. The length of the `ReadableStream` must be known....

JamesOP•4/2/22, 12:00 AM

Makes sense! We'll stand by for the update next week about updating our test scripts

Walshy•4/2/22, 12:00 AM

Will the API change be breaking or be in line with compat date? (For Worker binding)

Walshy•4/2/22, 12:03 AM

Oh I just read internally that the

r2_public_beta_bindings

r2_public_beta_bindings

is a compat flag right ok, makes sense

JamesOP•4/2/22, 12:04 AM

oh it's just a compat flag? Cool okay that's easy enough to update then

beelzabub•4/2/22, 12:04 AM

The public beta API and internal beta APIs are not source compatible. You will need to modify things (although hopefully simplifying at the same time). I've arbitrarily picked 4/18 as the time that

r2_public_beta_bindings

r2_public_beta_bindings

is on by default but for those that need to upgrade the compat date but keep the old bindings, you can opt to the old bindings via

r2_internal_beta_bindings

r2_internal_beta_bindings

. Anyone starting should use the public beta bindings. The old ones are not as complete and have edges.

Walshy•4/2/22, 12:04 AM

Yep seems so, makes this nice and easy

beelzabub•4/2/22, 12:05 AM

There's no "world broke please change". We learned our lesson

JamesOP•4/2/22, 12:05 AM

Yeah makes sense, we were warned the bindings would almost certainly change before launch

Walshy•4/2/22, 12:05 AM

hahaha ty for the detailed answer

beelzabub•4/2/22, 12:06 AM

At some point we'll make the old bindings start warning you in the logs. Any users still using them we'll try to make proactive contact with. At some point we may make the decision to remove them anyway to cleanup the codebase.

Walshy•4/2/22, 12:11 AM

expirationTtl is removed? Are there plans to have temp objects?

beelzabub•4/2/22, 12:19 AM

expirationTtl

expirationTtl

was never a thing and was never hooked up to anything IIRC (or maybe it was actually setting the cache expiry header? can't recall). There is a plan around it and we will reintroduce but we probably need to have object lifecycles comprehensively solved (although maybe we put in a hack before then - hard to say. we'll see how things develop). Talk to us internally if you absolutely need it but most likely our answer will be no.

beelzabub•4/2/22, 12:44 AM

Thanks to @mrbbot for pointing out a couple of things I missed.

Blob was missing from the accepted types. This was an oversight in the runtime code I wrote so the generated types are correct (if it's fixed next week in workers-types then it means the runtime also has it fixed)

ListOptions has a poorly documented and poorly typed

include

include

field which is strange since it's R2 specific. This is what is should be instead:

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

while (listed.length < limit) {
listed = myBucket.list({ limit, include: ['customMetadata'] })
}
*/ include: ("httpMetadata" | "customMetadata")[]
*/ include: ("httpMetadata" | "customMetadata")[]
Basically it controls if the returned
```
R2Object
```
```
R2Object
```
list will have a
```
httpMetadata
```
```
httpMetadata
```
/
```
customMetadata
```
```
customMetadata
```
populated. The performance notes are very preliminary and I wouldn't take them too seriously. Design around what's easiest for now. The performance of the system will be changing drastically in the coming weeks.

/** * If you populate this array, then items returned will include this metadata. * A tradeoff is that fewer results may be returned depending on how big this * data is. For now the caps are TBD but expect the total memory usage for a list * operation may need to be <1MB or even <128kb depending on how many list operations * you are sending into one bucket. Make sure to look at `truncated` for the result * rather than having logic like *

Sorry, beta in quarter 2 (April - Jun), and general availability in the second half of the year, Jul

Similar Threads

Similar Threads

Similar Threads