regarding the feature that automatically pulls and expires objects from an existing s3 bucket, will

B

beelzabub•3/30/22, 4:01 PM

I don't think it would.

B

beelzabub•3/30/22, 6:20 PM

Eventually yes. Automated object lifecycle management won't be something we have at open beta.

B

beelzabub•3/30/22, 6:22 PM

By eventually I mean the plan is sometime this year, not "indeterminate time in the future"

Aandrew regarding the feature that automatically pulls and expires objects from an exist...

B

beelzabub•3/30/22, 6:27 PM

I think the limit will be a bit under 5TB (more precisely ~5.3GB short of 5TB). It's on our roadmap to fix this limitation (which will also allow us to support files > 5TB).

B

beelzabub•3/30/22, 11:21 PM

It's on our roadmap but I don't know what shape it'll look like or what the timeline for that looks like.

What's the use-case your thinking of?

B

beelzabub•3/31/22, 12:34 AM

Ok. That's a bit different than what I was referring to which is cross account access for Cloudflare accounts. In this case it sounds like you have your own customers and they have accounts with you. What you would do is write a Worker to authorize the request (using whatever authorization mechanism you are using) to figure out which bucket the request has access to and then use the S3 API to upload to the right bucket.

Or you could make your life even easier and use a single bucket that's bound into your worker and your authorization is at the prefix level. Unlike S3 and other providers, buckets are organizational boundaries only. Each key will have roughly independent performance of any other key (within the bucket or in a different bucket). Same kind of thing will eventually apply to things like object lifecycles.

Of course you don't strictly need Workers. You could access R2 using the S3 endpoint from any server technology you want, just Workers will have the best performance and cost I think.

Is that helpful?

B

beelzabub•3/31/22, 12:47 AM

Gotcha. It sounds like you're looking for to build a SaaS product that provides a solution for each platform (ie. existing Azure Blobs, S3, and GCS customers use your product but manage buckets on the platform of their choice),. We don't yet have a good story for that use-case. I can't recall if that's on the roadmap for GA later this year. I think you'll find out more once we announce Open Beta.

B

beelzabub•3/31/22, 12:59 AM

Are you also needing requester pays billing on requests?

B

beelzabub•3/31/22, 1:00 AM

And what kind of access are you needing to the other account?

B

beelzabub•3/31/22, 1:01 AM

ie. read + write restricted to one bucket, read to one bucket, or is read+write to all buckets/read to all buckets sufficient?

B

beelzabub•3/31/22, 1:02 AM

For some reason I can't DM you.

E

Erwin•3/31/22, 1:11 AM

If you want those customers to give you access to one of their S3 buckets (or Azure/GCP equivalent), you can do that with regular API requests in a Worker..

B

beelzabub•4/1/22, 9:40 PM

Happy Friday everyone. As we're starting to head towards Open Beta, I just wanted to give you guys a sneak peek at the bindings we'll be publishing next week: https://gist.github.com/vlovich/9620194cd3e42a41e79d4cacbf1856ab

For users already onboarded in the private beta, you'll need to publish

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings sometime late next week (I'm not 100% sure when we'll allow such scripts - I'll keep you updated).

Please provide feedback. There's a few spots I've spotted:

R2Error

R2Error

R2Error

R2Error

stack

stack

stack

stack is

any

any

any

any instead of string (& in fact

name

name

/

message

message

message

message/

stack

stack

stack

stack can probably be elided because of inheritance? not sure).

R2Error

R2Error

R2Error

R2Error

code

code

code

code needs a documentation note. This code is the same code that the Cloudflare V4 API that will be documented.

onlyIf

onlyIf

onlyIf

onlyIf needs a documentation note. The

Headers

Headers

Headers

Headers it accepts require HTTP header names (

IfModifiedSince

IfModifiedSince

IfModifiedSince

IfModifiedSince/

IfNotModifiedSince

IfNotModifiedSince

IfNotModifiedSince

IfNotModifiedSince etc).

R2Object

R2Object

R2Object

R2Object

writeHttpMetadata

writeHttpMetadata

writeHttpMetadata

writeHttpMetadata is missing a note. It writes out the headers from

httpMetadata

httpMetadata

as you would expect (content-type, cache-control, etc)
Same goes for

httpMetadata

httpMetadata

on put (i.e.

cache-control

cache-control

->

cacheControl

cacheControl

cacheControl

cacheControl,

expires

expires

expires

expires ->

cacheExpiry

cacheExpiry

cacheExpiry

cacheExpiry). Note that S3 uses the HTTP "Expires" name but I think that confuses people into thinking that has something to do with lifecycle. It does not. It's simply the cache expiry metadata you associate with an object.

R2PutOptions

R2PutOptions

R2PutOptions

R2PutOptions has

md5

md5

md5

md5 and

sha1

sha1

sha1

sha1 as optionals to specify the digest of the body, but those are mutually exclusive and the type needs to be reified to reflect that (the runtime enforces this, it's just not reflected in the type system).

Enjoy everyone!

Gist

final r2 bindings.ts

GitHub Gist: instantly share code, notes, and snippets.

B

beelzabub•4/1/22, 9:44 PM

We've focused on a tight core set of functionality. The S3 API has some extra features currently (e.g. copy object, multipart uploads) but we wanted to keep the canvas a bit cleaner and make sure we kept good ergonomics for the API.

J

James•4/1/22, 9:53 PM

This is awesome! Content length is no longer a separate required argument to

put

put

it seems then - woot!

W

Walshy•4/1/22, 10:11 PM

No length anymore whooo

I

Isaac McFadyen•4/1/22, 10:25 PM

?r2

F

Flare•4/1/22, 10:25 PM

R2 is still in very early access, please do not ask to be added. There is no ETA for a public beta but watch out on the blog (https://blog.cloudflare.com/) or follow the #cloudflare-blog channel!

I

Isaac McFadyen•4/1/22, 10:25 PM

@raeesbhatti ^

JJames This is awesome! Content length is no longer a separate required argument to `pu...

B

beelzabub•4/1/22, 11:57 PM

You need to be careful though. The length of the

ReadableStream

ReadableStream

must be known. For example, incoming request with a

content-length

content-length

content-length

content-length and you're forwarding along the body. Or the response body of a fetch that has a

content-length

content-length

content-length

content-length. Or the readable half of a

FixedLengthStream

FixedLengthStream

FixedLengthStream

FixedLengthStream transform stream.

B

beelzabub•4/1/22, 11:58 PM

That is not a helpful bot. We're going to start opening things up for our loyal Discord users in advance of Open Beta. We're just trying to smooth out some rough edges right now that aren't fun.

Bbeelzabub You need to be careful though. The length of the `ReadableStream` must be known....

J

James•4/2/22, 12:00 AM

Makes sense! We'll stand by for the update next week about updating our test scripts

W

Walshy•4/2/22, 12:00 AM

Will the API change be breaking or be in line with compat date? (For Worker binding)

W

Walshy•4/2/22, 12:03 AM

Oh I just read internally that the

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings is a compat flag right ok, makes sense

J

James•4/2/22, 12:04 AM

oh it's just a compat flag? Cool okay that's easy enough to update then

B

beelzabub•4/2/22, 12:04 AM

The public beta API and internal beta APIs are not source compatible. You will need to modify things (although hopefully simplifying at the same time). I've arbitrarily picked 4/18 as the time that

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings is on by default but for those that need to upgrade the compat date but keep the old bindings, you can opt to the old bindings via

r2_internal_beta_bindings

r2_internal_beta_bindings

r2_internal_beta_bindings

r2_internal_beta_bindings. Anyone starting should use the public beta bindings. The old ones are not as complete and have edges.

W

Walshy•4/2/22, 12:04 AM

Yep seems so, makes this nice and easy

B

beelzabub•4/2/22, 12:05 AM

There's no "world broke please change". We learned our lesson

J

James•4/2/22, 12:05 AM

Yeah makes sense, we were warned the bindings would almost certainly change before launch

W

Walshy•4/2/22, 12:05 AM

hahaha ty for the detailed answer

B

beelzabub•4/2/22, 12:06 AM

At some point we'll make the old bindings start warning you in the logs. Any users still using them we'll try to make proactive contact with. At some point we may make the decision to remove them anyway to cleanup the codebase.

W

Walshy•4/2/22, 12:11 AM

expirationTtl is removed? Are there plans to have temp objects?

B

beelzabub•4/2/22, 12:19 AM

expirationTtl

expirationTtl

expirationTtl

expirationTtl was never a thing and was never hooked up to anything IIRC (or maybe it was actually setting the cache expiry header? can't recall). There is a plan around it and we will reintroduce but we probably need to have object lifecycles comprehensively solved (although maybe we put in a hack before then - hard to say. we'll see how things develop). Talk to us internally if you absolutely need it but most likely our answer will be no.

B

beelzabub•4/2/22, 12:44 AM

Thanks to @mrbbot for pointing out a couple of things I missed.

Blob was missing from the accepted types. This was an oversight in the runtime code I wrote so the generated types are correct (if it's fixed next week in workers-types then it means the runtime also has it fixed)

ListOptions has a poorly documented and poorly typed

include

include

field which is strange since it's R2 specific. This is what is should be instead:

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

while (listed.length < limit) {
listed = myBucket.list({ limit, include: ['customMetadata'] })
}
*/ include: ("httpMetadata" | "customMetadata")[]
```
*/
include: ("httpMetadata" | "customMetadata")[]
```
*/ include: ("httpMetadata" | "customMetadata")[]
```
*/
include: ("httpMetadata" | "customMetadata")[]
```
Basically it controls if the returned
```
R2Object
```
R2Object
```
R2Object
```
R2Object
list will have a
```
httpMetadata
```
```
httpMetadata
```
/
```
customMetadata
```
customMetadata
```
customMetadata
```
customMetadata
populated. The performance notes are very preliminary and I wouldn't take them too seriously. Design around what's easiest for now. The performance of the system will be changing drastically in the coming weeks.

A

andrewOP•4/2/22, 12:54 AM

wow really exciting to see things moving along. looking forward to trying out the beta once it's available

B

beelzabub•4/2/22, 12:54 AM

How much load can you throw at us?

B

beelzabub•4/2/22, 12:55 AM

and are you looking for accessing it via Workers or S3

A

andrewOP•4/2/22, 12:58 AM

one bit to pay attention to -- if a content-encoding header is specified during PUT, it should also be present for a GET. this is important if you're using deflate/brotli/etc and want browsers to auto-decompress content when fetching

B

beelzabub•4/2/22, 12:58 AM

Yup. That's exactly how R2 works

B

beelzabub•4/2/22, 12:58 AM

We also support overriding those headers during S3 GET (doesn't make sense for a worker)

A

andrewOP•4/2/22, 12:59 AM

cool

A

andrewOP•4/2/22, 12:59 AM

can metadata / http headers of an object be changed after upload, or does that need a reupload?

B

beelzabub•4/2/22, 12:59 AM

And I wanted to keep it a surprise but @john.spurlock's request here will hopefully make open beta because it's a semi breaking change so we want to level set expectations at launch

B

beelzabub•4/2/22, 1:01 AM

You can change the metadata/http headers in the S3 API but issuing a CopyObject with the same source and destination (sorry folks, that's how the S3 API works). I plan on plumbing through a more sane convenience name for Workers bindings eventually (which internally just aliases to the same operation internally that copy with identical source/dest bucket + name maps to).

B

beelzabub•4/2/22, 1:02 AM

We also have an extension to CopyObject. You can use "MERGE" as the

x-amz-meta-directive

x-amz-meta-directive

x-amz-meta-directive

x-amz-meta-directive which lets you update key/values without having to know the entire set ahead of time

B

beelzabub•4/2/22, 1:03 AM

I don't know how good tooling support on that is (so please ask your favorite libraries to add support) but we support it

A

andrewOP•4/2/22, 1:04 AM

sure makes sense, great stuff
once testing is available, i would be looking at high throughput GET usecases... and testing questions like "Can I achieve 1GB/s+ with multiple simultaneous GET requests?"
most likely via the S3 endpoint and fetching multiple objects simultaneously across multiple threads to VM destinations (would consider using workers, if that turns out better for performance than S3 GET endpoint)

B

beelzabub•4/2/22, 1:05 AM

Is that manual load to play around with or continuous sustained load?

regarding the feature that automatically pulls and expires objects from an existing s3 bucket, will

Similar Threads

Similar Threads

Similar Threads