This is awesome! Content length is no longer a separate required argument to `put` it seems then - w

W

Walshy•4/1/22, 10:11 PM

No length anymore whooo

I

Isaac McFadyen•4/1/22, 10:25 PM

?r2

F

Flare•4/1/22, 10:25 PM

R2 is still in very early access, please do not ask to be added. There is no ETA for a public beta but watch out on the blog (https://blog.cloudflare.com/) or follow the #cloudflare-blog channel!

I

Isaac McFadyen•4/1/22, 10:25 PM

@raeesbhatti ^

JJames This is awesome! Content length is no longer a separate required argument to `pu...

B

beelzabub•4/1/22, 11:57 PM

You need to be careful though. The length of the

ReadableStream

ReadableStream

must be known. For example, incoming request with a

content-length

content-length

content-length

content-length and you're forwarding along the body. Or the response body of a fetch that has a

content-length

content-length

content-length

content-length. Or the readable half of a

FixedLengthStream

FixedLengthStream

FixedLengthStream

FixedLengthStream transform stream.

B

beelzabub•4/1/22, 11:58 PM

That is not a helpful bot. We're going to start opening things up for our loyal Discord users in advance of Open Beta. We're just trying to smooth out some rough edges right now that aren't fun.

Bbeelzabub You need to be careful though. The length of the `ReadableStream` must be known....

J

JamesOP•4/2/22, 12:00 AM

Makes sense! We'll stand by for the update next week about updating our test scripts

W

Walshy•4/2/22, 12:00 AM

Will the API change be breaking or be in line with compat date? (For Worker binding)

W

Walshy•4/2/22, 12:03 AM

Oh I just read internally that the

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings is a compat flag right ok, makes sense

J

JamesOP•4/2/22, 12:04 AM

oh it's just a compat flag? Cool okay that's easy enough to update then

B

beelzabub•4/2/22, 12:04 AM

The public beta API and internal beta APIs are not source compatible. You will need to modify things (although hopefully simplifying at the same time). I've arbitrarily picked 4/18 as the time that

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings

r2_public_beta_bindings is on by default but for those that need to upgrade the compat date but keep the old bindings, you can opt to the old bindings via

r2_internal_beta_bindings

r2_internal_beta_bindings

r2_internal_beta_bindings

r2_internal_beta_bindings. Anyone starting should use the public beta bindings. The old ones are not as complete and have edges.

W

Walshy•4/2/22, 12:04 AM

Yep seems so, makes this nice and easy

B

beelzabub•4/2/22, 12:05 AM

There's no "world broke please change". We learned our lesson

J

JamesOP•4/2/22, 12:05 AM

Yeah makes sense, we were warned the bindings would almost certainly change before launch

W

Walshy•4/2/22, 12:05 AM

hahaha ty for the detailed answer

B

beelzabub•4/2/22, 12:06 AM

At some point we'll make the old bindings start warning you in the logs. Any users still using them we'll try to make proactive contact with. At some point we may make the decision to remove them anyway to cleanup the codebase.

W

Walshy•4/2/22, 12:11 AM

expirationTtl is removed? Are there plans to have temp objects?

B

beelzabub•4/2/22, 12:19 AM

expirationTtl

expirationTtl

expirationTtl

expirationTtl was never a thing and was never hooked up to anything IIRC (or maybe it was actually setting the cache expiry header? can't recall). There is a plan around it and we will reintroduce but we probably need to have object lifecycles comprehensively solved (although maybe we put in a hack before then - hard to say. we'll see how things develop). Talk to us internally if you absolutely need it but most likely our answer will be no.

B

beelzabub•4/2/22, 12:44 AM

Thanks to @mrbbot for pointing out a couple of things I missed.

Blob was missing from the accepted types. This was an oversight in the runtime code I wrote so the generated types are correct (if it's fixed next week in workers-types then it means the runtime also has it fixed)

ListOptions has a poorly documented and poorly typed

include

include

include

include field which is strange since it's R2 specific. This is what is should be instead:

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

while (listed.length < limit) {
listed = myBucket.list({ limit, include: ['customMetadata'] })
}
*/ include: ("httpMetadata" | "customMetadata")[]
```
*/
include: ("httpMetadata" | "customMetadata")[]
```
*/ include: ("httpMetadata" | "customMetadata")[]
```
*/
include: ("httpMetadata" | "customMetadata")[]
```
Basically it controls if the returned
```
R2Object
```
R2Object
```
R2Object
```
R2Object
list will have a
```
httpMetadata
```
httpMetadata
```
httpMetadata
```
httpMetadata
/
```
customMetadata
```
customMetadata
```
customMetadata
```
customMetadata
populated. The performance notes are very preliminary and I wouldn't take them too seriously. Design around what's easiest for now. The performance of the system will be changing drastically in the coming weeks.

A

andrew•4/2/22, 12:54 AM

wow really exciting to see things moving along. looking forward to trying out the beta once it's available

B

beelzabub•4/2/22, 12:54 AM

How much load can you throw at us?

B

beelzabub•4/2/22, 12:55 AM

and are you looking for accessing it via Workers or S3

A

andrew•4/2/22, 12:58 AM

one bit to pay attention to -- if a content-encoding header is specified during PUT, it should also be present for a GET. this is important if you're using deflate/brotli/etc and want browsers to auto-decompress content when fetching

B

beelzabub•4/2/22, 12:58 AM

Yup. That's exactly how R2 works

B

beelzabub•4/2/22, 12:58 AM

We also support overriding those headers during S3 GET (doesn't make sense for a worker)

A

andrew•4/2/22, 12:59 AM

cool

A

andrew•4/2/22, 12:59 AM

can metadata / http headers of an object be changed after upload, or does that need a reupload?

B

beelzabub•4/2/22, 12:59 AM

And I wanted to keep it a surprise but @john.spurlock's request here will hopefully make open beta because it's a semi breaking change so we want to level set expectations at launch

B

beelzabub•4/2/22, 1:01 AM

You can change the metadata/http headers in the S3 API but issuing a CopyObject with the same source and destination (sorry folks, that's how the S3 API works). I plan on plumbing through a more sane convenience name for Workers bindings eventually (which internally just aliases to the same operation internally that copy with identical source/dest bucket + name maps to).

B

beelzabub•4/2/22, 1:02 AM

We also have an extension to CopyObject. You can use "MERGE" as the

x-amz-meta-directive

x-amz-meta-directive

x-amz-meta-directive

x-amz-meta-directive which lets you update key/values without having to know the entire set ahead of time

B

beelzabub•4/2/22, 1:03 AM

I don't know how good tooling support on that is (so please ask your favorite libraries to add support) but we support it

A

andrew•4/2/22, 1:04 AM

sure makes sense, great stuff
once testing is available, i would be looking at high throughput GET usecases... and testing questions like "Can I achieve 1GB/s+ with multiple simultaneous GET requests?"
most likely via the S3 endpoint and fetching multiple objects simultaneously across multiple threads to VM destinations (would consider using workers, if that turns out better for performance than S3 GET endpoint)

B

beelzabub•4/2/22, 1:05 AM

Is that manual load to play around with or continuous sustained load?

A

andrew•4/2/22, 1:05 AM

would consider range requests as well

B

beelzabub•4/2/22, 1:05 AM

range requests are a thing

B

beelzabub•4/2/22, 1:06 AM

Our S3 API implementation is fairly complete

A

andrew•4/2/22, 1:06 AM

for the real world use case, it wouldn't be continuous load like that (1GB/s), but ability to burst to that level is very desirable/important

J

John Spurlock•4/2/22, 1:06 AM

looks like single range only? or will it support multiple ranges like the http range header?

B

beelzabub•4/2/22, 1:06 AM

Single range for now unfortunately

B

beelzabub•4/2/22, 1:06 AM

It might be a couple of years before we get to multi range

J

John Spurlock•4/2/22, 1:07 AM

k, and sorry which request of mine were you referring to? conditional puts?

B

beelzabub•4/2/22, 1:07 AM

I'll leave you to guess at based on the hint already provided

B

beelzabub•4/2/22, 1:07 AM

Which one might be a breaking change

A

andrew•4/2/22, 1:07 AM

multi ranges per request are interesting, i wasn't even aware of that existing in the wild

J

John Spurlock•4/2/22, 1:07 AM

ah gotcha

B

beelzabub•4/2/22, 1:07 AM

It doesn't

J

John Spurlock•4/2/22, 1:08 AM

this is amazing info, thanks for the detail!

B

beelzabub•4/2/22, 1:08 AM

Actually, I think I've blanket assumed we won't support multi range

B

beelzabub•4/2/22, 1:08 AM

I'll have to talk to the PM about it. It's technically very easy but there are potential billing implications

This is awesome! Content length is no longer a separate required argument to `put` it seems then - w

Similar Threads

Similar Threads

Similar Threads