You need to be careful though. The length of the `ReadableStream` must be known. For example, incomi

You need to be careful though. The length of the

ReadableStream

ReadableStream

must be known. For example, incoming request with a

content-length

content-length

and you're forwarding along the body. Or the response body of a fetch that has a

content-length

content-length

. Or the readable half of a

FixedLengthStream

FixedLengthStream

transform stream.

beelzabubOP•4/1/22, 11:58 PM

That is not a helpful bot. We're going to start opening things up for our loyal Discord users in advance of Open Beta. We're just trying to smooth out some rough edges right now that aren't fun.

Bbeelzabub You need to be careful though. The length of the `ReadableStream` must be known....

James•4/2/22, 12:00 AM

Makes sense! We'll stand by for the update next week about updating our test scripts

Walshy•4/2/22, 12:00 AM

Will the API change be breaking or be in line with compat date? (For Worker binding)

Walshy•4/2/22, 12:03 AM

Oh I just read internally that the

r2_public_beta_bindings

r2_public_beta_bindings

is a compat flag right ok, makes sense

James•4/2/22, 12:04 AM

oh it's just a compat flag? Cool okay that's easy enough to update then

beelzabubOP•4/2/22, 12:04 AM

The public beta API and internal beta APIs are not source compatible. You will need to modify things (although hopefully simplifying at the same time). I've arbitrarily picked 4/18 as the time that

r2_public_beta_bindings

r2_public_beta_bindings

is on by default but for those that need to upgrade the compat date but keep the old bindings, you can opt to the old bindings via

r2_internal_beta_bindings

r2_internal_beta_bindings

. Anyone starting should use the public beta bindings. The old ones are not as complete and have edges.

Walshy•4/2/22, 12:04 AM

Yep seems so, makes this nice and easy

beelzabubOP•4/2/22, 12:05 AM

There's no "world broke please change". We learned our lesson

James•4/2/22, 12:05 AM

Yeah makes sense, we were warned the bindings would almost certainly change before launch

Walshy•4/2/22, 12:05 AM

hahaha ty for the detailed answer

beelzabubOP•4/2/22, 12:06 AM

At some point we'll make the old bindings start warning you in the logs. Any users still using them we'll try to make proactive contact with. At some point we may make the decision to remove them anyway to cleanup the codebase.

Walshy•4/2/22, 12:11 AM

expirationTtl is removed? Are there plans to have temp objects?

beelzabubOP•4/2/22, 12:19 AM

expirationTtl

expirationTtl

was never a thing and was never hooked up to anything IIRC (or maybe it was actually setting the cache expiry header? can't recall). There is a plan around it and we will reintroduce but we probably need to have object lifecycles comprehensively solved (although maybe we put in a hack before then - hard to say. we'll see how things develop). Talk to us internally if you absolutely need it but most likely our answer will be no.

beelzabubOP•4/2/22, 12:44 AM

Thanks to @mrbbot for pointing out a couple of things I missed.

Blob was missing from the accepted types. This was an oversight in the runtime code I wrote so the generated types are correct (if it's fixed next week in workers-types then it means the runtime also has it fixed)

ListOptions has a poorly documented and poorly typed

include

include

field which is strange since it's R2 specific. This is what is should be instead:

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

/** 
 * If you populate this array, then items returned will include this metadata.
 * A tradeoff is that fewer results may be returned depending on how big this
 * data is. For now the caps are TBD but expect the total memory usage for a list
 * operation may need to be <1MB or even <128kb depending on how many list operations
 * you are sending into one bucket. Make sure to look at `truncated` for the result
 * rather than having logic like
 *

while (listed.length < limit) {
listed = myBucket.list({ limit, include: ['customMetadata'] })
}
*/ include: ("httpMetadata" | "customMetadata")[]
*/ include: ("httpMetadata" | "customMetadata")[]
Basically it controls if the returned
```
R2Object
```
```
R2Object
```
list will have a
```
httpMetadata
```
```
httpMetadata
```
/
```
customMetadata
```
```
customMetadata
```
populated. The performance notes are very preliminary and I wouldn't take them too seriously. Design around what's easiest for now. The performance of the system will be changing drastically in the coming weeks.

andrew•4/2/22, 12:54 AM

wow really exciting to see things moving along. looking forward to trying out the beta once it's available

beelzabubOP•4/2/22, 12:54 AM

How much load can you throw at us?

beelzabubOP•4/2/22, 12:55 AM

and are you looking for accessing it via Workers or S3

andrew•4/2/22, 12:58 AM

one bit to pay attention to -- if a content-encoding header is specified during PUT, it should also be present for a GET. this is important if you're using deflate/brotli/etc and want browsers to auto-decompress content when fetching

beelzabubOP•4/2/22, 12:58 AM

Yup. That's exactly how R2 works

beelzabubOP•4/2/22, 12:58 AM

We also support overriding those headers during S3 GET (doesn't make sense for a worker)

andrew•4/2/22, 12:59 AM

cool

andrew•4/2/22, 12:59 AM

can metadata / http headers of an object be changed after upload, or does that need a reupload?

beelzabubOP•4/2/22, 12:59 AM

And I wanted to keep it a surprise but @john.spurlock's request here will hopefully make open beta because it's a semi breaking change so we want to level set expectations at launch

beelzabubOP•4/2/22, 1:01 AM

You can change the metadata/http headers in the S3 API but issuing a CopyObject with the same source and destination (sorry folks, that's how the S3 API works). I plan on plumbing through a more sane convenience name for Workers bindings eventually (which internally just aliases to the same operation internally that copy with identical source/dest bucket + name maps to).

beelzabubOP•4/2/22, 1:02 AM

We also have an extension to CopyObject. You can use "MERGE" as the x-amz-meta-directivex-amz-meta-directive which lets you update key/values without having to know the entire set ahead of time

beelzabubOP•4/2/22, 1:03 AM

I don't know how good tooling support on that is (so please ask your favorite libraries to add support) but we support it

andrew•4/2/22, 1:04 AM

sure makes sense, great stuff
once testing is available, i would be looking at high throughput GET usecases... and testing questions like "Can I achieve 1GB/s+ with multiple simultaneous GET requests?"
most likely via the S3 endpoint and fetching multiple objects simultaneously across multiple threads to VM destinations (would consider using workers, if that turns out better for performance than S3 GET endpoint)

beelzabubOP•4/2/22, 1:05 AM

Is that manual load to play around with or continuous sustained load?

andrew•4/2/22, 1:05 AM

would consider range requests as well

beelzabubOP•4/2/22, 1:05 AM

range requests are a thing

beelzabubOP•4/2/22, 1:06 AM

Our S3 API implementation is fairly complete

andrew•4/2/22, 1:06 AM

for the real world use case, it wouldn't be continuous load like that (1GB/s), but ability to burst to that level is very desirable/important

John Spurlock•4/2/22, 1:06 AM

looks like single range only? or will it support multiple ranges like the http range header?

beelzabubOP•4/2/22, 1:06 AM

Single range for now unfortunately

beelzabubOP•4/2/22, 1:06 AM

It might be a couple of years before we get to multi range

John Spurlock•4/2/22, 1:07 AM

k, and sorry which request of mine were you referring to? conditional puts?

beelzabubOP•4/2/22, 1:07 AM

I'll leave you to guess at based on the hint already provided

beelzabubOP•4/2/22, 1:07 AM

Which one might be a breaking change

andrew•4/2/22, 1:07 AM

multi ranges per request are interesting, i wasn't even aware of that existing in the wild

John Spurlock•4/2/22, 1:07 AM

ah gotcha

beelzabubOP•4/2/22, 1:07 AM

It doesn't

John Spurlock•4/2/22, 1:08 AM

this is amazing info, thanks for the detail!

beelzabubOP•4/2/22, 1:08 AM

Actually, I think I've blanket assumed we won't support multi range

beelzabubOP•4/2/22, 1:08 AM

I'll have to talk to the PM about it. It's technically very easy but there are potential billing implications

andrew•4/2/22, 1:08 AM

personally i'm not sure how useful it would be for us

beelzabubOP•4/2/22, 1:08 AM

Otherwise I would have done it when I implemented ranges

beelzabubOP•4/2/22, 1:09 AM

It's niche

beelzabubOP•4/2/22, 1:09 AM

But a popular feature & really weird other providers don't do it

andrew•4/2/22, 1:09 AM

sounds like more for reducing request load

andrew•4/2/22, 1:09 AM

when you're doing a lot of random access

/** * If you populate this array, then items returned will include this metadata. * A tradeoff is that fewer results may be returned depending on how big this * data is. For now the caps are TBD but expect the total memory usage for a list * operation may need to be <1MB or even <128kb depending on how many list operations * you are sending into one bucket. Make sure to look at `truncated` for the result * rather than having logic like *

You need to be careful though. The length of the `ReadableStream` must be known. For example, incomi

Similar Threads

Similar Threads

Similar Threads