@Isaac McFadyen | YYZ01 do you know who's responsible for https://pricing.ceru.dev/r2 ? I'm curious

@Isaac McFadyen | YYZ01 do you know who's responsible for https://pricing.ceru.dev/r2 ? I'm curious if the pricing configuration accounts for the free limits correctly and the github source doesn't seem to include R2

Cloudflare Workers™ pricing calculator

Estimate your next bill, see the price difference. Unofficial calculator for Cloudflare Workers Unbound, Cloudflare R2, and KV.

Bbeelzabub @Isaac McFadyen | YYZ01 do you know who's responsible for https://pricing.ceru.d...

Isaac McFadyen•6/30/22, 2:50 AM

I do.

Isaac McFadyen•6/30/22, 2:51 AM

It's @toridev (Cerulean without the nickname).

Flare•6/30/22, 4:26 AM

The following features are currently on the roadmap for R2 but not yet implemented.

Public Buckets
Mapping a bucket to your custom domain
Pre-signed URLs are now implemented!

In the meantime, you can use a Worker to replicate the vast majority of functionality such as serving & caching public assets.
Take a look at the Demo Worker example here: https://developers.cloudflare.com/r2/examples/demo-worker/
Also, there's a great community project called Render that's made for R2:
https://github.com/kotx/render

Workers will not be required for making assets publicly available in the future, they are just an interim stop-gap.

beelzabubOP•6/30/22, 4:38 AM

Yes. It's part of the public buckets work but it'll be one of the last things implemented.

beelzabubOP•6/30/22, 4:39 AM

The current solution indeed is to have a Worker in between that adds the CORS information you need

beelzabubOP•6/30/22, 4:43 AM

If anyone writes a unit-tested Worker that evaluates S3 CORS rules and applies them to requests (as JS objects - I can do the XML -> JS piece) that'll make that piece go

faster

Bbeelzabub If anyone writes a unit-tested Worker that evaluates S3 CORS rules and applies t...

hinego•6/30/22, 5:32 AM

request two function of r2

support redirect object
support bind custom domain with prefix

For example:
have two object:
object1/test.txt the content is “test1111”
object2/test2.txt set redirect to object1/test.txt in metadata （the object is 0 bytes file only set matedata）

Then request
https://domain/object1/test.txt
response filename test.txt with content test1111
https://domain/object2/test2.txt
response filename test2.txt with content test1111

and if bind domain2 with prefix：object2

will support use

https://domain2/test2.txt

to request /object2/test2.txt

of course it can be implement with workers but it will cost more

�

𝖆𝖚𝖙𝖎𝖘𝖙•6/30/22, 6:54 AM

?r2-roadmap

Flare•6/30/22, 6:54 AM

The following features are currently on the roadmap for R2 but not yet implemented.

Public Buckets
Mapping a bucket to your custom domain
Pre-signed URLs are now implemented!

In the meantime, you can use a Worker to replicate the vast majority of functionality such as serving & caching public assets.
Take a look at the Demo Worker example here: https://developers.cloudflare.com/r2/examples/demo-worker/
Also, there's a great community project called Render that's made for R2:
https://github.com/kotx/render

Workers will not be required for making assets publicly available in the future, they are just an interim stop-gap.

Bbeelzabub If anyone writes a unit-tested Worker that evaluates S3 CORS rules and applies t...

kian•6/30/22, 7:22 AM

Hhinego request two function of r2 1. support redirect object 2. support bind custom do...

beelzabubOP•6/30/22, 1:36 PM

I'm aware of the request for symlink or redirect support (this is the 3rd time you're asking). Seems unlikely we'll implement it in any short term. There are several challenges.

That operation doesn't exist at all in S3 which means what does the shape of the REST query look like in a way that remains future-proofed against conflicting with future changes AWS makes?
This operation doesn't exist at all which means how do you make it accessible from tooling?
What is the expectation around behavior if the original file is deleted?
It complicates the metadata layer a bit.

It's probably tractable by hanging it off of CopyObjectCopyObject as a custom copy directive (we have REPLACEREPLACE so probably SYMLINKSYMLINK might work there). That solves 1 and 2 but makes 3 weird since after issuing a CopyObjectCopyObject to create object2/test2.txt --> object1/test.txtobject2/test2.txt --> object1/test.txt, DeleteObjectDeleteObject object1/test.txtobject1/test.txt, the copy "doesn't exist" and is dangling metadata that you'd be billed for that's difficult to cleanup (you'd have to basically try to access that object or encounter it in a list). What happens if you now put a new object1/test.txtobject1/test.txt

is the symlink expected to work? The solution here is probably that such a redirect shows up in ListObjectListObject but always returns a 404 on a GET (but does it respond to a HEAD request? maybe only if there's an option that says "don't resolve symlinks"?)

4 pops up because what happens if you:

PutObjectPutObject object1/test.txtobject1/test.txt
CopyObjectCopyObject object2/test2.txt --> object1/test.txtobject2/test2.txt --> object1/test.txt
CopyObjectCopyObject object1/test.txt --> object2/test2.txtobject1/test.txt --> object2/test2.txt
This isn't even getting into the concept of making cross-bucket symlinks.

The hand wavy concept is easy to comprehend but making sure all the corner cases have defined semantics that don't surprise you in behavior, less so. Object storage systems look like filesystems but they're not.

Hhinego request two function of r2 1. support redirect object 2. support bind custom do...

beelzabubOP•6/30/22, 1:36 PM

Re #2, that's the cname stuff I mentioned earlier.

hinego•6/30/22, 1:46 PM

Think’s for your reply

Bbeelzabub I'm aware of the request for symlink or redirect support (this is the 3rd time y...

beelzabubOP•6/30/22, 1:46 PM

Oh and writes are another challenge. If I have B --> AB --> A and I PutObjectPutObject to BB, does that destroy the symlink and create a new BB or does it overwrite AA? Lots of nuance.

hinego•6/30/22, 2:03 PM

I don't think you need to think so much

For example, if exists (a -> b)

When getting the object, just cast the request target to B internally! Return 404 if B does not exist!

When operating B, you do not need to care about A, and the user maintains it by himself

It doesn't matter if A is charged even after B is removed, because it's cheap!

R2 only needs to provide the function of internal jump (and only supports one jump, only from A→B, not A→B→C), and the jump is maintained by the user

hinego•6/30/22, 3:45 PM

For example, if the accessed object metadata exists [x-redirect-object], then change the access target to the value of [x-redirect-object]

For example I set metadata for the object /user3333/test222.txt
x-redirect-object:/object/test.bin

Then when I request /user3333/test222.txt, the target of the request will be redirected (internally) to /object/test.bin, but the returned file name can be test222.txt
If /object/test.bin does not exist, then return 404 normally

The main reason for this is to save storage space if you need to save the same file for multiple users, but they name them differently.

And, if you can support binding prefixes, you can bind a separate domain name to each user, and they can directly access files with their domain name

For example, now two users have saved the same file (test.png), but they are named differently,
User A is named: /static/image/test.png
User B named: /images/background.png
If bind prefix and redirection are supported
Then you can save the file as: /raw/test.png
Create an object for user A: /user_a/static/image/test.png
Create an object for user B: /user_b/images/background.png
All set [x-redirect-object=/raw/test.png] for them
Then bind the domain name [user1.domain.com] and prefix [/user_a] to user A
Bind the domain name [user2.domain.com] and prefix [/user_b] to user B

at this time
User A can use
user1.domain.com/static/image/test.png
User B can use
user2.domain.com/images/background.png
access object

and the object is only saved once

beelzabubOP•6/30/22, 3:50 PM

My job as a TL is to think through the consequences of decisions. I understand your use case is solved by the requirements you laid out but we don't build features just for 1 customer unless it's a very large ENT contract (& even then, maybe). I need to think through the problem comprehensively so that the feature is designed in a way that doesn't lead to a dead end or have sharp corner cases that surprise customers that aren't you in weird ways.

This is a cost reduction mechanism but only for the customers that can make use of it (& even then, storage is fairly cheap - I don't know how big of a cost reduction this is in practice).

AFAIK no other object storage system has something like this. That gives me pause and makes me want to make sure I've thought through every potential corner case & know how it should work.

beelzabubOP•6/30/22, 3:52 PM

A more general solution might even be to just optimize CopyObjectCopyObject so that copies don't count towards more storage space & reduce storage costs in a significantly more meaningful way. Then you don't even need symlinks. This is why I'm saying things need to be thought through.

hinego•6/30/22, 3:53 PM

It would be better if CopyObject was not billed

beelzabubOP•6/30/22, 3:55 PM

Anyway. My point wasn't to say we're going down a path. Just that there's a lot of nuance & options to think through before we pick a path.

hinego•6/30/22, 3:56 PM

ok thanks for your patience

beelzabubOP•6/30/22, 3:58 PM

And it's also possible we will choose to pick no paths & stick with existing primitives (i.e. not high value enough of a feature vs so many other things we need to go build)

kian•6/30/22, 4:03 PM

Not currently - if you was using a Worker for public access (or any kind of auth you run in a Worker) then you could DIY it up but that's about it at the moment

Bbeelzabub My job as a TL is to think through the consequences of decisions. I understand y...

zegevlier•6/30/22, 5:20 PM

Just curious: does TL stand for technical lead or something else?

Arnø•6/30/22, 8:54 PM

How can I limit the size of body in workers ? by default it's 100 mo but I want to limit to 4mo

kian•6/30/22, 8:55 PM

It's not a specific Workers limitation, it's your zone in general for anything proxied through your Cloudflare zone.

kian•6/30/22, 8:56 PM

You could do it in a firewall rule I guess?

beelzabubOP•6/30/22, 8:56 PM

Something like:

Number(request.headers.get('content-length')) >= 4 * 1024 * 1024

Number(request.headers.get('content-length')) >= 4 * 1024 * 1024

Just be careful as not all requests need to have a content-length. Firewall rule is probably cheaper than doing it in the worker

beelzabubOP•6/30/22, 8:56 PM

(I don't know how firewall billing works)

kian•6/30/22, 8:57 PM

Free - that's why I went with it haha. Could do it in a Worker but a request is a request as far as that billing side

kian•6/30/22, 8:57 PM

You will run into the same issue that you mentioned about a lack of content-length though

kian•6/30/22, 8:57 PM

I don't see any ruleset fields that have the body size sat as a nice value waiting for you

beelzabubOP•6/30/22, 8:58 PM

If those are coming into a worker handling R2 & it's not doing anything else, you can reject those requests anyway. The content length is required to be known

kian•6/30/22, 8:58 PM

That's a good point

Bbeelzabub Something like: ``` Number(request.headers.get('content-length')) >= 4 * 1024 *...

Arnø•6/30/22, 9:03 PM

ok thanks you

Bbeelzabub Something like: ``` Number(request.headers.get('content-length')) >= 4 * 1024 *...

asp•6/30/22, 9:13 PM

Cloudflare's web server (nginx-fl) buffers the entire request before sending it along, so all requests with a body should also have a Content-LengthContent-Length header. You won't see any requests with Transfer-Encoding: chunkedTransfer-Encoding: chunked in Workers. I believe the same is true for Firewall Rules.

kian•6/30/22, 9:19 PM

Blocks me if I don't include a content-lengthcontent-length header (i.e GETGET) - so will need to add the method in

kian•6/30/22, 9:21 PM

(http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT" }) and (not any(lower(http.request.headers.names[*])[*] contains "content-length") or any(http.request.headers["content-length"][*] > "4096"))

(http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT" }) and (not any(lower(http.request.headers.names[*])[*] contains "content-length") or any(http.request.headers["content-length"][*] > "4096"))

kian•6/30/22, 9:22 PM

Restrict POSTPOST and PUTPUT requests on fw-test.kian.org.ukfw-test.kian.org.uk to a Content-LengthContent-Length less than 40964096 (and block any that don't include it at all, will accept

for 0-length uploads)

kian•6/30/22, 9:23 PM

kian•6/30/22, 9:23 PM

Example POST - (domain goes nowhere, expected a 526)

Kkian ``` (http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT...

asp•6/30/22, 9:27 PM

I believe you can just do any(http.request.headers["content-length"][*] > 4194304)any(http.request.headers["content-length"][*] > 4194304). If there is no Content-LengthContent-Length header - because the body is empty - the expression will evaluate to false.

Edit: I am confusing myself

kian•6/30/22, 9:27 PM

Probably - ruleset engine hurts my brain

@Isaac McFadyen | YYZ01 do you know who's responsible for https://pricing.ceru.dev/r2 ? I'm curious

Similar Threads

Similar Threads

Similar Threads