I'm aware of the request for symlink or redirect support (this is the 3rd time you're asking). Seems

I'm aware of the request for symlink or redirect support (this is the 3rd time you're asking). Seems unlikely we'll implement it in any short term. There are several challenges.

That operation doesn't exist at all in S3 which means what does the shape of the REST query look like in a way that remains future-proofed against conflicting with future changes AWS makes?
This operation doesn't exist at all which means how do you make it accessible from tooling?
What is the expectation around behavior if the original file is deleted?
It complicates the metadata layer a bit.

It's probably tractable by hanging it off of

CopyObject

CopyObject

as a custom copy directive (we have

REPLACE

REPLACE

so probably

SYMLINK

SYMLINK

SYMLINK

SYMLINK might work there). That solves 1 and 2 but makes 3 weird since after issuing a

CopyObject

CopyObject

to create

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt,

DeleteObject

DeleteObject

DeleteObject

DeleteObject

object1/test.txt

object1/test.txt

object1/test.txt

object1/test.txt, the copy "doesn't exist" and is dangling metadata that you'd be billed for that's difficult to cleanup (you'd have to basically try to access that object or encounter it in a list). What happens if you now put a new

object1/test.txt

object1/test.txt

object1/test.txt

object1/test.txt

is the symlink expected to work? The solution here is probably that such a redirect shows up in
```
ListObject
```
ListObject
```
ListObject
```
ListObject
but always returns a 404 on a GET (but does it respond to a HEAD request? maybe only if there's an option that says "don't resolve symlinks"?)

4 pops up because what happens if you:

PutObject

PutObject

PutObject

PutObject

object1/test.txt

object1/test.txt

object1/test.txt

object1/test.txt

CopyObject

CopyObject

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt

object2/test2.txt --> object1/test.txt

```
CopyObject
```
```
CopyObject
```
```
object1/test.txt --> object2/test2.txt
```
object1/test.txt --> object2/test2.txt
```
object1/test.txt --> object2/test2.txt
```
object1/test.txt --> object2/test2.txt
This isn't even getting into the concept of making cross-bucket symlinks.

The hand wavy concept is easy to comprehend but making sure all the corner cases have defined semantics that don't surprise you in behavior, less so. Object storage systems look like filesystems but they're not.

Hhinego request two function of r2 1. support redirect object 2. support bind custom do...

beelzabubOP•6/30/22, 1:36 PM

Re #2, that's the cname stuff I mentioned earlier.

hinego•6/30/22, 1:46 PM

Think’s for your reply

Bbeelzabub I'm aware of the request for symlink or redirect support (this is the 3rd time y...

beelzabubOP•6/30/22, 1:46 PM

Oh and writes are another challenge. If I have B --> AB --> A and I

PutObject

PutObject

to BB, does that destroy the symlink and create a new BB or does it overwrite AA? Lots of nuance.

hinego•6/30/22, 2:03 PM

I don't think you need to think so much

For example, if exists (a -> b)

When getting the object, just cast the request target to B internally! Return 404 if B does not exist!

When operating B, you do not need to care about A, and the user maintains it by himself

It doesn't matter if A is charged even after B is removed, because it's cheap!

R2 only needs to provide the function of internal jump (and only supports one jump, only from A→B, not A→B→C), and the jump is maintained by the user

hinego•6/30/22, 3:45 PM

For example, if the accessed object metadata exists [x-redirect-object], then change the access target to the value of [x-redirect-object]

For example I set metadata for the object /user3333/test222.txt
x-redirect-object:/object/test.bin

Then when I request /user3333/test222.txt, the target of the request will be redirected (internally) to /object/test.bin, but the returned file name can be test222.txt
If /object/test.bin does not exist, then return 404 normally

The main reason for this is to save storage space if you need to save the same file for multiple users, but they name them differently.

And, if you can support binding prefixes, you can bind a separate domain name to each user, and they can directly access files with their domain name

For example, now two users have saved the same file (test.png), but they are named differently,
User A is named: /static/image/test.png
User B named: /images/background.png
If bind prefix and redirection are supported
Then you can save the file as: /raw/test.png
Create an object for user A: /user_a/static/image/test.png
Create an object for user B: /user_b/images/background.png
All set [x-redirect-object=/raw/test.png] for them
Then bind the domain name [user1.domain.com] and prefix [/user_a] to user A
Bind the domain name [user2.domain.com] and prefix [/user_b] to user B

at this time
User A can use
user1.domain.com/static/image/test.png
User B can use
user2.domain.com/images/background.png
access object

and the object is only saved once

beelzabubOP•6/30/22, 3:50 PM

My job as a TL is to think through the consequences of decisions. I understand your use case is solved by the requirements you laid out but we don't build features just for 1 customer unless it's a very large ENT contract (& even then, maybe). I need to think through the problem comprehensively so that the feature is designed in a way that doesn't lead to a dead end or have sharp corner cases that surprise customers that aren't you in weird ways.

This is a cost reduction mechanism but only for the customers that can make use of it (& even then, storage is fairly cheap - I don't know how big of a cost reduction this is in practice).

AFAIK no other object storage system has something like this. That gives me pause and makes me want to make sure I've thought through every potential corner case & know how it should work.

beelzabubOP•6/30/22, 3:52 PM

A more general solution might even be to just optimize

CopyObject

CopyObject

so that copies don't count towards more storage space & reduce storage costs in a significantly more meaningful way. Then you don't even need symlinks. This is why I'm saying things need to be thought through.

hinego•6/30/22, 3:53 PM

It would be better if CopyObject was not billed

beelzabubOP•6/30/22, 3:55 PM

Anyway. My point wasn't to say we're going down a path. Just that there's a lot of nuance & options to think through before we pick a path.

hinego•6/30/22, 3:56 PM

ok thanks for your patience

beelzabubOP•6/30/22, 3:58 PM

And it's also possible we will choose to pick no paths & stick with existing primitives (i.e. not high value enough of a feature vs so many other things we need to go build)

kian•6/30/22, 4:03 PM

Not currently - if you was using a Worker for public access (or any kind of auth you run in a Worker) then you could DIY it up but that's about it at the moment

Bbeelzabub My job as a TL is to think through the consequences of decisions. I understand y...

zegevlier•6/30/22, 5:20 PM

Just curious: does TL stand for technical lead or something else?

Arnø•6/30/22, 8:54 PM

How can I limit the size of body in workers ? by default it's 100 mo but I want to limit to 4mo

kian•6/30/22, 8:55 PM

It's not a specific Workers limitation, it's your zone in general for anything proxied through your Cloudflare zone.

kian•6/30/22, 8:56 PM

You could do it in a firewall rule I guess?

beelzabubOP•6/30/22, 8:56 PM

Something like:

Number(request.headers.get('content-length')) >= 4 * 1024 * 1024

Number(request.headers.get('content-length')) >= 4 * 1024 * 1024

Just be careful as not all requests need to have a content-length. Firewall rule is probably cheaper than doing it in the worker

beelzabubOP•6/30/22, 8:56 PM

(I don't know how firewall billing works)

kian•6/30/22, 8:57 PM

Free - that's why I went with it haha. Could do it in a Worker but a request is a request as far as that billing side

kian•6/30/22, 8:57 PM

You will run into the same issue that you mentioned about a lack of content-length though

kian•6/30/22, 8:57 PM

I don't see any ruleset fields that have the body size sat as a nice value waiting for you

beelzabubOP•6/30/22, 8:58 PM

If those are coming into a worker handling R2 & it's not doing anything else, you can reject those requests anyway. The content length is required to be known

kian•6/30/22, 8:58 PM

That's a good point

Bbeelzabub Something like: ``` Number(request.headers.get('content-length')) >= 4 * 1024 *...

Arnø•6/30/22, 9:03 PM

ok thanks you

Bbeelzabub Something like: ``` Number(request.headers.get('content-length')) >= 4 * 1024 *...

asp•6/30/22, 9:13 PM

Cloudflare's web server (nginx-fl) buffers the entire request before sending it along, so all requests with a body should also have a Content-LengthContent-Length header. You won't see any requests with Transfer-Encoding: chunkedTransfer-Encoding: chunked in Workers. I believe the same is true for Firewall Rules.

kian•6/30/22, 9:19 PM

Blocks me if I don't include a content-lengthcontent-length header (i.e GETGET) - so will need to add the method in

kian•6/30/22, 9:21 PM

(http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT" }) and (not any(lower(http.request.headers.names[*])[*] contains "content-length") or any(http.request.headers["content-length"][*] > "4096"))

(http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT" }) and (not any(lower(http.request.headers.names[*])[*] contains "content-length") or any(http.request.headers["content-length"][*] > "4096"))

kian•6/30/22, 9:22 PM

Restrict POSTPOST and PUTPUT requests on fw-test.kian.org.ukfw-test.kian.org.uk to a Content-LengthContent-Length less than 40964096 (and block any that don't include it at all, will accept 00 for 0-length uploads)

kian•6/30/22, 9:23 PM

kian•6/30/22, 9:23 PM

Example POST - (domain goes nowhere, expected a 526)

Kkian ``` (http.host eq "fw-test.kian.org.uk" and http.request.method in { "POST" "PUT...

asp•6/30/22, 9:27 PM

I believe you can just do any(http.request.headers["content-length"][*] > 4194304)any(http.request.headers["content-length"][*] > 4194304). If there is no Content-LengthContent-Length header - because the body is empty - the expression will evaluate to false.

Edit: I am confusing myself

kian•6/30/22, 9:27 PM

Probably - ruleset engine hurts my brain

Kkian Probably - ruleset engine hurts my brain

asp•6/30/22, 9:28 PM

Well, yes, don't do ruleset when tired haha

kian•6/30/22, 9:29 PM

What’s the worst that’s gonna happen, find another input validation vuln?

kian•6/30/22, 9:29 PM

kian•6/30/22, 9:34 PM

I wish the firewall rules highlighted which condition you met - but that’s another story

Kkian What’s the worst that’s gonna happen, find another input validation vuln?

asp•6/30/22, 9:35 PM

Thursday night VULN discovery = Friday afternoon incident. So yes, that might just be the worst thing that can happen

Walshy•6/30/22, 9:37 PM

Pls no

Cyb3r-Jak3•6/30/22, 9:41 PM

@beelzabub (sorry for the ping) I don't think the S3 terraform is going to work. AWS uses STS for auth. When I try I get

error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: aec3d1d9-a733-499a-a35f-ab954b462626, api error InvalidClientTokenId: The security token included in the request is invalid.

error configuring Terraform AWS Provider: error validating provider credentials: error calling sts:GetCallerIdentity: operation error STS: GetCallerIdentity, https response error StatusCode: 403, RequestID: aec3d1d9-a733-499a-a35f-ab954b462626, api error InvalidClientTokenId: The security token included in the request is invalid.

Kkagitac and IIRC from when I explored this there is a "bug" in the provider (or R2 shoul...

Cyb3r-Jak3•6/30/22, 11:08 PM

It doesn't like no object lock.

error creating S3 Bucket (my-tf-test-bucket): NotImplemented: Header 'x-amz-bucket-object-lock-enabled' with value 'false' not implemented

error creating S3 Bucket (my-tf-test-bucket): NotImplemented: Header 'x-amz-bucket-object-lock-enabled' with value 'false' not implemented

beelzabubOP•7/1/22, 12:26 AM

Yeah, we're pretty strict about headers to avoid having to worry about problems as we add features

beelzabubOP•7/1/22, 12:27 AM

Which operation is x-amz-bucket-object-lock-enabledx-amz-bucket-object-lock-enabled on?

beelzabubOP•7/1/22, 12:29 AM

I'll add a carve-out for supporting

false

false

next week

I'm aware of the request for symlink or redirect support (this is the 3rd time you're asking). Seems

Similar Threads

Similar Threads

Similar Threads