`<6>INFO : S3 bucket sdk-example: removing pending multipart upload for bucket "sdk-example" key "f

<6>INFO  : S3 bucket sdk-example: removing pending multipart upload for bucket "sdk-example" key "ferriswasm2.png" dated 2022-07-12 22:39:38.496 +0000 UTC (3m58.101980935s ago)

<6>INFO  : S3 bucket sdk-example: removing pending multipart upload for bucket "sdk-example" key "ferriswasm2.png" dated 2022-07-12 22:39:38.496 +0000 UTC (3m58.101980935s ago)

kianOP•7/12/22, 10:43 PM

kianOP•7/12/22, 10:43 PM

Awesomeeeeeeeee

kianOP•7/12/22, 10:51 PM

Before I move this into , are all of those query params now supported or are any still unimplemented?

beelzabub•7/12/22, 11:24 PM

I think we wanted to hold off on moving it to supported. Maybe it should have a

next to it since there may be bugs?

beelzabub•7/12/22, 11:25 PM

I know isn't correctly applied to all fields

kianOP•7/12/22, 11:30 PM

I'm not good at wording - , , blah - want to keep it concise like the other two.

kianOP•7/12/22, 11:30 PM

sounds like a branch or different endpoint - would probably be better or just the former

kianOP•7/12/22, 11:33 PM

The rest of the docs usage of just has at the end of it

beelzabub•7/12/22, 11:33 PM

I don't have strong opinions. I'll defer to Kate

beelzabub•7/12/22, 11:34 PM

Delimiter, prefix, and key-marker we can mark as supported

beelzabub•7/12/22, 11:34 PM

Same for max-uploads

beelzabub•7/12/22, 11:34 PM

Although technically the (hard to hit unless you're John) corner case bugs are when you do specify a delimiter so....

beelzabub•7/12/22, 11:35 PM

Actually I think even @john.spurlock didn't find the issues because they're bugs in

kianOP•7/12/22, 11:35 PM

If there's a bug I usually assume John or Albert found it

kianOP•7/12/22, 11:35 PM

beelzabub•7/12/22, 11:36 PM

This one was the developer who implemented it (but using John's test case)

John Spurlock•7/12/22, 11:38 PM

ListObjectsV1 ??? I don't know how to count that low

kianOP•7/12/22, 11:43 PM

When I first looked at Arq’s incompatibility like a month ago, I remember wondering why S3 and R2’s ListObjects responses were so different

kianOP•7/12/22, 11:43 PM

Eventually realised one was V1 and one was V2

kianOP•7/12/22, 11:43 PM

Not used to having to throw the query params in myself

Bbeelzabub Workers & R2 fall under the "Developer Platform" supplemental agreement in ToS. ...

Cat smoocher•7/12/22, 11:52 PM

That sorta clears it up, thanks! Still wrapping my head around the services. Regarding the Cloudflare Cache, doesn't it already cache default file types (including images) as mentioned here https://developers.cloudflare.com/cache/about/default-cache-behavior/, or am I mixing up the services

kianOP•7/12/22, 11:56 PM

Yes, no and maybe.

On a serious note, yeah. Cloudflare's 'cache' for any proxied records (or subrequest

fetch()

fetch()

calls in a Worker) caches based on file extension & media assets are included. The specific terminology in 2.8 of the self-serve terms is that a disproportionate amount of non-HTML content is not allowed.

If I have a blog, embedding a few images is no issue - maybe even a portfolio website might not be too bad if you're not caching the super high-res variants. But if I had a website dedicated to uploading or sharing images, effectively an image CDN, that would likely get the attention of sales.

Workers don't automatically use that cache outside of the subrequest

fetch()

fetch()

calls I mentioned - you would need to access it yourself using the

cache

cache

API. That's something you're going to want to avoid doing if you will be doing a lot of non-HTML storage/transfers with R2.

https://developers.cloudflare.com/workers/learning/how-the-cache-works/#interacting-with-the-cloudflare-cache

kianOP•7/12/22, 11:58 PM

Nothing in https://developers.cloudflare.com/r2/examples/demo-worker/ would prompt any cache usage, since there's no

fetch()

fetch()

calls (only using Workers bindings) and no

cache

cache

API usage.

kianOP•7/13/22, 12:07 AM

The fun, technical side that I might be completely wrong about is that Workers run before

so if I request your Worker, it will always invoke your Worker.

request to just never hits the cache 'layer'. If I do a

fetch

fetch

subrequest for (which goes through

cache

cache

origin

origin

) then I would hit the cache layer.

->

cache

cache

origin

origin

->

I can also access

cache

cache

specifically with the API (same cache namespace as

fetch

fetch

requests) or to open a new/existing namespaced cache.

I think custom domains (https://developers.cloudflare.com/workers/platform/routing/custom-domains/) might have cache in-front of them at some point but that's just wishful thinking.

Public buckets/domains for R2 might also do the same at some point in the future? geblobcatshrug

Cat smoocher•7/13/22, 12:16 AM

thanks for the explanation and demo @kian NOTED

! So I can use R2 to store images, serve them with my worker, but i should not manually invoke the cache API or a fetch() on those worker requests to r2 since that would cause some TOS issues while serving the content.

Since there's no caching allowed for that scenario, and since data we upload to r2 is only stored in one location, does that result in users in a different "region" experiencing higher latency than they would otherwise had that content been cached?

Cat smoocher•7/13/22, 12:19 AM

overall, just trying to figure out if r2 is right for my usecase, if I need caching or if im overthinking things, which i tend to do, lol PU_PeepoLaugh

kianOP•7/13/22, 12:19 AM

Buckets (and on the object-level I believe) is dictated by where the bucket was created, or object was uploaded.

So yeah, whilst the cache exists in every colo (all 350+ of them) and R2 has a lot fewer points of presence, there's lots of performance improvements on the way with R2.

kianOP•7/13/22, 12:19 AM

If most of your users are in the EU, then create bucket/upload objects from there & that fun stuff

kianOP•7/13/22, 12:20 AM

r2 is a good write-up by Vitali on the current 'status' of R2 and where it's at with durability/availability/performance with performance being the current focus

kianOP•7/13/22, 12:20 AM

kianOP•7/13/22, 12:20 AM

fwiw my betteruptime for Workers R2 bindings

kianOP•7/13/22, 12:22 AM

R2 is strongly consistent - you write, you read straight after, everyone sees the new value. That does make it a lot more difficult to have global replicas of data than something eventually consistent... like KV!

Workers KV is eventually consistent & cached (not the same cache we've been talking about), so that's good for write-once read-many scenarios if you're OK with global replication possibly taking 60 seconds or more. Cold reads might be around ~130ms, the same I'm getting from my R2 bucket, but once that key is 'warmed up' and in KV's cache then the reads are a lot faster at like ~20ms.

kianOP•7/13/22, 12:23 AM

They do have different use-cases though (aside from the consistency) - R2 is your normal blob object storage, with object sizes up to 5TB whereas KV is limited to 25MiB.

kianOP•7/13/22, 12:36 AM

Kate will know how to word it - Kate always knows

Kkian Buckets (and on the object-level I believe) is dictated by where the bucket was ...

Cat smoocher•7/13/22, 12:40 AM

For my usecase, i'd really just need to store and serve user uploaded images. The rest of my data lives in Google Firestore. I like the idea of not having to pay egress and all the other fees that S3/Google Storage would definitely require so r2 seemed like it could be a good fit. I did see that CF also has an Image service aPES_Think

If I just need to exclusively store/serve images, could that be a better solution? What are your thoughts?

Although given the improvements being planned for r2 and the response times you sent, it doesnt seem like the inability to "cache" is a big deal for me...maybe. I really dont know enough about CDN/caching to know if it's going to be a big enough impact to me. I dont have any benchmarks or real data since I'm still building my site out

kianOP•7/13/22, 12:44 AM

Images: fully managed image pipeline. Images are stored there and you can create variants for resizing, cropping, next-gen formats like WEBP and AVIF, compression. Like Imgix.

Pricing model: per image stored and per image delivered

Image Resizing: Similar to Images except your files can be stored wherever - IR just provides on-the-fly transformations like the above ones that Images can do.

Pricing model: per cache miss on a resized image, IR has its own cache.

KV: eventually consistent global key-value store up to 25MiB, like Redis - would serve your raw image

Pricing model: storage and operations, only realistically usable with a Worker so Worker requests

R2: strongly consistently object storage, like S3 or GCS - would serve your raw image

Pricing model: Ditto KV but public buckets are on the way

kianOP•7/13/22, 12:45 AM

Features like Polish will already compress and turn images into WEBP on your website - so maybe the transformations aren’t that valuable.

kianOP•7/13/22, 12:45 AM

It’s all up to you, and also planning for scaling - would you eventually allow arbitrary user uploads which might be big enough to need R2? Will you be OK managing KV for smaller files and R2 for bigger ones, or would you rather merge it?

kianOP•7/13/22, 12:57 AM

R2 also has a much more generous free tier if that matters to you

kianOP•7/13/22, 12:57 AM

KV has 1GB, 30k writes, 3.0mil reads (approx) and R2 has 10GB, 1.0mil writes and 10.0mil reads (approx) on their free tiers

Cat smoocher•7/13/22, 12:59 AM

you read my mind happycat

Price is probably one of the more important factors for me since I'm a team of 1 haha. That's what attracted me to r2. Images and the resizing functionality seemed really slick, but I think it could get expensive unexpectedly compared to r2.

kianOP•7/13/22, 1:00 AM

Images has a sourcing kit feature where you can import images from S3/R2 so you can always easily migrate in the future

Cat smoocher•7/13/22, 1:07 AM

good to know, thank you! I may start messing with r2 and CF Images both to see what feels better. I suppose I wont really know what solution fits me better until I get real traffic and users PES_Think

kianOP•7/13/22, 1:11 AM

Workers is free for 100,000 reqs a day and that’ll never exhaust the free tier (reads) of R2 - so stay under 10GB in R2 and it’ll be $0 overall

Cat smoocher•7/13/22, 1:19 AM

thanks for all the help PES_Happy

kianOP•7/13/22, 1:20 AM

Kkian I'm not good at wording - `Beta Feature Implemented`, `Feature Implemented in Be...

Karew•7/13/22, 2:00 AM

What about:

Feature Implemented

Feature In Beta (maybe buggy)

Not Implemented

beelzabub•7/13/22, 2:05 AM

I personally don't like "may be buggy". I feel like it delivers the wrong message. Bugs always exist & features we release generally have a pretty good quality bar already (even ListMultipartUploads). "Work in progress" or "Details in flux" is probably a better description of what's happening. The main change we're making is to get the upload ID to be stable (i.e. the same one as when you ). The next thing we're thinking about is whether has behaviors that are technically required but that no one actually cares about in practice. My hunch is the latter

Isaac McFadyen•7/13/22, 2:05 AM

How about:

Feature Implemented

Feature Implemented (may change before stable)

Not Implemented