Cloudflare Developers•3y ago

that sucks

Mmany_insects that sucks

Sid•11/8/23, 2:03 PM

I’m working on cleaning up/documenting the endpoint that’ll let you do this. It is a couple of weeks away though.

many_insectsOP•11/8/23, 2:03 PM

ah nice, good to know it's on the roadmap

digitalpoint•11/8/23, 3:54 PM

You can do it with the undocumented endpoint if you need it immediately. Not sure if it's being changed as part of the API cleanup/documenting, so "use at your own risk".

SSid I’m working on cleaning up/documenting the endpoint that’ll let you do this. It ...

digitalpoint•11/8/23, 4:00 PM

I don't know if the stuff you are doing is just for R2 or not, but if it's broader scale, some general consistency with the Cloudflare API would be welcome. For example I can think of 5 different ways that boolean values are presented/used for writes (maybe more, I just use a subset of API calls and this is what I've run into). A true/positive can be one of these (not interchangably, just different for different endpoints):

1
"1"
true
"on"
{value: true}

Was maddening... I ended up defining the format for each setting and letting my class abstract/unabstract things internally.

Original message was deleted

Hello, I’m Allie!•11/8/23, 6:53 PM

You probably tripped a firewall rule. You can fine-tune those, or (basically) disable them entirely

cake•11/8/23, 9:52 PM

We are seeing that R2 API tokens / secrets are expiring, and we're getting aws.sdk.kotlin.services.s3.model.S3Exception: Unauthorizedaws.sdk.kotlin.services.s3.model.S3Exception: Unauthorized a lot. If we replace the token, it temporarily fixes it

cake•11/8/23, 9:52 PM

what's going on?

kian•11/9/23, 1:23 PM

It won’t have any effect

cube2222•11/9/23, 1:28 PM

Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare address, but I see the R2 devs are very active here so might be a better place to ask that part.

We're currently working on our stable provider/module registry, really mostly the JSON serving layer (not the artifacts). It seems like R2 would basically be perfect for that. However, it seems like by default R2 is single-region. Would you be able to make a recommendation on how to host a bunch of json files (1.6GB of json spread across a couple tens of thousands of files) in a multi-region way behind a custom domain?

I suppose it would be multiple single-region R2 buckets frontend by a CDN / Worker (and we'd just rclone the jsons to all the buckets), but I'm far from an expert at the CloudFlare offering and can't find much info online about this.

We have a public rfc up for this, fwiw, which contains a bit more background: https://github.com/opentofu/opentofu/issues/841

GitHub

[RFC] Pre-generating OpenTofu Registry v1 API Responses · Issue #84...

Introduction Note This proposal is part of the Homebrew-like Registry Design. This RFC introduces a strategy for pre-generating API responses for the OpenTofu registry using predefined JSON files a...

cube2222•11/9/23, 1:39 PM

(or I suppose another question - is the R2 "Europe" region really single-region, or is it replicated across a few geographically distributed regions across Europe)

Ccube2222 (or I suppose another question - is the R2 "Europe" region really single-region,...

Erisa•11/9/23, 1:41 PM

The latter, each region covers a broad area and data is distributed within that region - but thats intended for availability, and doesn't impact performance.
This old message is still true r2

cube2222•11/9/23, 1:42 PM

Awesome, thanks for the quick reponse @Erisa | Support Engineer !

Ccube2222 Hey, Kuba from OpenTofu here. I did write an email to the opensource cloudflare ...

Erisa•11/9/23, 1:45 PM

Regarding

Would you be able to make a recommendation on how to host a bunch of json files (1.6GB of json spread across a couple tens of thousands of files) in a multi-region way behind a custom domain?

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

I will also note though if your data is small enough (<25MB per file) and you are okay with writing a Worker, you may have better performance with KV

Erisa•11/9/23, 1:45 PM

Though... uploading the data to KV would not be as simple as R2

Erisa•11/9/23, 1:45 PM

So its a game of tradeoffs

Erisa•11/9/23, 1:46 PM

There is some guidance here on choosing a storage product: https://developers.cloudflare.com/workers/learning/storage-options/

Storage options guide · Cloudflare Workers docs

Storage options available on Cloudflare's developer platform that can be used with Workers.

cube2222•11/9/23, 1:54 PM

If public bucket access is fine you can do it without a Worker and just need to attach a custom domain and then add a Cache Rule to the zone with "eligible for cache" and a large Edge TTL.

That's very useful information!

cube2222•11/9/23, 1:55 PM

Though... uploading the data to KV would not be as simple as R2
So its a game of tradeoffs

Makes sense. We're not too latency sensitive here, so I think we'll start with a public R2, and if we need to, we can always improve on that.

cube2222•11/9/23, 1:55 PM

Thanks a lot for the help!

Erisa•11/9/23, 1:56 PM

No problem, hope it goes well and let us know if you have more questions

sin•11/9/23, 4:46 PM

Does anyone know the expiration time of S3 Presigned URLs? If the client has not completed the upload before the expiration, will the connection be interrupted? Or does this refer only to the start time of the request?

Ssin Does anyone know the expiration time of S3 Presigned URLs? If the client has not...

Erisa•11/9/23, 5:07 PM

Start time, ongoing connections are not interrupted and complete like normal.
The same is true for downloads; you can start downloading a file from presigned url and will keep downloading even if it expires halfway

Ccake We are seeing that R2 API tokens / secrets are expiring, and we're getting `aws....

Sid•11/9/23, 5:26 PM

That's to be expected no? Your token expired so API access was revoked? I may be misunderstanding

KKentaro [R2 API Token] I received "Access Denied" for S3 API when I use "Object Read & W...

Sid•11/9/23, 5:27 PM

What action are you issuing? Object tokens will not allow bucket-level configuration actions

nitsky•11/9/23, 5:28 PM

Hi, I have a REST API where a subset of the routes return objects from R2 which can be made publicly available. Rather than reading the objects from my API to serve those requests, I would like to serve them directly from Cloudflare with the public bucket feature. However, I still want all other routes to hit my API servers. Is this possible? Thanks!

Nnitsky Hi, I have a REST API where a subset of the routes return objects from R2 which ...

Sid•11/9/23, 5:31 PM

Note that making a bucket public by attaching a custom domain will make the entire bucket public. There currently isn't a way to keep certain prefixes private.
I'd recommend putting up a Worker (or using your existing server) in front of your bucket to decide if you want to serve a certain object

SSid Note that making a bucket public by attaching a custom domain will make the enti...

nitsky•11/9/23, 5:32 PM

In my case, everything can be public! I am hoping to take advantage of that to reduce costs.

Nnitsky In my case, everything can be public! I am hoping to take advantage of that to r...

Sid•11/9/23, 5:33 PM

Ah, in that case, have you looked into Page Rules on your domain?

Original message was deleted

Chaika•11/9/23, 7:37 PM

This is the channel for Cloudflare's R2 Object Storage. If you have a question about AWS billing, you should probably reach out to AWS, not much we can do here.