Cloudflare Developers•2y ago

the s3 api isnt cached

Zzach the s3 api isnt cached

dave•3/11/24, 12:38 AM

I meant on the client side

zachOP•3/11/24, 12:39 AM

it isnt cached on the client side

zachOP•3/11/24, 12:39 AM

unless you do something like having a custom fetch adapter that stores it locally

zachOP•3/11/24, 12:39 AM

if you want caching on r2, use a public domain

zachOP•3/11/24, 12:39 AM

is there realistically a limit to the amount of objects you can have in a bucket?

Zzach is there realistically a limit to the amount of objects you can have in a bucket...

Karew•3/11/24, 4:51 AM

Not as far as we know

Ddave I don't think there's any way, but is it possible to have two presigned URLs be ...

Karew•3/11/24, 4:54 AM

Presigned URLs are not really cached anywhere. They are effectively HTTP requests waiting to be sent (also creating one does not hit the server, if that’s sort of what you’re working around)

hashdog•3/11/24, 4:57 AM

Please tell me, my primary domain name cannot be resolved to cloudflare, how can I access the content of r2 through the secondary domain name. For example, a.com is resolved to other service providers, and b.com is resolved to cloudflare. I hope to access the content of r2 through r2.a.com.

Hhashdog Please tell me, my primary domain name cannot be resolved to cloudflare, how can...

zachOP•3/11/24, 5:10 AM

you need to add the apex domain to your CF account, then add the domain as a public domain in r2 settings

Zzach you need to add the apex domain to your CF account, then add the domain as a pub...

hashdog•3/11/24, 5:16 AM

I cannot resolve the main domain name to the CF account. Can I use Custom Hostnames to achieve this?

zachOP•3/11/24, 5:16 AM

doubt it

zachOP•3/11/24, 5:17 AM

You'll need to onboard the domain you want onto cloudflare

Rasan II•3/11/24, 10:41 AM

I've got some videos uploaded on my r2 bucket on cloudflare, the links to those videos are sent to firebase, and then I retrieve them to my website

However, I find that sometimes the videos don't load, I'd say 7/10 load properly, and the other 3 do not load, and it is very random because as I refresh, every other time it's different ones that don't load

Is there any form of limiter that's on it that I do not know about ?

KKarew Presigned URLs are not really cached anywhere. They are effectively HTTP request...

dave•3/11/24, 10:47 AM

they are 100% cached in your client browser if you set the right headers. Try visiting this URL twice (don't reload, that'll force re-check, instead open a new tab), you'll notice a huge speed difference on the second visit. https://azure-di-source-files.6873d51718aa43a6389a532434554150.r2.cloudflarestorage.com/a97d0f7e5a329c86b7d4a5a68b039c12?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=a12b861ad3104f4719b3562466f6bb32%2F20240310%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20240310T170612Z&X-Amz-Expires=604800&X-Amz-SignedHeaders=host&X-Amz-Signature=da4fca7337bacc4bb0756abfe0a0fd16c3ba6d69620bc5db7c321a30f43ceb44

dave•3/11/24, 10:50 AM

the issue is, if I resign that URL, your client browser will treat it like a completely different cached object.

stars•3/11/24, 11:16 AM

hey @dave | ai.moda , i found out that you applied and get $150k azure credits from the microsoft startups founders hub. i have my startup website ready, we already launched few products, and everything is set up. i just want to know what you made for the demo video. did you record yourself showing the product or just create a product demo?

etya•3/11/24, 11:21 AM

Hi Team, Im trying to upload a large file with 6GB in size on cloudlfare through signed urls by breaking the files in chunks. After uploading few chunks the requests start to fail with 503 Service unavailable. How can i reolve this issue? Please help

danjenkins•3/11/24, 11:56 AM

Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I can't delete the bucket without the data all being gone.... theres a lot of files and directories so me deleting them isn't a short process.... Is it possible for someone from the Cloudflare team to do this internally?

Ddanjenkins Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I...

Hello, I’m Allie!•3/11/24, 11:58 AM

You can use https://rclone.org/commands/rclone_purge/ on a good connection with a high number of checkers, and it should run pretty fast. I had a bucket with a few tb of log files, and it finished it within 10 hours or so

rclone purge

Remove the path and all of its contents.

danjenkins•3/11/24, 11:58 AM

thanks @HardlyWorkin' I hadnt thought of using purge

asharpc•3/11/24, 12:05 PM

What is the parameter we need to send in order to complete multipart uploads in R2 using Boto3, I tried with "response = s3_client.complete_multipart_upload(
Bucket = AWS_S3_BUCKET_NAME,
Key = key,
UploadId = uplaod_id,
)"
But got error "botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well formed or did not validate against our published schema."

Could someone please explain how to complete multipart upload using Boto3.

mojo•3/11/24, 12:52 PM

Hey guys
I have a question for migration with r2 super slurper. Is there a possibility to pass a s3 region? Because I think thats why it doesn't work for me. The “Incremental migration” works with the same credentials but there I can pass a s3 region of the bucket.
Thanks for your help

Pprender ah so basically, if you apend your paypal or wahtever, you get 10GB, whatever co...

Chaika•3/11/24, 1:15 PM

Yes, R2 includes some free allocation like 10 GB/month - https://developers.cloudflare.com/r2/pricing/

I have another question. latency from r2 is quite high from 150ms - 800ms sometimes. is there any way to use tiered cache with a r2 bucket?

No, tiered cache can't be used at all, only normal cache. For GETs that's a bit high though, I'm in NY and to a bucket in enam I get ~160ms TTFB uncached pretty steadily, and of course cached it's just my latency to the local CF PoP (~40ms or so)

Chaika•3/11/24, 1:17 PM

well even with fetch and an r2 custom domain, tiered cache won't work. R2 Custom Domains just don't support tiered cache at all

Ggolyalpha I understand that, but.. If I'm running a service that relies on R2 custom domai...

arch•3/11/24, 2:31 PM

If you setup caching in front of it, less requests will land in R2. Plus, R2 bandwidth is free. So it'll take a lot of requests to start hitting your wallet (10 billion GetObject calls for free which if memory serves is triggered when fetching a resource).

Aarch If you setup caching in front of it, less requests will land in R2. Plus, R2 ban...

Unsmart•3/11/24, 2:46 PM

10 million not billion but yes

stachu•3/11/24, 2:49 PM

hi i'm having a problem uploading a file to my r2 bucket, when i do

rclone lsf r2:testin

rclone lsf r2:testin

it lists all the files correctly but when i try doing

rclone copy ~/video.mp4 r2:testin -P

rclone copy ~/video.mp4 r2:testin -P

it hits me with a 403 Access Denied.
here is my rclone.conf:

[r2]
type = s3
provider = Cloudflare
access_key_id = 
secret_access_key = 
region = auto
endpoint = 
acl = private

[r2]
type = s3
provider = Cloudflare
access_key_id = 
secret_access_key = 
region = auto
endpoint = 
acl = private

i don't have a lot of experience using rclone or r2 so i have no idea what could be causing this

stachu•3/11/24, 2:49 PM

the file is over 300mb but it should work since its outside of cloudflares ui thingy

stachu•3/11/24, 2:49 PM

also trying to upload a smaller file fails as well

Unsmart•3/11/24, 2:50 PM

Sounds like you only gave permissions to read not write

stachu•3/11/24, 2:50 PM

does it need the admin thing as well?

stachu•3/11/24, 2:51 PM

oh it does

stachu•3/11/24, 2:51 PM

well it works now

stachu•3/11/24, 2:51 PM

UUnsmart 10 million not billion but yes 😛

arch•3/11/24, 2:56 PM

Whooops

Sstachu oh it does

arch•3/11/24, 2:57 PM

It shouldn't :fry: I recently setup rclone and R2 myself. Did you make sure it had access to the specific bucket (or all of them)?

stachu•3/11/24, 2:58 PM

this is how it was configured

stachu•3/11/24, 2:59 PM

when i try setting it to the actual bucket it still errors out

stachu•3/11/24, 4:54 PM

that didn't work

stachu•3/11/24, 4:54 PM

plus only i will have access to the keys, i don't plan on sharing them with anyone

stachu•3/11/24, 5:01 PM

yea i guess, i can also just set the keys to the lowest permission level when im not using them if i can't find a solution for that problem

the s3 api isnt cached

Similar Threads

Similar Threads

Similar Threads