Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I can't delete the bu

Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I can't delete the bucket without the data all being gone.... theres a lot of files and directories so me deleting them isn't a short process.... Is it possible for someone from the Cloudflare team to do this internally?

Ddanjenkins Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I...

Hello, I’m Allie!•3/11/24, 11:58 AM

You can use https://rclone.org/commands/rclone_purge/ on a good connection with a high number of checkers, and it should run pretty fast. I had a bucket with a few tb of log files, and it finished it within 10 hours or so

rclone purge

Remove the path and all of its contents.

danjenkinsOP•3/11/24, 11:58 AM

thanks @HardlyWorkin' I hadnt thought of using purge

asharpc•3/11/24, 12:05 PM

What is the parameter we need to send in order to complete multipart uploads in R2 using Boto3, I tried with "response = s3_client.complete_multipart_upload(
Bucket = AWS_S3_BUCKET_NAME,
Key = key,
UploadId = uplaod_id,
)"
But got error "botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well formed or did not validate against our published schema."

Could someone please explain how to complete multipart upload using Boto3.

mojo•3/11/24, 12:52 PM

Hey guys
I have a question for migration with r2 super slurper. Is there a possibility to pass a s3 region? Because I think thats why it doesn't work for me. The “Incremental migration” works with the same credentials but there I can pass a s3 region of the bucket.
Thanks for your help

Pprender ah so basically, if you apend your paypal or wahtever, you get 10GB, whatever co...

Chaika•3/11/24, 1:15 PM

Yes, R2 includes some free allocation like 10 GB/month - https://developers.cloudflare.com/r2/pricing/

I have another question. latency from r2 is quite high from 150ms - 800ms sometimes. is there any way to use tiered cache with a r2 bucket?

No, tiered cache can't be used at all, only normal cache. For GETs that's a bit high though, I'm in NY and to a bucket in enam I get ~160ms TTFB uncached pretty steadily, and of course cached it's just my latency to the local CF PoP (~40ms or so)

Chaika•3/11/24, 1:17 PM

well even with fetch and an r2 custom domain, tiered cache won't work. R2 Custom Domains just don't support tiered cache at all

Ggolyalpha I understand that, but.. If I'm running a service that relies on R2 custom domai...

arch•3/11/24, 2:31 PM

If you setup caching in front of it, less requests will land in R2. Plus, R2 bandwidth is free. So it'll take a lot of requests to start hitting your wallet (10 billion GetObject calls for free which if memory serves is triggered when fetching a resource).

Aarch If you setup caching in front of it, less requests will land in R2. Plus, R2 ban...

Unsmart•3/11/24, 2:46 PM

10 million not billion but yes

stachu•3/11/24, 2:49 PM

hi i'm having a problem uploading a file to my r2 bucket, when i do rclone lsf r2:testinrclone lsf r2:testin it lists all the files correctly but when i try doing rclone copy ~/video.mp4 r2:testin -Prclone copy ~/video.mp4 r2:testin -P it hits me with a 403 Access Denied.
here is my rclone.conf:

[r2]
type = s3
provider = Cloudflare
access_key_id = 
secret_access_key = 
region = auto
endpoint = 
acl = private

[r2]
type = s3
provider = Cloudflare
access_key_id = 
secret_access_key = 
region = auto
endpoint = 
acl = private

i don't have a lot of experience using rclone or r2 so i have no idea what could be causing this

stachu•3/11/24, 2:49 PM

the file is over 300mb but it should work since its outside of cloudflares ui thingy

stachu•3/11/24, 2:49 PM

also trying to upload a smaller file fails as well

Unsmart•3/11/24, 2:50 PM

Sounds like you only gave permissions to read not write

stachu•3/11/24, 2:50 PM

does it need the admin thing as well?

stachu•3/11/24, 2:51 PM

oh it does

stachu•3/11/24, 2:51 PM

well it works now

stachu•3/11/24, 2:51 PM

UUnsmart 10 million not billion but yes 😛

arch•3/11/24, 2:56 PM

Whooops

Sstachu oh it does

arch•3/11/24, 2:57 PM

It shouldn't :fry: I recently setup rclone and R2 myself. Did you make sure it had access to the specific bucket (or all of them)?

stachu•3/11/24, 2:58 PM

this is how it was configured

stachu•3/11/24, 2:59 PM

when i try setting it to the actual bucket it still errors out

stachu•3/11/24, 4:54 PM

that didn't work

stachu•3/11/24, 4:54 PM

plus only i will have access to the keys, i don't plan on sharing them with anyone

stachu•3/11/24, 5:01 PM

yea i guess, i can also just set the keys to the lowest permission level when im not using them if i can't find a solution for that problem

stachu•3/11/24, 5:01 PM

then even if someone does get a hold of the key, they can't do much with it

asharpc•3/12/24, 3:38 AM

What is the parameter we need to send in order to complete multipart uploads in R2 using Boto3, I tried with "response = s3_client.complete_multipart_upload(
Bucket = AWS_S3_BUCKET_NAME,
Key = key,
UploadId = uplaod_id,
)"
But got error "botocore.exceptions.ClientError: An error occurred (MalformedXML) when calling the CompleteMultipartUpload operation: The XML you provided was not well formed or did not validate against our published schema." I posted the same question yesterday, but no one has responded.

kian•3/12/24, 6:13 AM

https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/client/complete_multipart_upload.html

kian•3/12/24, 6:14 AM

You can see it in https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html

kian•3/12/24, 6:14 AM

You're not providing parts.

jack•3/12/24, 6:21 AM

Hi! I'm currently using R2 for asset storage for a project of mine. For a while I was using a worker to serve the images, but realized I could switch to R2 public access to get costs down. I've now done this, but I'm not sure if I can cancel my workers paid subscription? It's not very clear on whether R2 is only available to the workers paid plan. I remember using it before purchasing paid, but the documentation says it's "Available on Workers Paid"

kian•3/12/24, 7:08 AM

You can cancel Workers Paid if you don't need it anymore.

kian•3/12/24, 7:08 AM

R2 is a separate product and plan.

kian•3/12/24, 7:09 AM

I see what you mean about the documentation, I'll raise that.

jack•3/12/24, 7:18 AM

Thank you!

Kkian <https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3/c...

asharpc•3/12/24, 8:14 AM

Can you tell me where I can get Etag to create parts?

kian•3/12/24, 8:17 AM

Look at the response headers when you upload the parts.

asharpc•3/12/24, 8:19 AM

Got it, Thanks

Nilesh Darji•3/12/24, 11:27 AM

Hi,
I had weird error while fetching object from public bucket,

open this first: https://assets.storzapp.com/63cca811-70c3-4080-9b5c-0a96623ee5c6/otherImage/training_video_img.png
then this: https://assets.storzapp.com/63cca811-70c3-4080-9b5c-0a96623ee5c6%2FotherImage%2Ftraining_video_img.png

when i open first url it gives error 404, second url works fine which has encoded prefix of object

some times second url also gives 404
when second url opened before opening first then first url work fine (in incognito mode)

Nilesh Darji•3/12/24, 11:57 AM

i encountered this error multiple times, i tried in different machine also

NNilesh Darji i encountered this error multiple times, i tried in different machine also

Hello, I’m Allie!•3/12/24, 11:59 AM

Just to check, is the response being cached? You should be able to see this in the Browser Network Panel

Nilesh Darji•3/12/24, 12:06 PM

in network panel it shows 404 status code

NNilesh Darji in network panel it shows 404 status code

Hello, I’m Allie!•3/12/24, 12:07 PM

Do you see any cache headers, or any indication that it isn't actually hitting the origin?

Could a cloudflare team member give me a hand deleting 1TB of data from R2.... I can't delete the bu

Similar Threads

Similar Threads

Similar Threads