Hi. I'm looking at moving from Google Cloud Storage to R2, primarily for the cost difference. Curren

Hi. I'm looking at moving from Google Cloud Storage to R2, primarily for the cost difference. Currently with Google we create a bucket per user but this isn't supported by default with R2 due to the inital bucket limit per account. I found this message (r2) stating that the current recommended way to achieve this in R2 is to use a UID prefix to create a "folder" per user however this presents issues when you want to delete a user's data and puts the burden of user usage collection on us.

Is this still the recommended way to structure user data in R2 and how likely are Cloudflare to increase the bucket limit for reasons such as this?

Also as I understand it at the moment there is no data replication with R2. Does this mean there is no protection against datacenter disasters?

JJon Robertson Hi. I'm looking at moving from Google Cloud Storage to R2, primarily for the cos...

Karew•1/26/24, 3:29 AM

Creating a bucket per user sounds like a bad idea in every blob storage service, not just R2. The prefixes are exactly what you should do. Deleting the user data should be easy since you can delete based on a prefix.

R2 does have disaster recovery internally, we don't have user-facing replication (like you can't have two buckets on your account that automatically copy each other)

Karew•1/26/24, 3:30 AM

R2 analytics can also be filtered on the prefix. If each user has a UUID, there would be no possibility of overlap

KKarew Creating a bucket per user sounds like a bad idea in every blob storage service,...

Jon RobertsonOP•1/26/24, 3:41 AM

That sounds perfect. I've not used the S3 API properly before. Is the only way to delete the files by prefix by using a lifecycle configuration? I couldn't see anything related to it on the standard delete object requests.

Original message was deleted

Jon RobertsonOP•1/26/24, 4:03 AM

Got it. Thanks for the help!

Erisa•1/26/24, 9:27 AM

Yes but you may need to add some code for it - https://developers.cloudflare.com/workers/reference/how-the-cache-works/

Erisa•1/26/24, 9:28 AM

For image resizing, check here https://developers.cloudflare.com/images/transform-images/transform-via-workers/

Erisa•1/26/24, 9:28 AM

Then on the dashboard, purchase the $0+usage plan of Cloudflare Images (which allows you to use your own storage)

AAnay-208 I'm using cloudflare r2. If I use images, only option I see after buying $0 plan...

Erisa•1/26/24, 9:33 AM

You use the $0 plan to enable Transforms and then use those in a Worker with your R2 as the storage, check this link again https://developers.cloudflare.com/images/transform-images/transform-via-workers/

AAnay-208 So if I'm right, is this how it'll work: - add a custom domain to worker and not...

Erisa•1/26/24, 9:46 AM

Correct

Erisa•1/26/24, 9:47 AM

You may need to implement Cache API within the worker however

Erisa•1/26/24, 9:47 AM

There is an example on that docs page: https://developers.cloudflare.com/images/transform-images/transform-via-workers/#an-example-worker

AAnay-208 Another question, so this is code of my worker function: ```js // Parse reques...

Erisa•1/26/24, 9:55 AM

Yes

AAnay-208 And with that, I'll get charged for $1/100000 images scaled, right?

Erisa•1/26/24, 9:59 AM

You get charged for unique transforms which according to https://developers.cloudflare.com/images/pricing/ is

$0.50 per 1K unique transformations per month - 5K transformations included

$0.50 per 1K unique transformations per month - 5K transformations included

Unique transformation being same image and same parameters, regardless of cache state

EErisa Its not a hard written limit and more of a "if you go too high, R2 starts to bre...

dave•1/26/24, 5:56 PM

is this documented somewhere? Surprised to see it missing from https://developers.cloudflare.com/r2/reference/limits/

Erisa•1/26/24, 6:10 PM

It's not

Unsmart•1/26/24, 6:10 PM

I'd assume its not there because its not a limit they impose directly its something that number of requests and concurrency can eventually break the system

UUnsmart I'd assume its not there because its not a limit they impose directly its someth...

dave•1/26/24, 7:29 PM

but doing like 1,000 concurrent puts on the same bucket seems like it'll always fail, no?

Unsmart•1/26/24, 7:33 PM

Always maybe right now but in the future not really. It might depend on size of files not sure

Original message was deleted

dave•1/27/24, 1:27 AM

Not yet. You could do a cron Worker that lists new files in a bucket and tracks it, but kinda a pain atm.

Original message was deleted

Hello, I’m Allie!•1/27/24, 12:57 PM

If you run all the uploads through a Worker, then you could just call a function there

Original message was deleted

Hello, I’m Allie!•1/27/24, 1:11 PM

Yes, it would just require a little bit of work

Original message was deleted

dave•1/27/24, 3:03 PM

You will be limited to 100MB on uploads to your Worker if you’re on the free or Pro plan.

Original message was deleted

dave•1/27/24, 3:04 PM

You could look into using gzip on your data, might save you a lot. I know for us, gzip saved us a ton on storing JSON in buckets.

Original message was deleted

dave•1/27/24, 3:05 PM

Ah, what runtime is doing the compression? Is it a Python script or something?

dave•1/27/24, 3:07 PM

If CPU time isn’t an issue, compressing with lzma should have higher compression.

Original message was deleted

dave•1/27/24, 3:08 PM

why not lz4?

dave•1/27/24, 3:09 PM

oh right I am using zstd for btrfs compression already

dave•1/27/24, 3:12 PM

If the download is going through a Worker, you can decompress it on the fly.

Ddave If the download is going through a Worker, you can decompress it on the fly.

dave•1/27/24, 3:13 PM

Or if it’s being viewed in a browser

Original message was deleted

dave•1/27/24, 3:20 PM

What’s annoying is that the etag won’t be the md5 hash of the entire file if you do chunked uploads.

AAnay-208 So do I’ve to use a worker for it then ?

dave•1/27/24, 4:24 PM

your Worker won't be able to accept POST'd 10GB file

Hi. I'm looking at moving from Google Cloud Storage to R2, primarily for the cost difference. Curren

Similar Threads

Similar Threads

Similar Threads