@R2 Object Storage - https://blog.cloudflare.com/r2-events-gcs-migration-infrequent-access

The Cloudflare Blog

R2 adds event notifications, support for migrations from Google Clo...

We’re excited to announce three new features for Cloudflare R2: event notifications, support for migrations from Google Cloud Storage, and an infrequent access storage tier

Raymond•4/3/24, 1:03 PM

Yes!

! Sereio do mar•4/3/24, 1:03 PM

Sim

boywonder350•4/3/24, 1:15 PM

Finally event notifications! I now can enable uploading to r2 directly from browser and not needing to proxy through my worker woooo great job

EErisa @R2 Object Storage - https://blog.cloudflare.com/r2-events-gcs-migration-infrequ...

AndyJessop•4/3/24, 1:19 PM

Really cool! I was in a thread the other day where this was cited as the reason to use S3 over R2. Well, not any more!

dave•4/3/24, 1:22 PM

yesssssss event notifications!!

AAndyJessop Really cool! I was in a thread the other day where this was cited as the reason ...

ErisaOP•4/3/24, 1:24 PM

Very timely! Great to hear

Frerduro•4/3/24, 1:25 PM

why do beta signups always want us to put personal info in a google form?

dave•4/3/24, 1:26 PM

hmm, is it just me or does the Infrequent Access pricing not really make sense?

dave•4/3/24, 1:26 PM

it's only $0.01 vs $0.015

Unsmart•4/3/24, 1:27 PM

It only makes sense if you have petabytes of data and never access it basically

FFrerduro why do beta signups always want us to put personal info in a google form?

Frerduro•4/3/24, 1:27 PM

why not use the beta signups feature that already exists in the CF dashboard?

dave•4/3/24, 1:28 PM

If your workload produces more than 400 notifications per second, messages may be dropped.

Ouch.

Unsmart•4/3/24, 1:29 PM

That’s very rough :NotLikeThis:

dave•4/3/24, 1:29 PM

that makes trusting this kinda hard, since I'll have to make a cron Worker to scan every so often for events that got dropped.

Frerduro•4/3/24, 1:33 PM

no b2

Frerduro•4/3/24, 1:33 PM

:pepe_cry:

Original message was deleted

Frerduro•4/3/24, 1:36 PM

true but I still have the issue of our backup software not supporting R2

Ddave that makes trusting this kinda hard, since I'll have to make a cron Worker to sc...

Sam•4/3/24, 1:37 PM

Luckily it states that it's during the beta. So should probably be higher when GA

Frerduro•4/3/24, 1:38 PM

Is anyone aware of any de-dupe backup software that supports R2?

FFrerduro true but I still have the issue of our backup software not supporting R2

Chaika•4/3/24, 1:39 PM

your backup software doesn't support S3?

Frerduro•4/3/24, 1:39 PM

it does but it breaks with R2

Frerduro•4/3/24, 1:39 PM

I tried many times to get it to work

FFrerduro it does but it breaks with R2

Sam•4/3/24, 1:39 PM

Also experienced this at a platform for my database backups. Using Digital Ocean spaces instead, until they get R2 to work.

Chaika•4/3/24, 1:40 PM

hmm, are you trying bucket scoped tokens or any bucket ones? iirc some apps still break on bucket scoped

CChaika hmm, are you trying bucket scoped tokens or any bucket ones? iirc some apps stil...

Frerduro•4/3/24, 1:40 PM

Not sure what it uses but we use duplicati

Frerduro•4/3/24, 1:40 PM

it supports B2 & S3

Chaika•4/3/24, 1:40 PM

well it's whatever you create. When you make the R2 Token, you either scope it to a specific bucket, or all buckets

Frerduro•4/3/24, 1:41 PM

yeah I tried that iirc

Frerduro•4/3/24, 1:41 PM

been a few months now

Frerduro•4/3/24, 1:42 PM

https://github.com/duplicati/duplicati/issues/4673

GitHub

Add support for cloudflare R2 · Issue #4673 · duplicati/duplicati

I have searched open and closed issues for duplicates. I have searched the forum for related topics. Environment info Duplicati version: latest Operating system: Fedora Backend: Cloudflare R2 Descr...

Frerduro•4/3/24, 1:43 PM

https://forum.duplicati.com/t/back-up-to-cloudflare-r2-storage-fails/15511/3

Duplicati

Back Up to Cloudflare R2 Storage Fails

Somewhat, but with lots of missing functionality, maybe explaining those (501) Not Implemented. Duplicati default logging is mostly some light logging in job database (which gives an advantage of OS independence). One problem with failed backups is it doesn’t make a job log in the usual job logs, but server log at About → Show log → Stored usu...

Frerduro•4/3/24, 1:43 PM

¯\_(ツ)_/¯

Frerduro•4/3/24, 1:43 PM

its been open since 2022

Chaika•4/3/24, 1:44 PM

guessing it's the streaming auth?

Frerduro•4/3/24, 1:44 PM

who are you responding to?

FFrerduro who are you responding to?

Chaika•4/3/24, 1:46 PM

you

Chaika•4/3/24, 1:47 PM

they have this note about the aws-sdk

DisablePayloadSigning = true must be passed as Cloudflare R2 does not currently support the Streaming SigV4 implementation used by AWSSDK.S3.

Chaika•4/3/24, 1:47 PM

and you can see it seems to be trying to use that streaming sigv4 auth

Chaika•4/3/24, 1:48 PM

might be possible to disable it if they're just using the aws sdk, might not be something they expose though

FFrerduro Is anyone aware of any de-dupe backup software that supports R2?

Chaika•4/3/24, 1:50 PM

To respond to your original question, I use Kopia with R2 on my own PC. It's been working fine in my experience and has helped me recover files before when I accidentally replaced and only noticed it a month later.

CChaika they have this note about the aws-sdk > DisablePayloadSigning = true must be pas...

Frerduro•4/3/24, 1:52 PM

I dont see anything like that even in advanced options

FFrerduro Is anyone aware of any de-dupe backup software that supports R2?

dave•4/3/24, 2:09 PM

I wrote my own Worker that uses hashes for filenames.

SSam Luckily it states that it's during the beta. So should probably be higher when G...

dave•4/3/24, 2:10 PM

hopefully! I feel like I'm gonna have to end up doing a Cron Worker eventually anyway to handle edge cases.

Ddave hopefully! I feel like I'm gonna have to end up doing a Cron Worker eventually a...

Sam•4/3/24, 2:12 PM

But how would you write this worker? Wouldn't this worker have to go through thousands of files to verify the thing you want

SSam But how would you write this worker? Wouldn't this worker have to go through tho...

dave•4/3/24, 2:13 PM

I'd mostly be doing list objects to figure out what's missing. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

ListObjectsV2 - Amazon Simple Storage Service

Returns some or all (up to 1,000) of the objects in a bucket with each request. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. A 200 OK response can contain valid or invalid XML. Make sure to design your application to parse the contents of the response and handle it appropriately. For mor...

dave•4/3/24, 2:17 PM

but yeah 1,000 at a time is gonna suck...

Ddave but yeah 1,000 at a time is gonna suck...

Sam•4/3/24, 2:18 PM

Yup lmao

Sam•4/3/24, 2:19 PM

We're at 10 million files, which each need (not really, but nice to have) 3 variants. So In my case it's even more

Sam•4/3/24, 2:19 PM

Cloudflare images is just to expensive lmao

Original message was deleted

Sam•4/3/24, 2:22 PM

What do you use to map your bucket? Are there any easy to use programs for that?