https://forum.duplicati.com/t/back-up-to-cloudflare-r2-storage-fails/15511/3

Duplicati

Somewhat, but with lots of missing functionality, maybe explaining those (501) Not Implemented. Duplicati default logging is mostly some light logging in job database (which gives an advantage of OS independence). One problem with failed backups is it doesn’t make a job log in the usual job logs, but server log at About → Show log → Stored usu...

FrerduroOP•4/3/24, 1:43 PM

¯\_(ツ)_/¯

FrerduroOP•4/3/24, 1:43 PM

its been open since 2022

Chaika•4/3/24, 1:44 PM

guessing it's the streaming auth?

FrerduroOP•4/3/24, 1:44 PM

who are you responding to?

FFrerduro who are you responding to?

Chaika•4/3/24, 1:46 PM

you

Chaika•4/3/24, 1:47 PM

they have this note about the aws-sdk

DisablePayloadSigning = true must be passed as Cloudflare R2 does not currently support the Streaming SigV4 implementation used by AWSSDK.S3.

Chaika•4/3/24, 1:47 PM

and you can see it seems to be trying to use that streaming sigv4 auth

Chaika•4/3/24, 1:48 PM

might be possible to disable it if they're just using the aws sdk, might not be something they expose though

FFrerduro Is anyone aware of any de-dupe backup software that supports R2?

Chaika•4/3/24, 1:50 PM

To respond to your original question, I use Kopia with R2 on my own PC. It's been working fine in my experience and has helped me recover files before when I accidentally replaced and only noticed it a month later.

CChaika they have this note about the aws-sdk > DisablePayloadSigning = true must be pas...

FrerduroOP•4/3/24, 1:52 PM

I dont see anything like that even in advanced options

FFrerduro Is anyone aware of any de-dupe backup software that supports R2?

dave•4/3/24, 2:09 PM

I wrote my own Worker that uses hashes for filenames.

SSam Luckily it states that it's during the beta. So should probably be higher when G...

dave•4/3/24, 2:10 PM

hopefully! I feel like I'm gonna have to end up doing a Cron Worker eventually anyway to handle edge cases.

Ddave hopefully! I feel like I'm gonna have to end up doing a Cron Worker eventually a...

Sam•4/3/24, 2:12 PM

But how would you write this worker? Wouldn't this worker have to go through thousands of files to verify the thing you want

SSam But how would you write this worker? Wouldn't this worker have to go through tho...

dave•4/3/24, 2:13 PM

I'd mostly be doing list objects to figure out what's missing. https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObjectsV2.html

ListObjectsV2 - Amazon Simple Storage Service

Returns some or all (up to 1,000) of the objects in a bucket with each request. You can use the request parameters as selection criteria to return a subset of the objects in a bucket. A 200 OK response can contain valid or invalid XML. Make sure to design your application to parse the contents of the response and handle it appropriately. For mor...

dave•4/3/24, 2:17 PM

but yeah 1,000 at a time is gonna suck...

Ddave but yeah 1,000 at a time is gonna suck...

Sam•4/3/24, 2:18 PM

Yup lmao

Sam•4/3/24, 2:19 PM

We're at 10 million files, which each need (not really, but nice to have) 3 variants. So In my case it's even more

Sam•4/3/24, 2:19 PM

Cloudflare images is just to expensive lmao

Original message was deleted

Sam•4/3/24, 2:22 PM

What do you use to map your bucket? Are there any easy to use programs for that?

Sam•4/3/24, 2:23 PM

Oh lmao

Sam•4/3/24, 2:25 PM

rclone basically executes the list object command over and over until all files are mapped?

Ddave hmm, is it just me or does the Infrequent Access pricing not really make sense?

Baby Beel•4/3/24, 2:26 PM

Especially when you compare it to Scaleway Glacier which is only 0.002$/Go, that's expensive for glacier.

Original message was deleted

itsmatteomanf•4/3/24, 2:27 PM

Yeah, it's more comparable to GCS Nearline

itsmatteomanf•4/3/24, 2:28 PM

Scaleway's Glacier is definitely cheaper, but also has multi-hour retrieval times.

Original message was deleted

Baby Beel•4/3/24, 2:28 PM

What's the difference?

kian•4/3/24, 2:28 PM

The blog doesn't talk about any minimum retention periods but I assume there will be some eventually

BBaby Beel What's the difference?

itsmatteomanf•4/3/24, 2:28 PM

The retrieval time. IA is instant, Glacier is up-to hours.

itsmatteomanf•4/3/24, 2:29 PM

The only Glacier type without retrieval times is GCS' Archive tier, which is cheap in storage, but very expensive in anything else.

Original message was deleted

Sam•4/3/24, 2:30 PM

Everything from AWS is

Original message was deleted

itsmatteomanf•4/3/24, 2:31 PM

GCS Archive is cheap storage wise. The issues arrive with retrieval, egress and request costs.

itsmatteomanf•4/3/24, 2:31 PM

It's cheaper than Scaleway's and it's instant retrieval.

itsmatteomanf•4/3/24, 2:31 PM

But you'll need a loan to pay for the retrieval fees themselves + egress, etc.

itsmatteomanf•4/3/24, 2:32 PM

Plus 1 year minimum retention.

Sam•4/3/24, 2:32 PM

For example for lambda, you need a lot more than just lambda. VPC, EC2, EFS, etc, etc. Just to get an outside connection, to download something from outside lambda.

Workers is so much better. Just hoping they bring an alternative to EFS to Workers.

Original message was deleted

Sam•4/3/24, 2:32 PM

Yeah that's just S3, right?

jazz•4/3/24, 2:34 PM

AWS is very cheap(in mentality):lul:

Baby Beel•4/3/24, 2:37 PM

How do you backup your r2?
Since there is no retention or immutable object.

Baby Beel•4/3/24, 2:38 PM

thanks, I'll have a look.

Original message was deleted

Sam•4/3/24, 2:38 PM

Isn't Glacier more for big companies that need to store for example files that are required to be stored for a certain period? Foe example in the Netherlands you're required to save all your invoices for about 5 years, in case the IRS needs them. Just push them in there and you're set.

Sam•4/3/24, 2:40 PM

Yeah

roni•4/3/24, 2:40 PM

Is there a way to protect R2 content with cookies?

Rroni Is there a way to protect R2 content with cookies?

Sam•4/3/24, 2:40 PM

You can via workers

roni•4/3/24, 2:41 PM

Any documentation how I can do that?

roni•4/3/24, 2:46 PM

Found it, thanks!

Original message was deleted

dave•4/3/24, 2:52 PM

not sure I follow how I could do a mapping if all files need to be synced

Venkat Dinavahi•4/3/24, 2:55 PM

Hey folks, the content I'm storing on R2 is highly compressible. Like a 500kb vs 15mb.
Will I want to gzip this myself or is there a way to let R2 take care of it for me? And if I don't store the data gzipped, will I be charged?

VVenkat Dinavahi Hey folks, the content I'm storing on R2 is highly compressible. Like a 500kb vs...

dave•4/3/24, 2:55 PM

you gotta upload it gzip'd yourself afaik, pretty easy to do with a Worker (that's what we do).

dave•4/3/24, 2:56 PM

make sure to set when uploading, that way CF can automatically decompress it for clients that don't support gzip too.

Venkat Dinavahi•4/3/24, 2:57 PM

Ah interesting... so it would get stored gzipped and downloading it this will happen transparently?

Venkat Dinavahi•4/3/24, 2:57 PM

Same if I fetch it using the R2 binding on the backend?