Anyone have any tips for how to increase large-file transfer speeds `rclone -> R2`? I seem to rememb

Anyone have any tips for how to increase large-file transfer speeds

rclone -> R2

rclone -> R2

? I seem to remember there was a way to up parallelization, but I don't remember what the config was...

Kkian with files you don't care about since this has no auth

itsmatteomanf•2/20/23, 3:43 PM

Yeah, I'd need to reconfigure it, it's private data (albeit client encrypted on the one I have now)

itsmatteomanf•2/20/23, 3:44 PM

Let's move to DMs, @kiannh

kian•2/20/23, 3:46 PM

@sdnts I'll test it with Matteo & let you know what patterns I see

I can probably use fast-xml-parser to modify the responses & see what satisfies Synology

kian•2/20/23, 3:46 PM

johtso•2/20/23, 3:47 PM

Does CORS take time to apply?

Sid•2/20/23, 3:47 PM

Oh I'll really appreciate that, thanks Kian and Matteo!

Jjohtso Does CORS take time to apply?

Sid•2/20/23, 3:47 PM

Nothing more than a few seconds

HHello, I’m Allie!Anyone have any tips for how to increase large-file transfer speeds `rclone -> R...

Hello, I’m Allie!OP•2/20/23, 3:48 PM

Does R2 support anything above Upload Concurrency 2?

HHello, I’m Allie!Does R2 support anything above Upload Concurrency 2?

Sid•2/20/23, 3:49 PM

It should now! I've done tests with up to 70

kian•2/20/23, 3:49 PM

iirc the notice about multipart concurrency was removed awhile ago

kian•2/20/23, 3:49 PM

yeah that

SSid It should now! I've done tests with up to 70

Hello, I’m Allie!OP•2/20/23, 3:49 PM

Weird. Mine freezes above 2. This is it with 3:

Hello, I’m Allie!OP•2/20/23, 3:50 PM

With two it runs at about 2 MB/s, but it runs

Hello, I’m Allie!OP•2/20/23, 3:50 PM

And my current connection is rated for about 400 MB/s, so it shouldn't be a network bottleneck...

SSid Nothing more than a few seconds

johtso•2/20/23, 3:51 PM

Strange.. I have my bucket public and exposed via a subdomain, and I then added my main domain as the allowed domain in the CORS settings, but I'm still getting CORS errors

Sid•2/20/23, 3:51 PM

It just... doesn't make any progress? I'd expect you to see 429s or something if it is too high..

Jjohtso Strange.. I have my bucket public and exposed via a subdomain, and I then added ...

Hello, I’m Allie!OP•2/20/23, 3:52 PM

Actually, same on this one. Does it matter that I changed the config through the Dash?

SSid It just... doesn't make any progress? I'd expect you to see 429s or something if...

Hello, I’m Allie!OP•2/20/23, 3:52 PM

No errors or anything. It just gets stuck

Sid•2/20/23, 3:52 PM

Someone yesterday was saying the same thing. CORS config is eventually consistent but it shouldn't take too long

SSid Someone yesterday was saying the same thing. CORS config _is_ eventually consist...

Hello, I’m Allie!OP•2/20/23, 3:54 PM

It's been about a day for me: https://pub-0177eb9df0874e6e892a9e84263b9075.r2.dev/sprites.json. Just to make sure though, it should send even if it isn't an request, right?

HHello, I’m Allie!No errors or anything. It just gets stuck

Sid•2/20/23, 3:54 PM

Try adding a argument to see what it's doing?

HHello, I’m Allie!It's been about a day for me: https://pub-0177eb9df0874e6e892a9e84263b9075.r2.de...

DaniFoldi•2/20/23, 3:55 PM

Same here, I've added an origin yesterday and cors headers are still not present on the actual response

SSid Try adding a `-vv` argument to see what it's doing?

Hello, I’m Allie!OP•2/20/23, 3:55 PM

johtso•2/20/23, 3:57 PM

I tested the url here and it seems to be fine https://cors-test.codehappy.dev/?url=https://data.infinitenovelty.com/db.4.sqlite&origin=https://infinitenovelty.com&method=get

johtso•2/20/23, 3:58 PM

but when actually trying to load the file on my site it gives the CORS error.. https://infinitenovelty.com

johtso•2/20/23, 3:58 PM

HHello, I’m Allie!It's been about a day for me: https://pub-0177eb9df0874e6e892a9e84263b9075.r2.de...

Sid•2/20/23, 4:00 PM

It should, yep

Jjohtso I tested the url here and it seems to be fine https://cors-test.codehappy.dev/?u...

Sid•2/20/23, 4:02 PM

Yeah on that CORS testing site I see a warning, so this does look like an issue

johtso•2/20/23, 4:03 PM

isn't that just saying that I have an explicitly allowed domain?

johtso•2/20/23, 4:03 PM

which is fine, I have one domain where the resource will be used

Sid•2/20/23, 4:05 PM

Yeah NVM I'm an idiot, I thought this meant that this header was missing (when I should have read the line right above the code block)

johtso•2/20/23, 4:07 PM

Ah, figured it out, I needed to allow HEAD!

Sid•2/20/23, 4:10 PM

For others who were seeing similar CORS issues, I'm guessing y'alls config is correct, right?

HHello, I’m Allie!It's been about a day for me: https://pub-0177eb9df0874e6e892a9e84263b9075.r2.de...

Sid•2/20/23, 4:10 PM

This one seems to be working fine for me as well now!

SSid For others who were seeing similar CORS issues, I'm guessing y'alls config is co...

johtso•2/20/23, 4:23 PM

the problem is, devtools doesn't tell you what kind of request is being made if there is a CORS error

Sid•2/20/23, 4:23 PM

The Network tab should, no?

johtso•2/20/23, 4:24 PM

kian•2/20/23, 4:25 PM

One of the queries that Synology makes results in a loop - forever

SSid This one seems to be working fine for me as well now!

Hello, I’m Allie!OP•2/20/23, 4:26 PM

Here's what I get for headers, live, with cache disabled locally:

Is it maybe that it is cached without the CORS headers?

Sid•2/20/23, 4:28 PM

So CORS rules aren't cached, they're distributed globally like all other CF config, and so we don't promise strong consistency there

SSid So CORS rules aren't _cached_, they're distributed globally like all other CF co...

Hello, I’m Allie!OP•2/20/23, 4:29 PM

No, I meant that it was cached on the Response via the CF Cache. But now I don't remember if R2 public domains even have cache...

Sid•2/20/23, 4:30 PM

They don't, out of the box. That's one of the reasons why we encourage you to use a custom domain, so you can configure whatever makes sense to your app

Ggaikwmsm Hi everyone, I have a few questions about R2. Thank you in advance for your assi...

Hello, I’m Allie!OP•2/20/23, 4:33 PM

#1: Not at the moment
#2: Caching is free on Cloudflare. The max size for a cached object is determined by the plan that the custom domain has.

Ggaikwmsm Anyone with any insight on this would be greatly appreciated

Sid•2/20/23, 4:34 PM

Also see https://developers.cloudflare.com/workers/platform/limits/#cache-api

Limits · Cloudflare Workers docs

You can request adjustments to limits that conflict with your project goals by contacting Cloudflare. To increase a limit, complete the Limit Increase …

Ggaikwmsm thanks

Hello, I’m Allie!OP•2/20/23, 4:34 PM

Happy to help!

Ggaikwmsm thanks

Hello, I’m Allie!OP•2/20/23, 4:47 PM

Just to clarify, because I misspoke, the max size for a cached object is determined by the plan that the custom domain has, and it isn't applied across your entire account.