AFAIK there are no rate limits for uploading objects, at least none that rclone will run into

KKarew AFAIK there are no rate limits for uploading objects, at least none that rclone ...

H

Suffice to say I saturated my home fiber connection before

rclone

rclone

rclone

rclone or

r2

r2

r2

r2 gave up

I

isaacryu•10/21/23, 10:28 PM

I'm running into the following problem trying to copy to an

r2

r2

r2

r2 bucket with

rclone

rclone

rclone

rclone

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

I

isaacryu•10/21/23, 10:28 PM

my config file:

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

I

isaacryu•10/21/23, 10:29 PM

rclone copy File.zip r2:/

rclone copy File.zip r2:/

rclone copy File.zip r2:/

rclone copy File.zip r2:/

I

isaacryu•10/21/23, 10:29 PM

I tried

copyto

copyto

copyto

copyto as well but that didn't work

I

isaacryu•10/21/23, 10:49 PM

i figured it out

I

isaacryu•10/21/23, 10:50 PM

endpoint needs to not have bucket name, remade my key with more permissiveness

I

isaacryu•10/21/23, 10:50 PM

that worked

KKarew AFAIK there are no rate limits for uploading objects, at least none that rclone ...

S

shep•10/22/23, 1:29 AM

yeah there wasnt got 300gb transfered superfast

S

SoulHarsh007•10/22/23, 7:37 PM

it looks like accessing R2 with @aws-sdk/client-s3 now requires using:

forcePathStyle: true

forcePathStyle: true

forcePathStyle: true

forcePathStyle: true or else when using PUTObjectCommand, it yeilds:

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

The documentation example should be updated to reflect it, if this is actually the case. I can confirm using the forcePathStyle set to Force fixes the issue for me.

K

kian•10/22/23, 7:57 PM

It shouldn't have any difference for R2.

T

Tobias•10/22/23, 8:28 PM

Does R2 actually replicate the data globally? When creating a new bucket, I am prompted to select the region for the bucket (or it autoselects based on my location). The workload I have however requires the data to be available everywhere around the world with low latency

TTobias Does R2 actually replicate the data globally? When creating a new bucket, I am p...

U

Unsmart•10/22/23, 8:30 PM

Data is single region currently but you can use cache to enhance warm reads.

T

Tobias•10/22/23, 8:31 PM

Hmm okay

C

Chris•10/22/23, 8:47 PM

I have an issue with CORS in one of my r2 buckets. I currently have two r2 buckets: one for dev, one for prod. I set each up bucket with similar CORS policies so I can PUT object using presigned URL's into my bucket.

The one for dev looks like this and works. I can PUT from localhost or from

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com just fine.

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

C

Chris•10/22/23, 8:47 PM

..but I have almost the exact same policy for my prod bucket...

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

...and I can PUT to a presigned URL from

localhost

localhost

localhost

localhost to the prod bucket just fine. BUT, when I try to PUT from

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com, I get a 403 CORS error like this...

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

C

Chris•10/22/23, 8:48 PM

Does anyone have any idea what is going on? I don't understand why localhost works fine for prod, but

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com does not even though it's in the

AllowedOrigins

AllowedOrigins

AllowedOrigins

AllowedOrigins array. Appreciate any help y'all have.

K

kian•10/22/23, 8:48 PM

Look at the response body

C

Chris•10/22/23, 8:50 PM

Okay. Can you let me know what I'm looking for?

C

Chris•10/22/23, 8:51 PM

Appreciate the help.

K

kian•10/22/23, 8:52 PM

Just see if it has an error - i.e if it's

SignatureDoesNotMatch

SignatureDoesNotMatch

SignatureDoesNotMatch

SignatureDoesNotMatch then it isn't a failure due to CORS, it would have failed anyways.

C

Chris•10/22/23, 8:53 PM

I think the response body is empty? There are response headers, but if I try to copy the response body from the JS console, it's just empty.

K

kian•10/22/23, 8:54 PM

I wouldn't expect to see it in the browser because of CORS - if you copy it to cURL and run it there, you'll probably see a body.

C

Chris•10/22/23, 8:57 PM

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. </Message>
</Error>

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. </Message>
</Error>

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. </Message>
</Error>

<Error>
<Code>SignatureDoesNotMatch</Code>
<Message>The request signature we calculated does not match the signature you provided. Check your secret access key and signing method. </Message>
</Error>

C

Chris•10/22/23, 8:57 PM

....so that's something.

K

kian•10/22/23, 8:58 PM

Yeah - so the issue isn't with CORS, that request would have always failed.

C

Chris•10/22/23, 8:58 PM

That's good to know.

C

Chris•10/22/23, 8:58 PM

I'm not understanding why the bucket is cool with localhost but not cool with the app sending it from the browser.

C

Chris•10/22/23, 9:09 PM

...like, I could use a suggestion on how to troubleshoot this. Currently, from localhost, I point to the production app's server. The app server gives me a presigned PUT url. I put from localhost. It works. I go to

app.name-of-my-app.com

app.name-of-my-app.com

app.name-of-my-app.com

app.name-of-my-app.com , which is pointing to the same server as localhost, and when I PUT to the presigned URL, there is a 403.

C

Chris•10/22/23, 9:15 PM

...actually, it looks like I'm proxying to the server from localhost. Let me see if I can connect directly and see if that makes it break locally.

C

Chris•10/22/23, 9:33 PM

Okay. I was able to duplicate the error on localhost. Checked the server settings and didn't see any issues. Went to the Cloudflare R2 dashboard and rolled the API credentials for the prod bucket. Update them in the server. Everything works fine meow. Thanks for your help @kian!

E

EyalAmir•10/23/23, 12:59 AM

I have a question regarding

rclone

rclone

rclone

rclone. What's the time unit of

rclone link --expire?

rclone link --expire?

rclone link --expire?

rclone link --expire? Is it seconds?

EEyalAmir I have a question regarding `rclone`. What's the time unit of `rclone link --exp...

E

Erisa•10/23/23, 1:45 AM

Its in seconds by default. You can get days instead by putting

d

d at the end, for example

--expire 1d

--expire 1d

--expire 1d

--expire 1d becomes 86400 seconds (1 day) whereas

--expire 1

--expire 1

--expire 1

--expire 1 is 1 second

F

Flaviu•10/23/23, 5:31 PM

Hi, can somebody help me to configure cloudflare with nodejs multer to upload photo/video?

S

Sam•10/23/23, 7:10 PM

Not sure if I should ask it here, but if I enable Image Resizing (via Pro) and use this feature in combination with R2. Will I be additionally billed? Or will I only be billed for uncached Class B operations?

SSam Not sure if I should ask it here, but if I enable Image Resizing (via Pro) and u...

S

Sid•10/23/23, 7:28 PM

You are always billed only for requests that actually get to R2, so cached requests are never counted.

FFlaviu Hi, can somebody help me to configure cloudflare with nodejs multer to upload ph...

S

Sid•10/23/23, 7:29 PM

Can you describe what problem(s) you’re running into?

SSid You are always billed only for requests that actually get to R2, so cached reque...

S

Sam•10/23/23, 7:30 PM

Yeah aware of that, but curious if I will be billed extra when enabling Image Resizing (https://developers.cloudflare.com/images/image-resizing/).

Probably not the right chat for this, asking in #general-discussions

S

Sid•10/23/23, 7:33 PM

Does anyone here depend on R2’s public policy/custom domains endpoints (the same ones we use in the dashboard)? It’s technically undocumented, but I’m planning on making a few changes to it so I can document it in the future. Thought I’d ask here so I can help people migrate when these changes roll out instead of breaking their apps.

A

Austin B•10/23/23, 7:39 PM

Is it possible to trigger a Super Slurper migration via API? I haven't been able to find a doc reference for this, but wanted to double check.

SSid Does anyone here depend on R2’s public policy/custom domains endpoints (the same...

K

kian•10/23/23, 7:42 PM

I think @digitalpoint does

SSid Does anyone here depend on R2’s public policy/custom domains endpoints (the same...

D

digitalpoint•10/23/23, 7:48 PM

Ya, I do for a a couple things not otherwise available via normal APIs. Creating a bucket, assigning a public domain to it, etc. Basically allowing users to provision R2 without them needing to go digging and setting something up wrong. I’m not close to a computer atm to double check, but it think that’s the only undocumented stuff related to R2 I’m doing.

Ddigitalpoint Ya, I do for a a couple things not otherwise available via normal APIs. Creatin...

S

Sid•10/23/23, 7:49 PM

Gotcha, to be clear, I won’t just get rid of this overnight, but I’ll let you know here when I have a documented API for you to switch to. Thanks for letting me know Kian!

D

digitalpoint•10/23/23, 7:49 PM

Effectively the things that make this possible for them.

D

digitalpoint•10/23/23, 7:50 PM

Oh getting the list of buckets on the account too. But ya, I have no issues switching APIs when I can

D

digitalpoint•10/23/23, 7:52 PM

Speaking of API changes, is there somewhere we can monitor for changes? The recent changes to the bot management API (official one) last week was annoying. First heard about it when users are opening tickets asking why things are broken.

V

vitor•10/23/23, 11:16 PM

I'm keep getting these errors, does anyone know what's going on? This happens when I import { getSignedUrl } from "@aws-sdk/s3-request-presigner".

X

xjl•10/24/23, 10:09 AM

I upload a file to r2, and when I download the file, it carries the file source information inside, similar to this: WebKitFormBoundaryYkAc99eyn8dnIydt
Content-Disposition: form-data; name="file"; filename="工作交接.pdf"
Content-Type: application/pdf. Can someone help me with this

Xxjl I upload a file to r2, and when I download the file, it carries the file source ...

K

KarewOP•10/24/23, 10:15 AM

You are sending a FormData object to R2 instead of only the file data/binary data

AFAIK there are no rate limits for uploading objects, at least none that rclone will run into

Similar Threads

Similar Threads

Similar Threads