any way to replicate list objects functionality through a public url?

Nnathaniel any way to replicate list objects functionality through a public url?

K

Karew•10/20/23, 6:20 PM

It’s intentionally not offered AFAIK

K

Karew•10/20/23, 6:21 PM

The public domain is for serving individual files through the whole Cloudflare proxy/CDN with no S3 API in the way

K

Karew•10/20/23, 6:21 PM

If you keep track of your file uploads in a database or something though you can “list” them that way

N

nathanielOP•10/20/23, 6:22 PM

gotcha

M

Mehanika•10/20/23, 8:25 PM

How fast can you migrate data to R2? I am looking to migrate 10 million pictures, but looks like that might take a second or two. Anyone tried something similar?

Z

zegevlier•10/20/23, 8:27 PM

Is the data in S3?

Z

zegevlier•10/20/23, 8:27 PM

There's an automatic migration tool for that

M

Mehanika•10/20/23, 8:28 PM

It is. But I am more considering the time than the actual migration

C

Chaika•10/20/23, 8:32 PM

no idea on the exact time, but It may be worth pointing out you don't need to do it at all once, you can do it over time as things are requested "Sippy": https://developers.cloudflare.com/r2/data-migration/sippy/

Sippy (beta) · Cloudflare R2 docs

Sippy is a data migration service that allows you to copy data from other cloud providers to R2 as the data is requested, without paying unnecessary …

C

Chaika•10/20/23, 8:33 PM

I guess it's still in beta though, forgot about that

M

Mehanika•10/20/23, 8:35 PM

Taking a look. Thanks

M

Mehanika•10/20/23, 8:37 PM

@Chaika I see you are tagged with knowing about images, and cannot find out what other channel ask so hope you can assist me here. If i turn on Image transform. pictures with various sizes etc saved as variants or as new pictures? Documentation is very unclear and maybe you have experience?

MMehanika @Chaika I see you are tagged with knowing about images, and cannot find out what...

C

Chaika•10/20/23, 8:45 PM

lol those are just tags for things you're interested in, anyone can get them <id:customize> , although I think Images is gone because the team isn't here anymore/no channel

Are you talking about Cloudflare Images, Flexible Variants? My understanding is neither, they're not new variants or pictures. You just get charged per image delivered so the cost is the same, it's just dynamic size/settings, instead of static variants, ex: https://imagedelivery.net/DoE71rnH9coHrk8hcmlMVA/2ce9e4b4-3925-4250-3f0d-7327e72f1700/w=800,sharpen=3

CChaika lol those are just tags for things you're interested in, anyone can get them <id...

M

Mehanika•10/20/23, 8:47 PM

Oh, haha! Did not know that. Thank you for clarifying two things at once!
Appreciate it once again!

S

shep•10/21/23, 2:54 AM

hey so when i curl test my upload it works fine

S

shep•10/21/23, 2:54 AM

im thinking maybe my headers are causing issues

S

shep•10/21/23, 2:56 AM

PUT /pictures/ava/-cbcc-4f98-acaa-81c556b59712_227f8f9c-6da4-4430-8f6b-1c56c869e6e8.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=e45bd1f0b9630ed2055abb90de24%2F20231021%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20231021T025045Z&X-Amz-Expires=300&X-Amz-Signature=7a90fffd87b621c7f5a3fe2d03621beb5275fa8fc4aa3c75235892783aeaff9c&X-Amz-SignedHeaders=host&x-id=PutObject HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer jwtxx
Connection: keep-alive
Content-Length: 186746
Content-Type: image/jpeg
DNT: 1
Host: pictures.a88a0a0ec0a380338bfd981ac867c85a.r2.cloudflarestorage.com
Origin: http://localhost:5174
Referer: http://localhost:5174/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
sec-ch-ua: "Not=A?Brand";v="99", "Chromium";v="118"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"

PUT /pictures/ava/-cbcc-4f98-acaa-81c556b59712_227f8f9c-6da4-4430-8f6b-1c56c869e6e8.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=e45bd1f0b9630ed2055abb90de24%2F20231021%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20231021T025045Z&X-Amz-Expires=300&X-Amz-Signature=7a90fffd87b621c7f5a3fe2d03621beb5275fa8fc4aa3c75235892783aeaff9c&X-Amz-SignedHeaders=host&x-id=PutObject HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer jwtxx
Connection: keep-alive
Content-Length: 186746
Content-Type: image/jpeg
DNT: 1
Host: pictures.a88a0a0ec0a380338bfd981ac867c85a.r2.cloudflarestorage.com
Origin: http://localhost:5174
Referer: http://localhost:5174/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
sec-ch-ua: "Not=A?Brand";v="99", "Chromium";v="118"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"

PUT /pictures/ava/-cbcc-4f98-acaa-81c556b59712_227f8f9c-6da4-4430-8f6b-1c56c869e6e8.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=e45bd1f0b9630ed2055abb90de24%2F20231021%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20231021T025045Z&X-Amz-Expires=300&X-Amz-Signature=7a90fffd87b621c7f5a3fe2d03621beb5275fa8fc4aa3c75235892783aeaff9c&X-Amz-SignedHeaders=host&x-id=PutObject HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer jwtxx
Connection: keep-alive
Content-Length: 186746
Content-Type: image/jpeg
DNT: 1
Host: pictures.a88a0a0ec0a380338bfd981ac867c85a.r2.cloudflarestorage.com
Origin: http://localhost:5174
Referer: http://localhost:5174/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
sec-ch-ua: "Not=A?Brand";v="99", "Chromium";v="118"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"

PUT /pictures/ava/-cbcc-4f98-acaa-81c556b59712_227f8f9c-6da4-4430-8f6b-1c56c869e6e8.jpg?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Content-Sha256=UNSIGNED-PAYLOAD&X-Amz-Credential=e45bd1f0b9630ed2055abb90de24%2F20231021%2Fauto%2Fs3%2Faws4_request&X-Amz-Date=20231021T025045Z&X-Amz-Expires=300&X-Amz-Signature=7a90fffd87b621c7f5a3fe2d03621beb5275fa8fc4aa3c75235892783aeaff9c&X-Amz-SignedHeaders=host&x-id=PutObject HTTP/1.1
Accept: */*
Accept-Encoding: gzip, deflate, br
Accept-Language: en-US,en;q=0.9
Authorization: Bearer jwtxx
Connection: keep-alive
Content-Length: 186746
Content-Type: image/jpeg
DNT: 1
Host: pictures.a88a0a0ec0a380338bfd981ac867c85a.r2.cloudflarestorage.com
Origin: http://localhost:5174
Referer: http://localhost:5174/
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: cross-site
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/118.0.0.0 Safari/537.36
sec-ch-ua: "Not=A?Brand";v="99", "Chromium";v="118"
sec-ch-ua-mobile: ?0
sec-ch-ua-platform: "macOS"

S

shep•10/21/23, 2:56 AM

is there any issues with this?

S

shep•10/21/23, 2:57 AM

[
  {
    "AllowedOrigins": [
      "http://localhost:5174"
    ],
    "AllowedMethods": [
      "PUT",
      "POST",
      "GET",
      "DELETE",
      "HEAD"
    ],
    "AllowedHeaders": [
      "x-requested-by",
      "Content-Type"
    ],
    "ExposeHeaders": []
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:5174"
    ],
    "AllowedMethods": [
      "PUT",
      "POST",
      "GET",
      "DELETE",
      "HEAD"
    ],
    "AllowedHeaders": [
      "x-requested-by",
      "Content-Type"
    ],
    "ExposeHeaders": []
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:5174"
    ],
    "AllowedMethods": [
      "PUT",
      "POST",
      "GET",
      "DELETE",
      "HEAD"
    ],
    "AllowedHeaders": [
      "x-requested-by",
      "Content-Type"
    ],
    "ExposeHeaders": []
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:5174"
    ],
    "AllowedMethods": [
      "PUT",
      "POST",
      "GET",
      "DELETE",
      "HEAD"
    ],
    "AllowedHeaders": [
      "x-requested-by",
      "Content-Type"
    ],
    "ExposeHeaders": []
  }
]

K

kian•10/21/23, 3:12 AM

Why is there an Authorization header meowthinking

S

shep•10/21/23, 3:23 AM

true ill try removing it

K

kian•10/21/23, 3:23 AM

That’s the only thing that stands out personally

S

shep•10/21/23, 3:30 AM

yoooo

S

shep•10/21/23, 3:30 AM

it worked thanks!!

S

shep•10/21/23, 5:54 AM

what are the rate limits on r2?

S

shep•10/21/23, 5:55 AM

im using rclone to transfer a couple million 50kb files from gcp

⚡

⚡ techb0x 🌟•10/21/23, 4:41 PM

hello all, I am trying to download a exe file from a public bucket R2 from a powershell script, like

Invoke-WebRequest -Uri "https://xxxxxxxx" -OutFile "C:\Windows\Temp\xxxxxxx.exe"

Invoke-WebRequest -Uri "https://xxxxxxxx" -OutFile "C:\Windows\Temp\xxxxxxx.exe"

, but I get a "forbidden access", from a browser, it works, is there any CORS policies I need to setup ? can you give some advice ? thanks,

R

Raymond•10/21/23, 4:53 PM

I love sippy.

Sshep what are the rate limits on r2?

K

Karew•10/21/23, 7:39 PM

AFAIK there are no rate limits for uploading objects, at least none that rclone will run into

KKarew AFAIK there are no rate limits for uploading objects, at least none that rclone ...

H

Hello, I’m Allie!•10/21/23, 7:40 PM

Suffice to say I saturated my home fiber connection before

rclone

rclone

or

r2

r2

gave up

I

isaacryu•10/21/23, 10:28 PM

I'm running into the following problem trying to copy to an

r2

r2

bucket with

rclone

rclone

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

2023/10/21 17:52:57 ERROR : Attempt 1/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 2/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 ERROR : Attempt 3/3 failed with 1 errors and: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.
2023/10/21 17:52:57 Failed to copyto: InvalidParameter: 1 validation error(s) found.
- minimum field size of 1, HeadObjectInput.Key.

I

isaacryu•10/21/23, 10:28 PM

my config file:

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

[r2]
type = s3
provider = Cloudflare
access_key_id = ***
secret_access_key = ***
region = auto
endpoint = https://***.r2.cloudflarestorage.com/bucket-name

I

isaacryu•10/21/23, 10:29 PM

rclone copy File.zip r2:/

rclone copy File.zip r2:/

I

isaacryu•10/21/23, 10:29 PM

I tried

copyto

copyto

as well but that didn't work

I

isaacryu•10/21/23, 10:49 PM

i figured it out

I

isaacryu•10/21/23, 10:50 PM

endpoint needs to not have bucket name, remade my key with more permissiveness

I

isaacryu•10/21/23, 10:50 PM

that worked

KKarew AFAIK there are no rate limits for uploading objects, at least none that rclone ...

S

shep•10/22/23, 1:29 AM

yeah there wasnt got 300gb transfered superfast

S

SoulHarsh007•10/22/23, 7:37 PM

it looks like accessing R2 with @aws-sdk/client-s3 now requires using:

forcePathStyle: true

forcePathStyle: true

or else when using PUTObjectCommand, it yeilds:

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

SignatureDoesNotMatch: The request signature we calculated does not match the signature you provided. Check your secret access key and signing method.

The documentation example should be updated to reflect it, if this is actually the case. I can confirm using the forcePathStyle set to Force fixes the issue for me.

K

kian•10/22/23, 7:57 PM

It shouldn't have any difference for R2.

T

Tobias•10/22/23, 8:28 PM

Does R2 actually replicate the data globally? When creating a new bucket, I am prompted to select the region for the bucket (or it autoselects based on my location). The workload I have however requires the data to be available everywhere around the world with low latency

TTobias Does R2 actually replicate the data globally? When creating a new bucket, I am p...

U

Unsmart•10/22/23, 8:30 PM

Data is single region currently but you can use cache to enhance warm reads.

T

Tobias•10/22/23, 8:31 PM

Hmm okay

C

Chris•10/22/23, 8:47 PM

I have an issue with CORS in one of my r2 buckets. I currently have two r2 buckets: one for dev, one for prod. I set each up bucket with similar CORS policies so I can PUT object using presigned URL's into my bucket.

The one for dev looks like this and works. I can PUT from localhost or from

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com

https://dev.name-of-my-app.com just fine.

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://dev.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

C

Chris•10/22/23, 8:47 PM

..but I have almost the exact same policy for my prod bucket...

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "https://app.name-of-my-app.com"
    ],
    "AllowedMethods": [
      "PUT",
      "GET"
    ],
    "AllowedHeaders": [
      "*"
    ]
  }
]

...and I can PUT to a presigned URL from

localhost

localhost

localhost

localhost to the prod bucket just fine. BUT, when I try to PUT from

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com, I get a 403 CORS error like this...

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

Access to XMLHttpRequest at 'https://...r2.cloudflarestorage.com/path/to/uploaded/file.txt?X-Amz-Algorithm=...' from origin 'https://app.name-of-my-app.com' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource.

C

Chris•10/22/23, 8:48 PM

Does anyone have any idea what is going on? I don't understand why localhost works fine for prod, but

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com

https://app.name-of-my-app.com does not even though it's in the

AllowedOrigins

AllowedOrigins

AllowedOrigins

AllowedOrigins array. Appreciate any help y'all have.

K

kian•10/22/23, 8:48 PM

Look at the response body

C

Chris•10/22/23, 8:50 PM

Okay. Can you let me know what I'm looking for?

C

Chris•10/22/23, 8:51 PM

Appreciate the help.

K

kian•10/22/23, 8:52 PM

Just see if it has an error - i.e if it's

SignatureDoesNotMatch

SignatureDoesNotMatch

SignatureDoesNotMatch

SignatureDoesNotMatch then it isn't a failure due to CORS, it would have failed anyways.

any way to replicate list objects functionality through a public url?

Similar Threads

Similar Threads

Similar Threads