Is there any documentation that says what IAM rules/roles are needed to enable sippy on a r2 bucket

Mmadelaney Is there any documentation that says what IAM rules/roles are needed to enable s...

H

harshil1712•12/2/24, 5:00 PM

You can find it here: https://developers.cloudflare.com/r2/data-migration/sippy/#amazon-s3

Cloudflare Docs

Sippy is a data migration service that allows you to copy data from other cloud providers to R2 as the data is requested, without paying unnecessary cloud egress fees typically associated with moving large amounts of data.

M

madelaneyOP•12/2/24, 5:03 PM

Thanks @harshil1712 . What about on the CloudFlare side? I have a valid API token (I verified via

curl https://api.cloudflare.com/client/v4/user/tokens/verify

curl https://api.cloudflare.com/client/v4/user/tokens/verify

curl https://api.cloudflare.com/client/v4/user/tokens/verify

curl https://api.cloudflare.com/client/v4/user/tokens/verify), but when I try to call

https://api.cloudflare.com/client/v4/accounts/<id>/r2/buckets/<bucket name>/sippy

https://api.cloudflare.com/client/v4/accounts/<id>/r2/buckets/<bucket name>/sippy

https://api.cloudflare.com/client/v4/accounts/<id>/r2/buckets/<bucket name>/sippy

https://api.cloudflare.com/client/v4/accounts/<id>/r2/buckets/<bucket name>/sippy, I'm getting an authetnication error.

M

madelaneyOP•12/2/24, 5:05 PM

I am resource scoping the api token to

com.cloudflare.edge.r2.bucket.<account id>_<jurisdiction>_<bucket name>

com.cloudflare.edge.r2.bucket.<account id>_<jurisdiction>_<bucket name>

com.cloudflare.edge.r2.bucket.<account id>_<jurisdiction>_<bucket name>

com.cloudflare.edge.r2.bucket.<account id>_<jurisdiction>_<bucket name>; maybe that syntax is incorrect?

D

dedboi•12/2/24, 6:24 PM

Hi! can we list a public bucket path that's not root on r2 using a s3 client with

--no-sign-request

--no-sign-request

--no-sign-request

--no-sign-request ?

D

dedboi•12/2/24, 6:26 PM

Yep

Original message was deleted

D

dedboi•12/2/24, 6:28 PM

Damn, for real? The docs only mention the root path as a path we can't list :/

Original message was deleted

D

dedboi•12/2/24, 6:28 PM

Oh okay, thanks!

D

dedboi•12/2/24, 6:29 PM

Yeah would be great

D

dedboi•12/2/24, 6:29 PM

S3 with list-objects-v2

D

dedboi•12/2/24, 6:31 PM

Yeah, this could work

D

dedboi•12/2/24, 6:32 PM

Through a worker or (authenticated?) api call ?

D

dedboi•12/2/24, 6:33 PM

Perfect, thank you!

D

dedboi•12/2/24, 6:34 PM

For sure, thanks!

D

DCsan•12/2/24, 6:35 PM

understand this is the place for R2 support

https://community.cloudflare.com/t/cannot-find-r2-bucket-items-added-via-the-dashboard/742928

I uploaded an image via the dashboard to an R2 bucket.
but the image cannot be found when using the API.
other objects added via API display fine, so I know it’s not a problem with the bucket.
are items added via the dashboard somehow not available via the API?

What are the steps to reproduce the issue?
image is available here:

https://dash.cloudflare.com/ed5926d2699189886ec701e9e567d729/r2/default/buckets/wishes/objects/eclipse.jpg/details

async function getObjectOrDefault(env: any, objId: string) {
  let object = await env.WISHES_BUCKET.get(objId as string);
  if (object === null) {
    console.error('Object Not Found', objId);

async function getObjectOrDefault(env: any, objId: string) {
  let object = await env.WISHES_BUCKET.get(objId as string);
  if (object === null) {
    console.error('Object Not Found', objId);

async function getObjectOrDefault(env: any, objId: string) {
  let object = await env.WISHES_BUCKET.get(objId as string);
  if (object === null) {
    console.error('Object Not Found', objId);

async function getObjectOrDefault(env: any, objId: string) {
  let object = await env.WISHES_BUCKET.get(objId as string);
  if (object === null) {
    console.error('Object Not Found', objId);

Original message was deleted

D

dedboi•12/2/24, 6:40 PM

Will list-objects call be possible in the future though?

Original message was deleted

D

dedboi•12/2/24, 6:43 PM

Oh ok, thank you so much!

⚡

⚡ techb0x 🌟•12/2/24, 10:42 PM

from R2, .pkg Mac package are streamed to the browser instead of lauch the download action. is there something I can do on my end ? any experience ?

S

Spike•12/3/24, 1:27 AM

Just wondering if there's a way to move a bucket to another region while keeping its name and content?

SSpike Just wondering if there's a way to move a bucket to another region while keeping...

C

Chaika•12/3/24, 1:31 AM

Sadly not. Best you can do is make a second bucket and then rclone the contents over.
It may be worth pointing out as well

Location Hints are only honored the first time a bucket with a given name is created. If you delete and recreate a bucket with the same name, the original bucket’s location will be used.

CChaika Sadly not. Best you can do is make a second bucket and then rclone the contents ...

S

Spike•12/3/24, 1:33 AM

Ah, that's what's happened. I definitely hinted at APAC but I think that region wasn't available when I first created the bucket a while back.

CChaika Sadly not. Best you can do is make a second bucket and then rclone the contents ...

S

Spike•12/3/24, 1:33 AM

I assume I could also use the super slurper to do the migration?

SSpike I assume I could also use the super slurper to do the migration?

C

Chaika•12/3/24, 1:36 AM

As long as you fit within the limits yea, can't do objects over 1 tb

CChaika As long as you fit within the limits yea, can't do objects over 1 tb

S

Spike•12/3/24, 1:37 AM

All good

It was just a recent migration from S3 so definitely fits that condition.

SSpike Ah, that's what's happened. I definitely hinted at APAC but I think that region ...

C

Chaika•12/3/24, 1:37 AM

hmm I'm pretty sure APAC has always been available & they've only added hints with actual support in the region, possible it didn't listen or something else happened though, whatever it says for bucket location under settings is what matters though

CChaika hmm I'm pretty sure APAC has always been available & they've only added hints wi...

S

Spike•12/3/24, 1:38 AM

Interesting. I'll definitely make sure it's the case before I begin the migration

B

BluB•12/3/24, 3:51 AM

full cors

Full Cors

[
  {
    "AllowedOrigins": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "POST",
      "PUT",
      "DELETE"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": [
      "*"
    ],
    "MaxAgeSeconds": 3600
  }
]

[
  {
    "AllowedOrigins": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "POST",
      "PUT",
      "DELETE"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": [
      "*"
    ],
    "MaxAgeSeconds": 3600
  }
]

[
  {
    "AllowedOrigins": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "POST",
      "PUT",
      "DELETE"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": [
      "*"
    ],
    "MaxAgeSeconds": 3600
  }
]

[
  {
    "AllowedOrigins": [
      "*"
    ],
    "AllowedMethods": [
      "HEAD",
      "GET",
      "POST",
      "PUT",
      "DELETE"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": [
      "*"
    ],
    "MaxAgeSeconds": 3600
  }
]

B

BluB•12/3/24, 3:51 AM

so I have this code

import { S3Client, ListObjectsV2Command, ListBucketsCommand } from "@aws-sdk/client-s3";

async function checkAuthentication() {
        client.current = new S3Client({
            region: "auto",
            credentials: {
                accessKeyId: accessKeyId,
                secretAccessKey: secretAccessKey,
            },
            endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
            forcePathStyle: true,
        });
        setAuthMessage("Case 1")


        try {
            // @ts-ignore
            await refreshBuckets(null, true)
            setIsAuthenticated(true)
            setAuthMessage("Verified")
        } catch (err) {
            // @ts-ignore
            console.error("S3 Client authentication failed:", err.message);
            setIsAuthenticated(false)
            setAuthMessage(`err ${err}`)
        }
        // setAuthMessage("??")
    }

 async function refreshBuckets(_e: MouseEventHandler<HTMLButtonElement>, throwErr = false) {
        try {
            // @ts-ignore
            /*
             {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT", "DELETE"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3000
  }
             */
            client.current.send(new ListBucketsCommand({})).then((r) => {
                setA(r.$metadata)
            }).catch((err) => {
                setA(err.toString())
            })
            const data = await client.current.send(new ListBucketsCommand({}))
            // console.log(data);
        } catch (err) {
            // data = []
            if (throwErr) {
                throw err;
            }
        }
    }

import { S3Client, ListObjectsV2Command, ListBucketsCommand } from "@aws-sdk/client-s3";

async function checkAuthentication() {
        client.current = new S3Client({
            region: "auto",
            credentials: {
                accessKeyId: accessKeyId,
                secretAccessKey: secretAccessKey,
            },
            endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
            forcePathStyle: true,
        });
        setAuthMessage("Case 1")


        try {
            // @ts-ignore
            await refreshBuckets(null, true)
            setIsAuthenticated(true)
            setAuthMessage("Verified")
        } catch (err) {
            // @ts-ignore
            console.error("S3 Client authentication failed:", err.message);
            setIsAuthenticated(false)
            setAuthMessage(`err ${err}`)
        }
        // setAuthMessage("??")
    }

 async function refreshBuckets(_e: MouseEventHandler<HTMLButtonElement>, throwErr = false) {
        try {
            // @ts-ignore
            /*
             {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT", "DELETE"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3000
  }
             */
            client.current.send(new ListBucketsCommand({})).then((r) => {
                setA(r.$metadata)
            }).catch((err) => {
                setA(err.toString())
            })
            const data = await client.current.send(new ListBucketsCommand({}))
            // console.log(data);
        } catch (err) {
            // data = []
            if (throwErr) {
                throw err;
            }
        }
    }

import { S3Client, ListObjectsV2Command, ListBucketsCommand } from "@aws-sdk/client-s3";

async function checkAuthentication() {
        client.current = new S3Client({
            region: "auto",
            credentials: {
                accessKeyId: accessKeyId,
                secretAccessKey: secretAccessKey,
            },
            endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
            forcePathStyle: true,
        });
        setAuthMessage("Case 1")


        try {
            // @ts-ignore
            await refreshBuckets(null, true)
            setIsAuthenticated(true)
            setAuthMessage("Verified")
        } catch (err) {
            // @ts-ignore
            console.error("S3 Client authentication failed:", err.message);
            setIsAuthenticated(false)
            setAuthMessage(`err ${err}`)
        }
        // setAuthMessage("??")
    }

 async function refreshBuckets(_e: MouseEventHandler<HTMLButtonElement>, throwErr = false) {
        try {
            // @ts-ignore
            /*
             {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT", "DELETE"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3000
  }
             */
            client.current.send(new ListBucketsCommand({})).then((r) => {
                setA(r.$metadata)
            }).catch((err) => {
                setA(err.toString())
            })
            const data = await client.current.send(new ListBucketsCommand({}))
            // console.log(data);
        } catch (err) {
            // data = []
            if (throwErr) {
                throw err;
            }
        }
    }

import { S3Client, ListObjectsV2Command, ListBucketsCommand } from "@aws-sdk/client-s3";

async function checkAuthentication() {
        client.current = new S3Client({
            region: "auto",
            credentials: {
                accessKeyId: accessKeyId,
                secretAccessKey: secretAccessKey,
            },
            endpoint: `https://${accountId}.r2.cloudflarestorage.com`,
            forcePathStyle: true,
        });
        setAuthMessage("Case 1")


        try {
            // @ts-ignore
            await refreshBuckets(null, true)
            setIsAuthenticated(true)
            setAuthMessage("Verified")
        } catch (err) {
            // @ts-ignore
            console.error("S3 Client authentication failed:", err.message);
            setIsAuthenticated(false)
            setAuthMessage(`err ${err}`)
        }
        // setAuthMessage("??")
    }

 async function refreshBuckets(_e: MouseEventHandler<HTMLButtonElement>, throwErr = false) {
        try {
            // @ts-ignore
            /*
             {
    "AllowedOrigins": ["*"],
    "AllowedMethods": ["GET", "POST", "PUT", "DELETE"],
    "AllowedHeaders": ["*"],
    "MaxAgeSeconds": 3000
  }
             */
            client.current.send(new ListBucketsCommand({})).then((r) => {
                setA(r.$metadata)
            }).catch((err) => {
                setA(err.toString())
            })
            const data = await client.current.send(new ListBucketsCommand({}))
            // console.log(data);
        } catch (err) {
            // data = []
            if (throwErr) {
                throw err;
            }
        }
    }

I get

TypeError: Load failed

TypeError: Load failed

TypeError: Load failed

TypeError: Load failed every time, can anyone tell me why?

S

Sam•12/3/24, 2:27 PM

Same @Bnael

S

Sam•12/3/24, 2:27 PM

It's reaaally slow.

J

JulienDoku•12/3/24, 2:30 PM

Soooo slow!!

I'm based in Paris, France

J

JulienDoku•12/3/24, 2:31 PM

Should be related to this: https://www.cloudflarestatus.com/incidents/dxtbbpscpqm6

Network Performance Issues in Frankfurt, Germany

E

embraxx•12/3/24, 2:46 PM

Several errors from Singapore too, I don't think it's related to the Frankfurt node.

R2 giving either 524 or even 500 errors randomly

C

Chaika•12/3/24, 2:50 PM

Isaac is escalating the issue

C

Chaika•12/3/24, 2:50 PM

yea should be part of status page

Eembraxx Several errors from Singapore too, I don't think it's related to the Frankfurt n...

C

Chaika•12/3/24, 2:50 PM

is your bucket weur/eeur though? (you can see it in the dashboard under your bucket -> settings -> location)

C

Chaika•12/3/24, 2:52 PM

sorry that was to embraxx who said they were getting errors in SIN, was curious about his bucket location

I

Isaac McFadyen•12/3/24, 2:52 PM

Yeah we escalated, thanks for reports.

Is there any documentation that says what IAM rules/roles are needed to enable sippy on a r2 bucket

Similar Threads

Similar Threads

Similar Threads