I am using AWS SDK V3 to upload/fetch files from R2. I am testing out the CreateBucketCommand and th

I am using AWS SDK V3 to upload/fetch files from R2. I am testing out the CreateBucketCommand and the ListBucketsCommand.
I am receiving these responses from both methods but no data:
response1= {
'$metadata': {
httpStatusCode: 200,
requestId: undefined,
extendedRequestId: undefined,
cfId: undefined,
attempts: 1,
totalRetryDelay: 0
}
}

I also added 'forcePathStyle: true' to the config which fixed the previous error of 'NoSuchBucket' when trying to create a bucket,
not sure how or why I had to do that. I've also tested this with AWS S3 and it creates the buckets and lists the buckets with no problems.

The code:

const options = {
  region: 'us-east-1',
  accessKeyId: process.env.ACCESS_KEY_ID!,
  secretAccessKey: process.env.SECRET_ACCESS_KEY!,
  endpoint: process.env.S3_ENDPOINT!,
  bucketName: 'test-bucket-1234-abcd-5678-efgh',
  maxFileSize: 10,
  validMimeType: [],
  format: 'url',
  urlExpiration: 10,
  path: path.join(__dirname, 'downloads')
};

describe('s3Controller', () => {
  it('should create a bucket', async () => {
    const input = {
      Bucket: options.bucketName,
    };

    try {
      const command = new CreateBucketCommand(input);
      const response = await client.send(command);
      console.log('response1=', response);

    } catch(err) {
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

describe('s3Controller', () => {
  it('should list buckets', async () => {

    try {
      const command = new ListBucketsCommand('');
      const response = await client.send(command);
      console.log('response2=', response);

    } catch(err) { 
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

const options = {
  region: 'us-east-1',
  accessKeyId: process.env.ACCESS_KEY_ID!,
  secretAccessKey: process.env.SECRET_ACCESS_KEY!,
  endpoint: process.env.S3_ENDPOINT!,
  bucketName: 'test-bucket-1234-abcd-5678-efgh',
  maxFileSize: 10,
  validMimeType: [],
  format: 'url',
  urlExpiration: 10,
  path: path.join(__dirname, 'downloads')
};

describe('s3Controller', () => {
  it('should create a bucket', async () => {
    const input = {
      Bucket: options.bucketName,
    };

    try {
      const command = new CreateBucketCommand(input);
      const response = await client.send(command);
      console.log('response1=', response);

    } catch(err) {
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

describe('s3Controller', () => {
  it('should list buckets', async () => {

    try {
      const command = new ListBucketsCommand('');
      const response = await client.send(command);
      console.log('response2=', response);

    } catch(err) { 
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

const options = {
  region: 'us-east-1',
  accessKeyId: process.env.ACCESS_KEY_ID!,
  secretAccessKey: process.env.SECRET_ACCESS_KEY!,
  endpoint: process.env.S3_ENDPOINT!,
  bucketName: 'test-bucket-1234-abcd-5678-efgh',
  maxFileSize: 10,
  validMimeType: [],
  format: 'url',
  urlExpiration: 10,
  path: path.join(__dirname, 'downloads')
};

describe('s3Controller', () => {
  it('should create a bucket', async () => {
    const input = {
      Bucket: options.bucketName,
    };

    try {
      const command = new CreateBucketCommand(input);
      const response = await client.send(command);
      console.log('response1=', response);

    } catch(err) {
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

describe('s3Controller', () => {
  it('should list buckets', async () => {

    try {
      const command = new ListBucketsCommand('');
      const response = await client.send(command);
      console.log('response2=', response);

    } catch(err) { 
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

const options = {
  region: 'us-east-1',
  accessKeyId: process.env.ACCESS_KEY_ID!,
  secretAccessKey: process.env.SECRET_ACCESS_KEY!,
  endpoint: process.env.S3_ENDPOINT!,
  bucketName: 'test-bucket-1234-abcd-5678-efgh',
  maxFileSize: 10,
  validMimeType: [],
  format: 'url',
  urlExpiration: 10,
  path: path.join(__dirname, 'downloads')
};

describe('s3Controller', () => {
  it('should create a bucket', async () => {
    const input = {
      Bucket: options.bucketName,
    };

    try {
      const command = new CreateBucketCommand(input);
      const response = await client.send(command);
      console.log('response1=', response);

    } catch(err) {
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

describe('s3Controller', () => {
  it('should list buckets', async () => {

    try {
      const command = new ListBucketsCommand('');
      const response = await client.send(command);
      console.log('response2=', response);

    } catch(err) { 
      console.log('err=', err);
    }
    expect(response).toEqual({});
  });
});

Any help would be much appreciated.

Big AlOP•1/19/24, 5:40 PM

I followed the example layed out here: https://developers.cloudflare.com/r2/examples/aws/aws-sdk-js-v3/
Is there any further documentation on aws sdk v3 besides that?

aws-sdk-js-v3 · Cloudflare R2 docs

You must generate an Access Key before getting started. All examples will utilize access_key_id and access_key_secret variables which represent the …

BBig Al I am using AWS SDK V3 to upload/fetch files from R2. I am testing out the Create...

Ɽ

ⱤØ₥₳₦•1/19/24, 5:50 PM

Thanks for idea

now I'll use Lambada edge function to upload gigantic files to R2

Barry_Based_Benson•1/19/24, 7:22 PM

Does generating a presigned url count as an operation? If so which one? A or B?

BBarry_Based_Benson Does generating a presigned url count as an operation? If so which one? A or B?

Karew•1/19/24, 7:24 PM

Presigning does not involve the server, it doesn’t cost anything

Barry_Based_Benson•1/19/24, 7:25 PM

But isn't the file content served by cloudflare? Doesn't it count as a read operation?

Barry_Based_Benson•1/19/24, 7:25 PM

@Karew

Unsmart•1/19/24, 7:27 PM

Generating a presigned url doesnt ever touch cloudflare. But when you actually go to the presigned url it will do whatever operation you told it and count

Karew•1/19/24, 7:27 PM

You asked about presigning. Your local S3 client presigns an unsent request. It doesn’t cost anything to presign. It only costs an operation if someone actually makes the request

Erisa•1/19/24, 8:57 PM

And then the type of operation charged depends on the type of request you created and used

Artefactoid•1/20/24, 12:10 AM

I just had a somewhat, um, weird experience with adding an R2 subscription. When I looked at my Billing Subscriptions, R2 was enabled. But when I attempted to configure it, all I got was the R2 intro funnel page with a big blue button asking me if wanted to add it to my subscriptions. Then I went to cancel the subscription. Only once it was canceled could I then access the actual R2 control panel. But it seems like I can't resubscribe to R2 now. Who should I be asking for help to be allowed to resubscribe to R2? TIA.

kian•1/20/24, 1:04 PM

I’ll take a look in 5 - there was an issue where the error is wrong for multi part uploads but I think that was fixed

kian•1/20/24, 1:11 PM

Did you test it w/ the example folder in the repo?

kian•1/20/24, 1:15 PM

How comes your logs don't have a remaining part? meowthinking

kian•1/20/24, 1:15 PM

kian•1/20/24, 1:20 PM

Good point

kian•1/20/24, 1:21 PM

timingSafeEqual is the same, where you need to encode the text beforehand and the two buffers have to be the same length but that means you can't rely on string.lengthstring.length

kian•1/20/24, 1:21 PM

> "a".length
1
> "à".length
1
> new TextEncoder().encode("a").byteLength
1
> new TextEncoder().encode("à").byteLength
2

> "a".length
1
> "à".length
1
> new TextEncoder().encode("a").byteLength
1
> new TextEncoder().encode("à").byteLength
2

kian•1/20/24, 1:21 PM

This lovely stuff

kian•1/20/24, 1:23 PM

This is in OP's GitHub bio so it's possible meowlul

kian•1/20/24, 1:36 PM

I tested with aa and àà & that does seem to be the issue

kian•1/20/24, 1:36 PM

that's not very fun

kian•1/20/24, 1:48 PM

Yeah, I don't think you need to consider the plan

kian•1/20/24, 8:59 PM

You'd need to see the request/response that fails specifically to be able to tell what's going wrong

Ercan•1/21/24, 6:23 PM

Hi all!

We have been trying to find a way to how to migrate from Cloudflare images to R2, the reason is cloudflare images is becoming so pricey with a large amount of images. And this will not be scalable. Can someone help us figure out what kind of process we should follow for this. And we are also not happy with the cloudflare images because we did not realize it would be this much vendor lock in.

If someone knows a way for this cloudflare images to R2 migration, that would help us a lot. As we see, there is no API or way to export all image from cloudflare images, and dump it into r2 etc.

Original message was deleted

Ercan•1/21/24, 8:29 PM

Thank you Daniel!

Original message was deleted

Ercan•1/21/24, 8:30 PM

Thank you as well for providing the example!

Ercan•1/21/24, 8:32 PM

And @nora another question, to migrate all images, I will basically need to fetch every single image and put it on r2, and I believe this will be reflected in the billing for "Delivered Image" count for cloudflare images right? I am gonna migrate around 10M+ images.

Original message was deleted

Ercan•1/21/24, 8:48 PM

I see thanks, should be good for us.

And a question related to the R2 read-after-write consistency.

Is this also valid for using custom domain for r2 access? We experienced some issues in the past where we put objects and tried to directly read by using the bucket custom domain.

kian•1/21/24, 8:49 PM

Custom domains have the default CDN caching behaviour

kian•1/21/24, 8:50 PM

So, objects may be cached and writing to R2 won't purge that

Kkian So, objects may be cached and writing to R2 won't purge that

Ercan•1/21/24, 8:52 PM

Hm, but if an object from r2 is already not in the cache and just uploaded, and if I try to access it using custom domain, why would it fail if it is already not in the cache, and it should retrieve it from the source right? Maybe I am wrong, can you explain a bit more

kian•1/21/24, 8:53 PM

404's are cached but it's impossible to tell why it happened without a repro

Kkian 404's are cached but it's impossible to tell why it happened without a repro

Ercan•1/21/24, 8:54 PM

I see, I will try to get the error and time, if I see one more time, so we could investigate maybe.

Kkian 404's are cached but it's impossible to tell why it happened without a repro

Ercan•1/21/24, 8:56 PM

I also wanna mention what I am planning for 100% relaibility. I will have a custom image resizer server running that will handle resizing etc, and I will source it to directly r2 without custom domain. But I will connect my own cloudlfare domain to my image resizer, so the results of my image resizer would be cached. And I am hoping I can reach the maximum reliability with this setup.

pinky•1/21/24, 9:21 PM

Hello, when I have a private bucket and allow access via Worker with the following code, is it going to load the file into Worker's memory and is there any usage impact compared to accessing a file in a public bucket?

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

const object = await env.BUCKET.get("key from request");
const headers = new Headers();
object.writeHttpMetadata(headers);
headers.set('etag', object.httpEtag);

return new Response(object.body, { headers });

kian•1/21/24, 9:21 PM

Nope

kian•1/21/24, 9:22 PM

Public buckets are better if you're not adding auth though

pinky•1/21/24, 9:26 PM

Is it also the case with the upload? Because I see various ways to upload a file when using worker API.

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

// e.g. via curl with --data-binary "@somefile.jpg"
await bucket.put(key, req.body, { httpMetadata: { contentType } });

// from a web page
const formData = await req.formData();
const file = formData.get('file') as File
const body = await file.arrayBuffer();
await bucket.put(file.name, body, { httpMetadata: { contentType: file.type } });

Just to confirm, the first approach is also memory free and almost not blocking the worker but the second approach needs to load everything?

kian•1/21/24, 9:38 PM

Yes

kian•1/21/24, 9:39 PM

FormData can't be uploaded as-is since it isn't just the file, it has other things

chip•1/22/24, 1:20 AM

Hey! Is it possible to slap on x-amz-expirationx-amz-expiration in metadata for objects to delete them, without having set up a lifecycle in r2?

hoalongnatsu•1/22/24, 1:45 AM

I added my credit card and active R2 subscription, but when I go to R2 page then it shows like the image below. I press "Add R2 subscription to my account account" many times but it won't work

hoalongnatsu•1/22/24, 1:47 AM

How can I work with this problem?

Cchip Hey! Is it possible to slap on `x-amz-expiration` in metadata for objects to del...

kian•1/22/24, 1:49 AM

Nope, but I also don't think S3 supports that either?

Cchip Hey! Is it possible to slap on `x-amz-expiration` in metadata for objects to del...

Karew•1/22/24, 1:49 AM

No, only lifecycles are supported at the moment

chip•1/22/24, 1:55 AM

Sounds good! Thanks!

I am using AWS SDK V3 to upload/fetch files from R2. I am testing out the CreateBucketCommand and th

Similar Threads

Similar Threads

Similar Threads