`CopyObject` S3 compatibility

CopyObject
CopyObject
S3 compatibility

S3 clients will send the

x-amz-tagging-directive

x-amz-tagging-directive

which is unsupported as per https://developers.cloudflare.com/r2/api/s3/api/.

Has anyone actually been able to use CopyObject with R2 in that case? The S3 API will either use a value of

REPLACE

REPLACE

COPY

COPY

and defaults to

COPY

COPY

and I can definitely see that

COPY

COPY

is unsupported at the very least.

Can someone confirm that R2 works without this header being passed through, and that internally instead of defaulting to

COPY

COPY

it's then no-op? If so, it shouldn’t take too long to just at least not throw an error to allow the clients to copy objects. Otherwise, I will have to go extremely low level and send an API request manually to bypass client behaviour of sending this header.

It's not really dev friendly for CloudFlare to not adhere to default client implementation when trying to provide S3 compatibility...

Erisa•12/10/23, 2:42 PM

I don't understand the question. Tags are not supported, so any options or endpoints relating to them will fail or be no-op. Every effort is taken to ensure compatability with library implementations when deciding how to handle these unsupported properties.

Are you having any specific errors and if so can you share an example with e.g. the AWS SDK code that reproduces the issue?

t3rminatorOP•12/11/23, 12:45 PM

They’re not, but the SDK will send the header

t3rminatorOP•12/11/23, 12:45 PM

And CloudFlare only does a no-op if it’s not there

t3rminatorOP•12/11/23, 12:45 PM

But SDK default behaviour is to send it

t3rminatorOP•12/11/23, 12:46 PM

using Amazon.S3;
using Amazon.S3.Model;

var client = new AmazonS3Client(config);

var request = new CopyObjectRequest
{
    SourceBucket = originBucket,
    SourceKey = fileIdString,
    DestinationBucket = destinationBucket,
    DestinationKey = fileIdString
};

await client.CopyObjectAsync(request);

using Amazon.S3;
using Amazon.S3.Model;

var client = new AmazonS3Client(config);

var request = new CopyObjectRequest
{
    SourceBucket = originBucket,
    SourceKey = fileIdString,
    DestinationBucket = destinationBucket,
    DestinationKey = fileIdString
};

await client.CopyObjectAsync(request);

Original message was deleted

t3rminatorOP•12/11/23, 12:46 PM

The point is that while it’s not supported, the copy request should still go through

t3rminatorOP•12/11/23, 12:47 PM

Above code yields:

Amazon.S3.AmazonS3Exception: Header 'x-amz-tagging-directive' with value 'COPY' not implemented

Amazon.S3.AmazonS3Exception: Header 'x-amz-tagging-directive' with value 'COPY' not implemented

t3rminatorOP•12/11/23, 12:48 PM

I’ve pointed to the SDK code here: https://stackoverflow.com/a/77633341/4800344

Stack Overflow

Why doesn't CopyObject for CloudFlare R2 not work with the AWS SDK ...

I'm trying to move my object between two buckets.
I execute a CopyRequest, then a DeleteRequest. But when I'm trying to copy the object, I get the following error:

Amazon.S3.AmazonS3Exception: Hea...

t3rminatorOP•12/11/23, 12:48 PM

The same happens using the JS, Python and Java SDKs too

t3rminatorOP•12/17/23, 12:21 AM

@Erisa | Support Engineer Is that clearer? Happy to provide any more info as needed

Valerii M•1/10/24, 6:05 PM

Hey @ermiya , did you have a chance to find the solution for your problem?
Looks like we have the same problem when using the .NET SDK and R2 CopyObject method.

t3rminatorOP•1/10/24, 6:24 PM

No, problem still persists

t3rminatorOP•1/10/24, 6:24 PM

I’m disappointed that I’ve provided full reproduction steps and nothing from CloudFlare

t3rminatorOP•1/10/24, 6:24 PM

It’d be solved with under 10 lines of code; I’d do a PR if it was open source but alas not

Valerii M•1/10/24, 6:30 PM

Yeah, I've found that we may try to solve this problem with JS AWS SDK: https://developers.cloudflare.com/r2/examples/aws/custom-header/#set-a-header-for-all-requests-with-aws-sdk-js-v3 , as I understand AWS allow us to execute middleware after the request is formed by their Marshall.
But it sounds insanely disproportionate to deploy separate app just to copy objects between buckets...

So @Erisa | Support Engineer , @Brandon | Support Engineer , we'd really appreciate your help on guidance how this problem should be solved.

Configure custom headers · Cloudflare R2 docs

Some of R2’s extensions require setting a specific header when using them in the S3 compatible API. For some functionality you may want to set a …

Tt3rminator I’m disappointed that I’ve provided full reproduction steps and nothing from Clo...

Erisa•1/10/24, 6:31 PM

Sorry your last ping slipped through the cracks as help on here is best effort, but I've now raised this internally with the R2 team and will see what they can do to help with this

t3rminatorOP•1/10/24, 6:35 PM

Thank you, let me know please if they can’t reproduce or at least have and now is filled as a bug

t3rminatorOP•1/10/24, 6:35 PM

Appreciate it

Valerii M•1/10/24, 6:37 PM

If you need any help from my side as well – let me know, it's reproducible for our team

Tt3rminator Thank you, let me know please if they can’t reproduce or at least have and now i...

Erisa•1/10/24, 6:38 PM

Yeah I'll keep this thread updated, the reproduction you gave seems pretty clear so don't think we'll have problems but will let you know

Tt3rminator Thank you, let me know please if they can’t reproduce or at least have and now i...

Erisa•1/10/24, 9:41 PM

The R2 team have filed an internal ticket to track getting this fixed
cc @Valerii Maslenykov

t3rminatorOP•1/10/24, 9:49 PM

@Erisa | Support Engineer Legend thank you

t3rminatorOP•1/10/24, 9:49 PM

Valerii M•1/11/24, 1:38 AM

@Erisa | Support Engineer , thanks for the update. Do you have any ETA for this ticket to be completed?

Knowing the size of organization tickets like these can be postponed for a very long time, so I’d like to know whether you plan to do this in the short term

Valerii M•1/11/24, 3:52 PM

Hello, @Erisa | Support Engineer . Just a kind reminder about my question

VValerii M @Erisa | Support Engineer , thanks for the update. Do you have any ETA for this ...

Erisa•1/11/24, 3:58 PM

Hi, sorry, I'm not able to give an ETA on this, but I'll keep here updated if I have more to share. I just know its on their radar now and will be tackled when they can

Valerii M•1/11/24, 4:21 PM

@Erisa | Support Engineer , thank you

Valerii M•1/11/24, 4:44 PM

@ermiya , if you're still interested in whether CopyObject works without AWS SDK, the answer is yes.
I was able to make it work using CloudFlare's Postman collection https://developers.cloudflare.com/r2/examples/postman/

Planning to use AWS SDK or something like this: https://github.com/tsibelman/aws-signer-v4-dot-net

Postman · Cloudflare R2 docs

Postman is an API platform that makes interacting with APIs easier. This guide will explain how to use Postman to make authenticated R2 requests to …

GitHub

GitHub - tsibelman/aws-signer-v4-dot-net: Sign HttpRequestMessage u...

Sign HttpRequestMessage using AWS Signature v4 using request information and credentials. - GitHub - tsibelman/aws-signer-v4-dot-net: Sign HttpRequestMessage using AWS Signature v4 using request i...

Erisa•1/11/24, 5:38 PM

Are you also able to try alternate libraries such as https://github.com/minio/minio-dotnet ?

GitHub

GitHub - minio/minio-dotnet: MinIO Client SDK for .NET

MinIO Client SDK for .NET. Contribute to minio/minio-dotnet development by creating an account on GitHub.

Valerii M•1/11/24, 5:46 PM

@Erisa | Support Engineer , thanks for your suggestion, that may be one of many solutions for our problem. It sounds like it should work properly, as far as in their code they set this header only when tags are provided.
https://github.com/minio/minio-dotnet/blob/4c4a8917e758a1cb6de20d1bd4f7e5fcb7f24606/Minio/DataModel/Args/CopyObjectRequestArgs.cs#L125

GitHub

minio-dotnet/Minio/DataModel/Args/CopyObjectRequestArgs.cs at 4c4a8...

MinIO Client SDK for .NET. Contribute to minio/minio-dotnet development by creating an account on GitHub.

Erisa•1/11/24, 5:47 PM

Yeah

SeanK•1/25/24, 12:52 PM

I tried Min.io .net client which works if you have no nesting in your copy:

var srcArgs = new CopySourceObjectArgs().WithBucket("sourceBucket").WithObject("file.txt");
var copyArgs = new CopyObjectArgs().WithBucket("targetBucket").WithObject("file.txt").WithCopyObjectSource(srcArgs);
await minioClient.CopyObjectAsync(copyArgs);

But this fails:

var srcArgs = new CopySourceObjectArgs().WithBucket("sourceBucket").WithObject("folder/file.txt");
var copyArgs = new CopyObjectArgs().WithBucket("targetBucket").WithObject("folder/file.txt").WithCopyObjectSource(srcArgs);
await minioClient.CopyObjectAsync(copyArgs);

As an alternative, what I ended up doing is running rclone inside a process:

var psi = new ProcessStartInfo();
psi.FileName = "C:\pathToRClone\rclone.exe";
psi.Arguments = "connection:sourceBucket/folder/file.txt connection:targetbucket/folder";
psi.RedirectStandardError = true;
psi.UseShellExecute = false;

var process = Process.Start(psi);

var standardErrorTask = process.StandardError.ReadToEndAsync();

var internalTimeOut = TimeSpan.FromHours(1);
var timeoutSignal = new CancellationTokenSource(internalTimeOut);

try
{
await process.WaitForExitAsync(timeoutSignal.Token);

var standardError = await standardErrorTask;
if (process.ExitCode != 0)
{
throw new Exception($"Proccess failed. Exit Code {process.ExitCode}. StdError {standardError}");
}
}
catch
{
process.Kill();
process.WaitForExit();

throw;
}

Erisa•1/25/24, 2:35 PM

But this fails:

Whats the error you get back?

SeanK•1/25/24, 3:55 PM

I did another test and it's not the nesting that's the problem, it's the type of file (.ts which I'm sure I could sort out by setting the right content type somewhere). I can transfer a m3u8 text file nested without issues so Min.io is a solid workaround as suggested.

t3rminatorOP•1/31/24, 7:51 PM

Running rclone in a process like this is incredibly dangerous and extremely non practical for serverless environments

t3rminatorOP•1/31/24, 7:51 PM

Other SDK implementations may work but the main AWS SDK should be the one that is used for verify compatibility - it is S3-compatible not Minio-Compatible

Valerii M•2/2/24, 12:42 PM

@Erisa | Support Engineer , are there any updates on the issue?
We decided to deprioritize features where the CopyObject is used, to avoid using multiple SDK clients on our project (cause a lot of stuff was built on AWS SDK already).

using Amazon.S3; using Amazon.S3.Model; var client = new AmazonS3Client(config); var request = new CopyObjectRequest { SourceBucket = originBucket, SourceKey = fileIdString, DestinationBucket = destinationBucket, DestinationKey = fileIdString }; await client.CopyObjectAsync(request);

`CopyObject` S3 compatibility

Similar Threads

Similar Threads

Similar Threads