Another small release for helping out with Terraform: * S3 CreateBucket request can specify `x-a

Another small release for helping out with Terraform:

S3 CreateBucket request can specify
```
x-amz-bucket-object-lock-enabled
```
x-amz-bucket-object-lock-enabled
```
x-amz-bucket-object-lock-enabled
```
x-amz-bucket-object-lock-enabled
with a value of
```
false
```
```
false
```
and not have the requested rejected with a
```
NotImplemented
```
NotImplemented
```
NotImplemented
```
NotImplemented
error. A value of
```
true
```
```
true
```
will continue to be rejected as R2 does not yet support object locks.

Bbeelzabub Another small release for helping out with Terraform: * S3 CreateBucket requ...

Cyb3r-Jak3•7/1/22, 9:01 PM

Just tested and it kinda of works but not really. You can create the bucket but the plan doesn't successfully complete and you can't refresh the plan or destroy the bucket. It tries to check the bucket ACL via GetBucketACLGetBucketACL which isn't implemented.

beelzabubOP•7/1/22, 9:10 PM

Is there any way to skip that?

Cyb3r-Jak3•7/1/22, 9:32 PM

It doesn't look like it. When reading a bucket resource it seems to always attempt to read the ACL.
https://github.com/hashicorp/terraform-provider-aws/blob/main/internal/service/s3/bucket.go#L964-L970

beelzabubOP•7/1/22, 9:39 PM

Might make sense to fix the terraform provider to be more graceful about servers not implementing various features? It handles empty responses so why not treat notimplemented as empty

Cyb3r-Jak3•7/1/22, 9:45 PM

Sounds good. I'll raise an issue there

beelzabubOP•7/1/22, 9:49 PM

The (potentially incomplete) set of commands that don't handle

NotImplemented

NotImplemented

gracefully:

GetBucketAcl
GetBucketVersioning
GetBucketLifecycleConfiguration
GetBucketReplication
GetBucketEncryption (but R2 does implement it)
GetObjectLockConfiguration
GetBucketTagging

beelzabubOP•7/1/22, 9:50 PM

The rest seem to gracefully ignore

NotImplemented

NotImplemented

Cyb3r-Jak3•7/1/22, 10:18 PM

Issue raised: https://github.com/hashicorp/terraform-provider-aws/issues/25660
Thanks for your help on this!

SnowRunescape•7/1/22, 10:39 PM

?r2-roadmap

Flare•7/1/22, 10:39 PM

The following features are currently on the roadmap for R2 but not yet implemented.

Public Buckets
Mapping a bucket to your custom domain
Pre-signed URLs are now implemented!

In the meantime, you can use a Worker to replicate the vast majority of functionality such as serving & caching public assets.
Take a look at the Demo Worker example here: https://developers.cloudflare.com/r2/examples/demo-worker/
Also, there's a great community project called Render that's made for R2:
https://github.com/kotx/render

Workers will not be required for making assets publicly available in the future, they are just an interim stop-gap.

Pharynx•7/1/22, 10:42 PM

any news on when we'll be able to upload single objects with more concurrency?

beelzabubOP•7/2/22, 3:34 AM

Not sure.

kian•7/2/22, 1:56 PM

Where is the ReadableStream coming from? If it’s the body of a fetch / request and it has a Content-Length then it shouldn’t be an issue

James•7/2/22, 2:07 PM

If you know the length, you may be able to use a FixedLengthStream instead

kian•7/2/22, 2:13 PM

R2 won't accept it if the content length isn't known at upload so I'm not entirely sure how you'd get around that in a Worker - if you'd run out of resources waiting for it to complete

kian•7/2/22, 2:18 PM

I'm fairly certain you'd run into the same restriction with most S3-compatible providers - a Content-Length is required.

i.e: from the Java AWS SDK

This field is required when uploading objects to S3, but the Amazon Web Services S3 Java client will automatically set it when working directly with files. When uploading directly from a stream, set this field if possible. Otherwise the client must buffer the entire stream in order to calculate the content length before sending the data to Amazon S3.

kian•7/2/22, 2:18 PM

Workers don't support multipart but the S3 API does - so you can upload in chunks (of the same size).

Mmax it's possible to stream into s3

kian•7/2/22, 2:21 PM

With which operation?

kian•7/2/22, 2:21 PM

Everything I'm aware of uses multipart like I mentioned

kian•7/2/22, 2:26 PM

The issue with doing multipart in a Worker would be before resources - there is no multipart bindings in Workers. The putput is PutObjectPutObject in S3 but you need the CreateMultipartUploadCreateMultipartUpload, then UploadPartUploadPart for each part and then CompleteMultipartUploadCompleteMultipartUpload

kian•7/2/22, 2:26 PM

The idea I suppose would be that you're processing a chunk of data, uploading it, and then onto the next

kian•7/2/22, 2:27 PM

https://github.com/mhart/aws4fetch might be able to do multipart uploads via the S3 API endpoint for R2 within the Worker

kian•7/2/22, 2:28 PM

The smallest chunk I believe is 5MiB but you can do as high as you want really, worth experimenting and seeing when the Worker gets upset with memory

kian•7/2/22, 2:29 PM

I'm not entirely sure how that'll fair with the other limit, CPU time, depending how big your dataset is.

kian•7/2/22, 2:29 PM

Doing the crypto stuff for SigV4 signing for lots of parts might throw you over the 10ms free limit or 50ms bundled limit, and Unbound which has up to 30 seconds might be the better usage model

kian•7/2/22, 2:31 PM

IMO Workers right now are suited well for smaller uploads and serving files, or uploading files from fetch untransformed - anything else and I’d stick with the S3 API using other libraries or tools

kian•7/2/22, 2:31 PM

Fetching a 3GB file and pushing that into R2 is easy - assuming you never touch that body and just stream it into R2 - but your use-case requires that

beelzabubOP•7/2/22, 6:35 PM

How big is the aggregate file you’re generating?

kian•7/3/22, 2:11 AM

100MB is only a Workers/zone thing - if you can’t use the S3 API for whatever reason then all you can really do is multipart in Workers (however that’d work) or upgrade your plan to Business

Another small release for helping out with Terraform: * S3 CreateBucket request can specify `x-a

Similar Threads

Similar Threads

Similar Threads