@beelzabub are you able to say... 1. If/when SHA-256 checksums will be supported in S3 CompleteMulti

@beelzabub are you able to say...

If/when SHA-256 checksums will be supported in S3 CompleteMultipartUpload operations
(https://docs.aws.amazon.com/AmazonS3/latest/API/API_CompleteMultipartUpload.html#API_CompleteMultipartUpload_RequestSyntax)?
If/when SHA-256 checksums will be supported in bindings with
```
env.R2.put(body, {sha256: '<checksum>'})
```
env.R2.put(body, {sha256: '<checksum>'})
```
env.R2.put(body, {sha256: '<checksum>'})
```
env.R2.put(body, {sha256: '<checksum>'})
?

I know the SHA-256 checksum is already calculated for S3 PutObject operations as part of validating the signature (client provides the

X-Amz-Content-Sha256

X-Amz-Content-Sha256

X-Amz-Content-Sha256

X-Amz-Content-Sha256 header), so I assume it would be relatively easy to implement it for bindings as well. I don't think the same can be said for calculating the checksum of multipart uploads though.

Erisa•7/1/22, 10:28 AM

Pages indeed doesn't support R2 yet

Erisa•7/1/22, 10:32 AM

�

𝖆𝖚𝖙𝖎𝖘𝖙•7/1/22, 4:00 PM

wym @derfrisson ? KEKWiggle

Aasp @beelzabub are you able to say... 1. If/when SHA-256 checksums will be supported...

beelzabub•7/1/22, 4:34 PM

I wouldn't hold my breath
No immediate plans but I can add it to the backlog

VVertex Hey all, are there any plans to implement v2 signatures for S3 compatibility?

Isaac McFadyen•7/1/22, 7:03 PM

I kinda doubt it? Even S3 has deprecated them as of 2020.

Isaac McFadyen•7/1/22, 7:04 PM

But who knows, I don't speak for the R2 team

kian•7/1/22, 7:04 PM

SigV2 hasn't been supported on any AWS regions launched after 2013

kian•7/1/22, 7:04 PM

I don't think it's on the way personally

Kkian SigV2 hasn't been supported on any AWS regions launched after 2013

Isaac McFadyen•7/1/22, 7:04 PM

2013, whew.

kian•7/1/22, 7:05 PM

The Europe (Frankfurt), US East (Ohio), Canada (Central), Europe (London), Asia Pacific (Seoul), Asia Pacific (Mumbai), Europe (Paris), China (Ningxia), Europe (Stockholm), Asia Pacific (Osaka Local), AWS GovCloud (US-East), and Asia Pacific (Hong Kong) Regions were launched after 2013, and support SigV4 but not SigV2. If you have code that accesses S3 buckets in that region, it is already making exclusive use of SigV4.

kian•7/1/22, 7:05 PM

kian•7/1/22, 7:05 PM

It do be oldddddd

Isaac McFadyen•7/1/22, 7:05 PM

What about v3? lul

kian•7/1/22, 7:06 PM

The HyperText Transfer Protocol (HTTP) 411 Length Required client error response code indicates that the server refuses to accept the request without a defined Content-Length header.

kian•7/1/22, 7:06 PM

R2 requires a Content-LengthContent-Length header for uploads

kian•7/1/22, 7:08 PM

Somewhat - r2

kian•7/1/22, 7:08 PM

It'll work on S3 when using MP uploads iirc but that's about it

beelzabub•7/1/22, 7:17 PM

Hmm... content length isn't required for MP part uploads?

beelzabub•7/1/22, 7:17 PM

I'll look into it

beelzabub•7/1/22, 7:23 PM

A release was made:

Unsupported search paramters to ListObjects/ListObjectsV2 are now rejected with 501 NotImplemented.
Fixes for Listing:
- Fix listing behavior when the number of files within a folder exceeds the limit (you'd end up seeing a CommonPrefix for that large folder N times where N = number of children within the CommonPrefix / limit).
- Fix corner case where listing could cause objects with sharing the base name of a "folder" to be skipped.
- Fix listing over some files that shared a certain common prefix.
DeleteObjects can now handle 1000 objects at a time.

Bbeelzabub A release was made: * Unsupported search paramters to ListObjects/ListObject...

beelzabub•7/1/22, 7:23 PM

@john.spurlock for visibility on the listing fixes.

kian•7/1/22, 7:24 PM

Bbeelzabub @john.spurlock for visibility on the listing fixes.

John Spurlock•7/1/22, 7:26 PM

nice, thanks!

John Spurlock•7/1/22, 7:26 PM

semi-permeable code freeze

beelzabub•7/1/22, 7:26 PM

Oh & deletion of multiple objects should now be faster too.

beelzabub•7/1/22, 7:27 PM

Code freeze window ended this morning

John Spurlock•7/1/22, 7:27 PM

right before the holiday eh

VVertex would be great - this is basically the only thing blocking us from replacing our...

beelzabub•7/1/22, 8:23 PM

So I looked into it and there's this:
https://stackoverflow.com/questions/69899949/upload-with-high-level-multipart-still-gives-no-content-length-specified-warning

This indicates it is indeed required even for S3. I haven't figured out a good way to actually test out omitting the content-length

Stack Overflow

Upload with high level multipart still gives no content length spec...

Even though I'm using the high level multipart I'm still getting the warning in the console:
WARN - com.amazonaws.services.s3.AmazonS3Client - No content length specified for stream data. Stream

beelzabub•7/1/22, 8:25 PM

And this: https://gitlab.com/gitlab-org/gitlab-foss/-/issues/44142

GitLab

Implement S3 compatible uploads for unknown content-length (#44142)...

New description proposal Based on gitlab-org/gitlab-ee#4184 we know that only GCS (Google Cloud Storage) accepts chunked-encoding, all the other S3-compatible

beelzabub•7/1/22, 8:25 PM

Based on gitlab-org/gitlab-ee#4184 we know that only GCS (Google Cloud Storage) accepts chunked-encoding, all the other S3-compatible Object Storage requires Content-Length HTTP header.

beelzabub•7/1/22, 8:27 PM

That's not part of S3 AFAIK.

beelzabub•7/1/22, 8:28 PM

So you're saying that GitLab is working for you with AWS S3 & not with R2?

beelzabub•7/1/22, 8:29 PM

Or are you using GitLab with GCS?

beelzabub•7/1/22, 8:30 PM

Do you make sure to disable enable_signature_v4_streamingenable_signature_v4_streaming ?

@beelzabub are you able to say... 1. If/when SHA-256 checksums will be supported in S3 CompleteMulti

Similar Threads

Similar Threads

Similar Threads