Kian finding possible R2 bugs

Kian solving R2 bugs

58 Replies

@kian Now you don't clutter up the main R2 channel 🙂

kian•2y ago

Aaaaaaaaaaaaand it was because I didn't include a body in my replayed requests. So where does NoSuchBucket come from in Ben's case blobcatthinking

is it the aws s3 tool that you're using or rclone? NoSuchBucket that I was getting was just a bad request (no body in the PUT), thought I'd copied everything over but evidently not. A few other people have reported NoSuchBucket in the past with multipart uploads though, including rclone 1.59, so I'm still curious as to how it happens

Unknown User•2y ago

Message Not Public

kian•2y ago

Sure, go for it

Unknown User•2y ago

Message Not Public

kian•2y ago

My assumption is that it has a hiccup somewhere, possibly something gets lost in transit, and a bad request causes NoSuchBucket but I honestly can't understand how a client would do it. I can cause that easily, but that's by sending a bad request. How big are the files?

Unknown User•2y ago

Message Not Public

kian•2y ago

I expect it to be a pretty niche bug I think multipart concurrency in R2 can be a bit temperamental at a certain level which could cause it to intermittently say no - but in my experience that returns a signature error

Unknown User•2y ago

Message Not Public

kian•2y ago

Does that upload failed exit the command entirely?

Unknown User•2y ago

Message Not Public

kian•2y ago

Fair enough - that was pretty much what I was wondering, if that's how or not. Lemme try the same scenario against S3 and see what they return

Unknown User•2y ago

Message Not Public

kian•2y ago

Seems to be Content-Length related. Since I sent with no body, it was sending Content-Length: 0 Ehhhh, maybe not so simple - I mean I am saying that by sending Content-Length to 1 when I'm not actually sending 1 byte ~~The NoSuchBucket scenario doesn't actually give me anything but a 200 OK from AWS~~ So resending an existing part is fine (200) and then an aborted one returns NoSuchUpload.

kian•2y ago

kian•2y ago

To clarify: S3 returns NoSuchUpload when the multipart upload is cancelled, regardless of body. R2 returns NoSuchBucket if there is no body and NoSuchUpload if there is a body. As far as resending an UploadPart with a different body to the original one: S3 accepts it with a 200 OK and returns the ETag in a header R2 returns InternalError if there is a body and NoSuchBucket if there is no body

kian•2y ago

So I can just resend any random UploadPart to S3 with whatever body I want, or no body at all, and it'll accept it with a 200 OK status.

kian•2y ago

Doing the same in R2 returns InternalError if I provide a body, or NoSuchBucket if I don't provide a body. I don't know if this is an actual bug since it seems like a pretty niche case that is entirely made-up with my Postman requests, or if it's just a design decision that's up to the provider.

Unknown User•2y ago

Message Not Public

kian•2y ago

Unknown User•2y ago

Message Not Public

kian•2y ago

It'd explain why you couldn't line up the error messages (but could line up the timestamps) earlier

Unknown User•2y ago

Message Not Public

kian•2y ago

Funnily enough, if I do actually send an invalid bucket name, I get NoSuchUpload rather than NoSuchBucket

Unknown User•2y ago

Message Not Public

kian•2y ago

Unknown User•2y ago

Message Not Public

kian•2y ago

The InternalError I get when I'm re-sending an UploadPart in Postman, I also can't replicate in aws-sdk-js

[AWS s3 200 0.97s 0 retries] uploadPart({
  Bucket: 'sdk-example',
  Key: 'test.dat',
  PartNumber: 1,
  Body: 'abc',
  UploadId: 'AE9eaMe15Tv7HIxpQiVGBoLqsLc4wZQz+JxvRcCgu7NbZEBhM9z7RrHsI5Xrpp24Y/qmEz+TiFpIoJiprVrUpn2RaX+hCxpk//vMsHN10Vt3t8Hh6n33bwtOFJbQ9wdzfm2rFzPnq2qQsz0dpGpEdpk/r6TlvqnGfvWU60EtwJgym8tSLrXS+Px8W/URohF3VPaYlE883FBwQY2dETnTx81S7F41JxuP8ngcl/WFjlXH/q7D6QFpY8OsamPwLBRKvqKE/ZfUrSK6LuHkNraxfQ0ERthLMjTWGRjkn5zepQ2s2HC+or/ZBOlPAuGl29GTEA=='
})
uploadPart1 status code: 200
[AWS s3 200 1.034s 0 retries] uploadPart({
  Bucket: 'sdk-example',
  Key: 'test.dat',
  PartNumber: 1,
  Body: 'aaa',
  UploadId: 'AE9eaMe15Tv7HIxpQiVGBoLqsLc4wZQz+JxvRcCgu7NbZEBhM9z7RrHsI5Xrpp24Y/qmEz+TiFpIoJiprVrUpn2RaX+hCxpk//vMsHN10Vt3t8Hh6n33bwtOFJbQ9wdzfm2rFzPnq2qQsz0dpGpEdpk/r6TlvqnGfvWU60EtwJgym8tSLrXS+Px8W/URohF3VPaYlE883FBwQY2dETnTx81S7F41JxuP8ngcl/WFjlXH/q7D6QFpY8OsamPwLBRKvqKE/ZfUrSK6LuHkNraxfQ0ERthLMjTWGRjkn5zepQ2s2HC+or/ZBOlPAuGl29GTEA=='
})
replayPart1 status code: 200

[AWS s3 200 0.97s 0 retries] uploadPart({
  Bucket: 'sdk-example',
  Key: 'test.dat',
  PartNumber: 1,
  Body: 'abc',
  UploadId: 'AE9eaMe15Tv7HIxpQiVGBoLqsLc4wZQz+JxvRcCgu7NbZEBhM9z7RrHsI5Xrpp24Y/qmEz+TiFpIoJiprVrUpn2RaX+hCxpk//vMsHN10Vt3t8Hh6n33bwtOFJbQ9wdzfm2rFzPnq2qQsz0dpGpEdpk/r6TlvqnGfvWU60EtwJgym8tSLrXS+Px8W/URohF3VPaYlE883FBwQY2dETnTx81S7F41JxuP8ngcl/WFjlXH/q7D6QFpY8OsamPwLBRKvqKE/ZfUrSK6LuHkNraxfQ0ERthLMjTWGRjkn5zepQ2s2HC+or/ZBOlPAuGl29GTEA=='
})
uploadPart1 status code: 200
[AWS s3 200 1.034s 0 retries] uploadPart({
  Bucket: 'sdk-example',
  Key: 'test.dat',
  PartNumber: 1,
  Body: 'aaa',
  UploadId: 'AE9eaMe15Tv7HIxpQiVGBoLqsLc4wZQz+JxvRcCgu7NbZEBhM9z7RrHsI5Xrpp24Y/qmEz+TiFpIoJiprVrUpn2RaX+hCxpk//vMsHN10Vt3t8Hh6n33bwtOFJbQ9wdzfm2rFzPnq2qQsz0dpGpEdpk/r6TlvqnGfvWU60EtwJgym8tSLrXS+Px8W/URohF3VPaYlE883FBwQY2dETnTx81S7F41JxuP8ngcl/WFjlXH/q7D6QFpY8OsamPwLBRKvqKE/ZfUrSK6LuHkNraxfQ0ERthLMjTWGRjkn5zepQ2s2HC+or/ZBOlPAuGl29GTEA=='
})
replayPart1 status code: 200

I think Postman is just cursed, somewhere. What else is apart of the uploadId? User-Agent or anything like that?

Unknown User•2y ago

Message Not Public

kian•2y ago

Is what's apart of uploadId up to implementation or is it defined in the S3 spec? i.e AWS just says Upload ID identifying the multipart upload whose part is being uploaded.

Unknown User•2y ago

Message Not Public

kian•2y ago

Gotcha

Unknown User•2y ago

Message Not Public

kian•2y ago

Oh yeah, I think I recall that discussion when ncw was getting the R2 provider into rclone or at least I remember seeing it then - could just be losing my mind

Unknown User•2y ago

Message Not Public

kian•2y ago

R2 needs to have a deep-dive blog post, or like the 'How Workers Work' pages in the documentation - it all sounds pretty blobcatthinking

don't know why I'm googling about the two now since I'm fairly certain I won't be finding R2 on Google.

Unknown User•2y ago

Message Not Public

kian•2y ago

The question is how many keystrokes it'd take to fix the bug haha Just a bad comparison/match?

Unknown User•2y ago

Message Not Public

kian•2y ago

uploadEmptyPartResp.val.category == BucketErrorCategory.BucketNotFound, errContext admittedly confuses me

Unknown User•2y ago

Message Not Public

kian•2y ago

So aws s3 defaults to 10 concurrency, uploadEmptyPartResp.val.category was BucketErrorCategory.TooMuchConcurrency but ended up going down the first path & returning error.NoSuchBucket?

Unknown User•2y ago

Message Not Public

Isaac McFadyen•2y ago

K, I gotta ask. Whats the a == b, c syntax do?

kian•2y ago

so in my psuedo-JS to replicate the same stuff (and console.log instead of return), somehow it matches every if statement because of errContext errContext being 'truthy' is matching the if blocks, regardless of uploadEmptyPartResp.val.category blobcatoh

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Operators/Comma_Operator

Isaac McFadyen•2y ago

Huh

Unknown User•2y ago

Message Not Public

kian•2y ago

bless eslint Welp, interesting bug haha I'd never heard of the comma operator before 2 or 3 I believe 3 works but could be a little flaky? Id use 2

kian•2y ago

Found this when searching for concurrency :^) I guess my InternalError for a changed part size will be BadUpload eventually and the NoSuchBucket will return the proper NoSuchUpload

Unknown User•2y ago

Message Not Public

kian•2y ago

The word regression just reminds me of Sentry emails - Sentry marked FOO-123 as a regression - constantly. I couldn’t live without Sentry though Our apps are PHP/Laravel and the stack traces & context it brings with exceptions is invaluable

Unknown User•2y ago

Message Not Public

kian•2y ago

Will archive this

Gaming

Programming

Kian finding possible R2 bugs