Cloudflare Developers•6mo ago

R2 Access Via Domain is Down?

Howdy! I got a report from some of my users that R2 is no longer serving content. Everything from my custom domain is returning a 200 with 0 bytes (even if the file does not exist). For example: https://cdn.btst.io/test.html

curl -i https://cdn.btst.io/test.html
HTTP/2 200
date: Thu, 11 Jan 2024 02:34:45 GMT
content-length: 0
access-control-allow-origin: *
vary: Origin
access-control-allow-credentials: true
access-control-allow-headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=8OA78wCnvcRLW8CC5ljmtce%2BUhoPFbwNekVtEOQFJR2pMlYT5PVokesuCbId8U2u05wrqi3%2FkxP3kUy3K92R%2BvEg3w7XGcMk5vQraiHBgoKxbYMbUfBYchbrM4woyQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8439c3330d605cae-RDU
alt-svc: h3=":443"; ma=86400

curl -i https://cdn.btst.io/test.html
HTTP/2 200
date: Thu, 11 Jan 2024 02:34:45 GMT
content-length: 0
access-control-allow-origin: *
vary: Origin
access-control-allow-credentials: true
access-control-allow-headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
report-to: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=8OA78wCnvcRLW8CC5ljmtce%2BUhoPFbwNekVtEOQFJR2pMlYT5PVokesuCbId8U2u05wrqi3%2FkxP3kUy3K92R%2BvEg3w7XGcMk5vQraiHBgoKxbYMbUfBYchbrM4woyQ%3D%3D"}],"group":"cf-nel","max_age":604800}
nel: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
server: cloudflare
cf-ray: 8439c3330d605cae-RDU
alt-svc: h3=":443"; ma=86400

I don't know how long this has been happening, but I don't see any evidence reported anywhere? Is this an issue on y'all's side?

23 Replies

cole•6mo ago

For what it's worth, workers fetching from R2 seem to be working fine. I would hate to rearchitect during downtime, but that seems like a viable path at this point.

kian•6mo ago

I can't reproduce this with a custom domain, and the fact it also serves a 200 OK on / despite / having never worked on custom domains is weird. What is the CORS config on that bucket? I suspect this isn't actually hitting R2, especially since it's serving CORS headers without me passing an Origin header.

cole•6mo ago

I just swapped cdn.btst.io over to a worker, so it's working now. cdn2.btst.io is pointed at the bucket now and still showing the weird behavior

cole•6mo ago

✗ curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:17 GMT
Content-Type: text/html
Content-Length: 25
Connection: keep-alive
ETag: "68219c28daba0ca0e67295f1d71aaf37"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x4xcZfVXl5sGxXIIlP3uehMNGl7SfVRPkBu%2BF9WPODYldFIi%2FaGDLfUra4%2B6WjiF6nL%2F3B184A3k9pUj2cacA6nFS1mKa%2BkZ84y8jfpkg8%2Fx8tGRsMbTsn7hwyUn0w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea1278115cb1-RDU
alt-svc: h3=":443"; ma=86400

<html>
Success!!
</html>
✗ curl -i cdn2.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400

✗ curl -i cdn.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:17 GMT
Content-Type: text/html
Content-Length: 25
Connection: keep-alive
ETag: "68219c28daba0ca0e67295f1d71aaf37"
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=x4xcZfVXl5sGxXIIlP3uehMNGl7SfVRPkBu%2BF9WPODYldFIi%2FaGDLfUra4%2B6WjiF6nL%2F3B184A3k9pUj2cacA6nFS1mKa%2BkZ84y8jfpkg8%2Fx8tGRsMbTsn7hwyUn0w%3D%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea1278115cb1-RDU
alt-svc: h3=":443"; ma=86400

<html>
Success!!
</html>
✗ curl -i cdn2.btst.io/test.html
HTTP/1.1 200 OK
Date: Thu, 11 Jan 2024 03:01:22 GMT
Content-Length: 0
Connection: keep-alive
Access-Control-Allow-Origin: *
Vary: Origin
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: Content-Type, Vary, Origin, Referer, User-Agent, Authorization
Access-Control-Allow-Methods: GET, POST, PUT, DELETE, OPTIONS, PATCH, HEAD
Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=ioFpioSLw6%2B4yTuBiRiId5XFaKbmupwDEngFurqI00Psx6r77PrA68uWDly8vNZSbBuqTfIZpRsTaoOZttQMCoqP0REN3ytXbH4w1KKsckwtB791dE8qq3QuQ6Z0PGg%3D"}],"group":"cf-nel","max_age":604800}
NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
Server: cloudflare
CF-RAY: 8439ea2cc8ed5ca1-RDU
alt-svc: h3=":443"; ma=86400

Yeah, super weird issue haha. It was working up until tonight though haha

kian•6mo ago

See what https://developers.cloudflare.com/fundamentals/basic-tasks/trace-request/ says, you'll have the trace tool on the sidebar in the dashboard.

cole•6mo ago

That's a cool little service!

cole•6mo ago

Same response for the working one with a worker, FWIW This still seems to be affected. I just realized that my less popular sites have broken R2 bucket CDNs as well 😞

kian•6mo ago

@Sid | R2 might be able to help - I don't think the traffic is hitting R2, but I also don't have the ability to figure out why it isn't going to R2.

cole•6mo ago

Thanks! That'd be helpful!! Am I asking in the right place? Or should I escalate / push somewhere else / wait for Sid? I am not super confident that I found the proper channel 😅 I'm also quite surprised that nobody else has reported.

kian•6mo ago

I would be surprised if it's affecting anyone else but you. I can't repro it and there's no other reports on the community forums or here You'd probably be better off posting in #r2

cole•6mo ago

Ah good to know! I wonder if a newer worker route is matching the cdn urls

cole•6mo ago

Ugh. Indeed. I had a worker route added that started trapping requests before they made it to R2 😭 It's so interesting to me the way that worker routes are global across all subdomains.

cole•6mo ago

Anyways, thanks for your help @kian !!

kian•6mo ago

*/* would be expected to match everything You might want to scope it down to a more specific hostname

cole•6mo ago

Oh yeah, I knocked it down. That originally came from some experiments with the "SaaS custom domain routes" for workers

kian•6mo ago

Unfortunately the trace tool you used earlier doesn't yet tell you if a Worker route was matched

cole•6mo ago

You kinda need a wildcard to catch those routes for whatever domains a SaaS user puts in and you want to run your worker on Which basically means you need a dedicated TLD per function to run a SaaS w/ custom domains on. Unless I'm missing something or you add a route for each domain

Chaika•6mo ago

You can do *btst.io/*, Service: none to stop it from taking over subdomains for that domain, the more specific one wins docs: https://developers.cloudflare.com/cloudflare-for-platforms/cloudflare-for-saas/start/advanced-settings/worker-as-origin/

Workers as your fallback origin · Cloudflare for Platforms docs

Learn how to use a Worker as the fallback origin for your SaaS zone.

cole•6mo ago

Thanks @Chaika ! Is there something like a "not match" blocker, so I can do a * that matches SaaS domains, without matching domains on my TLD? In my testing SaaS domains (i.e. user provided domains), I needed a matching rule defined in order for my worker to serve traffic to them

Chaika•6mo ago

Right, like the tutorial there says, you can make one more specific like *.<zonename>.com/* service none along with the */* for SaaS, and the more specific one wins (so service none will apply to your own domain)

cole•6mo ago

Ohhh I missed that nuance! That "None" is super helpful, thanks!

Gaming

Programming

R2 Access Via Domain is Down?